C-SDK: max buffer size for ziti_write() function?

while implementing a process that receives requests from ziti endpoints, forwards the (“de-zitified”) content to an upstream server and sends the response from that interaction back to the requesting ziti endpoint, I ran into an error when the size of the data parameter exceeded 32663 bytes. In those error cases, the client would not receive any data, yet the response/status code from the invocation of ziti-write() would suggest success. I did not find any errors in the log that seemed relevant.

I arrived at the above number of bytes through trial and error and the lack of error response was definitely unexpected. Is that a known restriction?

In my code I started by essentially copying the on_client_data(ziti_connection clt, uint8_t *data, ssize_t len) function from sample-host.c. Are there more conditions to check around the existing ziti session / connection?
FYI, the current (working) code I have is:

ssize_t ngx_ziti_downstream_on_client_data(ziti_connection clt, uint8_t *data, ssize_t len) {
    // ZITI_LOG(DEBUG, "--- ngx_ziti_downstream_on_client_data\n");

    if (len > 0) {
        ZITI_LOG(DEBUG, "client sent %d bytes", (int) len);
        ZITI_LOG(TRACE, "client sent these %d bytes:\n%.*s", (int) len, (int) len, data);

        // forward request to upstream server
        char *reply = malloc(RCV_BUFFER_SIZE);
        memset(reply,0,RCV_BUFFER_SIZE);
        int reply_len = talk_to_upstream(reply, "odoo.bomk", 8069, len, data);

        ZITI_LOG(DEBUG, "response from upstream server is %d bytes long.", reply_len);
        ZITI_LOG(TRACE, "response from upstream server %d bytes:\n%.*s", reply_len, reply_len, reply);
        
        /* send the reply via ziti */
        int sent = 0;
        // int ziti_chunk_len = 32663; //maximum allowed by ziti?
        int ziti_chunk_len = 32000; 
        if (reply_len - sent < ziti_chunk_len){
                ziti_chunk_len = reply_len - sent;
            } 
        do {
            char *ziti_chunk = malloc(ziti_chunk_len+1);
            strncpy(ziti_chunk, reply+sent, ziti_chunk_len);
            ZITI_LOG(TRACE, "ziti_chunk to write %d bytes:\n%.*s", ziti_chunk_len, ziti_chunk_len, ziti_chunk);
            int rc = ziti_write(clt, (uint8_t *) ziti_chunk, ziti_chunk_len, ngx_ziti_downstream_on_client_write, ziti_chunk);
            sent+=ziti_chunk_len;
            ZITI_LOG(DEBUG, "ziti_write return code after writing %d of %d bytes: %d", sent, reply_len, rc);
            if (reply_len - sent < ziti_chunk_len){
                ziti_chunk_len = reply_len - sent;
            }
        } while (sent < reply_len);
        free(reply);
    }
    else if (len == ZITI_EOF) {
        ZITI_LOG(DEBUG, "client disconnected");
        ziti_close(clt, NULL);
    }
    else {
        ZITI_LOG(DEBUG, "error: %zd(%s)", len, ziti_errorstr(len));
    }
    return len;
}

Thanks,
Olaf

I ran into an error when the size of the data parameter exceeded 32663 ... Is that a known restriction?

I don't think it's anything known but MTU's can complicate things when the packets are that big. What's the MTU on your interface? There's some bytes of overhead on any ziti packet which might be variable - I expect some of that is the reason for you 'odd' max size you're seeing (32663 bytes)

All the references to MTU I've found in our code are even - 16k, 32k, 64k etc. I'll ask some of the other fellas who have more experience with bigger packets like this and see if I can get an answer.

the MTU size on the ziti interface:

utun8: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> mtu 4000
	options=6403<RXCSUM,TXCSUM,CHANNEL_IO,PARTIAL_CSUM,ZEROINVERT_CSUM>
	inet 100.64.0.1 --> 100.64.0.1 netmask 0xffc00000

there is a limit imposed by the fabric - it splits large(over 64K) payload messages into multiple. this causes end-to-end encryption fail. C-SDK should handle that (does not right now: Better payload handling · Issue #279 · openziti/ziti-sdk-c · GitHub).

That said, this is another bug in underlying transport library: uv_mbed_write() fails to flush outbound SSL buffer on large payloads · Issue #87 · netfoundry/uv-mbed · GitHub

2 Likes

Release v0.23.0 · openziti/ziti-sdk-c · GitHub fixes the second issue from above.

Olaf, can you try with latest release, please?

just tried, wasn't successful.
set ziti max chunk size to 64k
#define ZITI_MAX_CHUNK_SIZE 65536

starting nginx:
make && make install && ZITI_LOG=4 /usr/local/nginx/sbin/nginx [ 0.004] INFO ../library/ziti.c:230 ziti_init_async() ztx[0] Ziti C SDK version 0.23.0.8 @49e7442(main) starting at (2021-05-11T14:55:38.900) [ 0.004] INFO ../library/ziti.c:231 ziti_init_async() ztx[0] Loading from config[/Users/oluetkeh/Developer/ngx_ziti_downstream/work/ngx_downstream.json] controller[https://18.210.117.20:443] [ 0.004] DEBUG ../library/ziti.c:251 ziti_init_async() ztx[0] using metrics interval: 6

requesting file
`
curl -v --max-time 2 http://odoo.ngx.ziti:80/web/content/407-a5d4a6a/web.assets_common.css

  • Trying 100.64.2.3...
  • TCP_NODELAY set
  • Connected to odoo.ngx.ziti (100.64.2.3) port 80 (#0)

GET /web/content/407-a5d4a6a/web.assets_common.css HTTP/1.1
Host: odoo.ngx.ziti
User-Agent: curl/7.64.1
Accept: /

  • Empty reply from server
  • Connection #0 to host odoo.ngx.ziti left intact
    curl: (52) Empty reply from server
  • Closing connection 0
    `

log snippet
[ 10.213] DEBUG /Users/oluetkeh/Developer/ngx_ziti_downstream/src/ngx_ziti_downstream_module.c:275 ngx_ziti_downstream_on_client() incoming connection from 'Luetkehoelter.Olaf.marketing' [ 10.213] DEBUG /Users/oluetkeh/Developer/ngx_ziti_downstream/src/ngx_ziti_downstream_module.c:281 ngx_ziti_downstream_on_client() got app data: 122 bytes [ 10.213] DEBUG ../library/channel.c:195 ziti_channel_add_receiver() ch[0] added receiver[1] [ 10.222] DEBUG /Users/oluetkeh/Developer/ngx_ziti_downstream/src/ngx_ziti_downstream_module.c:291 ngx_ziti_downstream_on_client_connect() --- ngx_ziti_downstream_on_client_connect [ 10.246] DEBUG /Users/oluetkeh/Developer/ngx_ziti_downstream/src/ngx_ziti_downstream_module.c:302 ngx_ziti_downstream_on_client_data() client sent 122 bytes [ 10.246] DEBUG /Users/oluetkeh/Developer/ngx_ziti_downstream/src/ngx_ziti_downstream_module.c:310 ngx_ziti_downstream_on_client_data() response from upstream server is 146451 bytes long. [ 10.246] DEBUG /Users/oluetkeh/Developer/ngx_ziti_downstream/src/ngx_ziti_downstream_module.c:325 ngx_ziti_downstream_on_client_data() ziti_write return code after writing 65536 of 146451 bytes: 0 [ 10.246] DEBUG /Users/oluetkeh/Developer/ngx_ziti_downstream/src/ngx_ziti_downstream_module.c:325 ngx_ziti_downstream_on_client_data() ziti_write return code after writing 131072 of 146451 bytes: 0 [ 10.246] DEBUG /Users/oluetkeh/Developer/ngx_ziti_downstream/src/ngx_ziti_downstream_module.c:325 ngx_ziti_downstream_on_client_data() ziti_write return code after writing 146451 of 146451 bytes: 0 [ 10.315] DEBUG /Users/oluetkeh/Developer/ngx_ziti_downstream/src/ngx_ziti_downstream_module.c:343 ngx_ziti_downstream_on_client_write() --- ngx_ziti_downstream_on_client_write status: 65536 [ 10.315] DEBUG /Users/oluetkeh/Developer/ngx_ziti_downstream/src/ngx_ziti_downstream_module.c:343 ngx_ziti_downstream_on_client_write() --- ngx_ziti_downstream_on_client_write status: 65536 [ 10.338] DEBUG /Users/oluetkeh/Developer/ngx_ziti_downstream/src/ngx_ziti_downstream_module.c:343 ngx_ziti_downstream_on_client_write() --- ngx_ziti_downstream_on_client_write status: 15379 [ 20.154] DEBUG ../library/posture.c:186 ziti_send_posture_data() posture checks either never sent or session changed, must_send = true [ 20.154] DEBUG ../library/posture.c:472 ziti_pr_send_bulk() no change in posture data, not sending [ 20.189] ERROR ../library/channel.c:670 connect_timeout() ch[1] connect timeout [ 20.189] INFO ../library/channel.c:720 reconnect_channel() ch[1] reconnecting in 7896 ms (attempt = 1) [ 20.189] ERROR ../library/channel.c:831 on_channel_connect_internal() ch[1] failed to connect [-89/operation canceled] [ 20.189] INFO ../library/channel.c:720 reconnect_channel() ch[1] reconnecting in 8130 ms (attempt = 2) [ 28.321] DEBUG ../library/channel.c:699 reconnect_cb() ch[1] connecting to 192.168.123.119:443 2021/05/11 10:56:08 [notice] 5289#1673376: signal 2 (SIGINT) received from 38715, exiting