Threading with uv_queue_work()

SDK Questions sdk-c
1

per Eugene’s earlier guidance, I went ahead and implemented processing the upstream request in a separate thread (i.e. not on the event loop) and sending the response back via ziti transport on the event loop. The latest code uses the invocation uv_queue_work(uv_thread_loop, work, process_upstream, respond_to_client);

I have verified via debugger that processing of the function process_upstream(uv_work_t *work) occurs on a separate thread. My problem at this time is that the 2nd callback respond_to_client(uv_work_t *work, int status), i.e. the ‘after processing’ is never invoked. Would you have any pointers as to what I can investigate next to fix this?

Here’s relevant detail from the logs:

[        4.583] VERBOSE ../library/connect.c:781 conn_inbound_data_msg() conn[0.1] decrypting 94 bytes
[        4.583] VERBOSE ../library/connect.c:785 conn_inbound_data_msg() conn[0.1] decrypted 77 bytes
[        4.583]   TRACE ../library/connect.c:736 flush_to_client() conn[0.1] flushing 77 bytes to client
[        4.583]   DEBUG /Users/oluetkeh/Developer/ngx_ziti_downstream/src/ngx_ziti_downstream_module.c:275 ngx_ziti_downstream_on_client_data() client sent 77 bytes
[        4.583]   TRACE /Users/oluetkeh/Developer/ngx_ziti_downstream/src/ngx_ziti_downstream_module.c:276 ngx_ziti_downstream_on_client_data() client sent these 77 bytes:
GET / HTTP/1.1
Host: odoo.ngx.ziti
User-Agent: curl/7.64.1
Accept: */*


[        4.584]   DEBUG /Users/oluetkeh/Developer/ngx_ziti_downstream/src/ngx_ziti_downstream_module.c:393 talk_to_upstream() ngx_ziti_downstream: request sent upstream. 77 bytes
[        4.584]   DEBUG /Users/oluetkeh/Developer/ngx_ziti_downstream/src/ngx_ziti_downstream_module.c:413 talk_to_upstream() ngx_ziti_downstream: response received from upstream. 552 bytes
[        4.584]   TRACE /Users/oluetkeh/Developer/ngx_ziti_downstream/src/ngx_ziti_downstream_module.c:414 talk_to_upstream() response from upstream server 552 bytes:
HTTP/1.0 303 SEE OTHER
Content-Type: text/html; charset=utf-8
Content-Length: 215
Location: http://odoo.ngx.ziti/web
Set-Cookie: session_id=d0b15c0e45759a222d3d106691062108a19231d5; Expires=Tue, 24-Aug-2021 15:30:47 GMT; Max-Age=7776000; HttpOnly; Path=/
Server: Werkzeug/0.16.1 Python/3.8.6
Date: Wed, 26 May 2021 15:30:47 GMT

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>Redirecting...</title>
<h1>Redirecting...</h1>
<p>You should be redirected automatically to target URL: <a href="/web">/web</a>.  If not click the link.
[        4.584]   DEBUG /Users/oluetkeh/Developer/ngx_ziti_downstream/src/ngx_ziti_downstream_module.c:308 process_upstream() finished upstream communication, got 552 bytes
[        5.544]   TRACE _deps/uv-mbed-src/src/tls_link.c:112 TLS(0x1005046c0)[2]: 100

thx
Olaf

2

Thanks for sharing the code… I took a look. Is it possible that the loop is exiting? I see a couple loops created and assigned to your zscf->uv_thread_loop.

I can see your log message at the end of the function so it seems like it should be firing that callback but if the loop is dying that might be why.

I might be wrong but this seems like just uv issue - not ziti so much, right?

3

Thanks for the response.
I’ve added a line after the dispatch (uv_queue_work(uv_thread_loop, work, process_upstream, respond_to_client);) to make sure the loop has not exited. With that one can see that the code (i.e. loop) continues after the call. Also, multiple requests are handled the same way, iow new client requests coming in are being handled.

Yes, the problem I’m facing looks definitely uv specific. I’d consider libuv an integral part of OpenZiti, though. I’ll look for guidance in other forums (too).

thx,
Olaf

4

I totally agree that libuv is strongly-tied with the C SDK. That’s why I was wondering if you are on teh proper loop or if one of your loops is exiting - or getting overwritten.

For me in the past most of my libuv issues were “the loop isn’t started”, “the loop exited”, or “you used the wrong loop”. Perhaps if you just use “default loop” for everything for now just to rule that out it might help? If i get a chance to look more - I’ll see if anything else strikes me and comment back.

5

Going forward I will do some more lower level debugging but probably look into implementing the threading piece without the use of uv_queue_work().

On that note, do you guys have any plans of hiding the uv loop interactions with ziti functions?

6

“plans” - no. You really need to understand that you’re running in a loop context. But we’ve talked a bit about adding something similar to async (like you see in Node.js). Would you find something like that useful?

7

Note: if you use one of the Ziti SDK wrapper libs, e.g., ziti-sdk-nodejs, or ziti-sdk-swift you don’t need to understand libuv (though all is still async of course)