Hello everyone,
I'm having an issue using proxy mode with Zrok. I've created a tunnel to a certain location, and I'm using curl to send a HTTP POST request to that location. When I try to perform a request, this is the output I'm getting. Could anyone tell me why is the error happening and how can I resolve it so I can get back the desired response?
Many thanks in advance.
│} │
│looking up listener │
│[ 11.249] DEBUG sdk-golang/ziti/edge/network.(*edgeConn).newChildConnection: {token=[6c04e5b7-0525-4553-8666-fc0f32ad7713] connId=[1]│
│} │
│listener found. checking for router provided connection id │
│[ 11.250] DEBUG sdk-golang/ziti/edge/network.(*edgeConn).newChildConnection: {connId=[1] token=[6c04e5b7-0525-4553-8666-fc0f32ad7713]│
│} │
│using router provided connection id 2147483649 │
│[ 11.253] DEBUG sdk-golang/ziti/edge/network.(*edgeConn).newChildConnection: {circuitId=[6c04e5b7-0525-4553-8666-fc0f32ad7713] │
│marker=[XvD6GHr-] connId=[2147483649] parentConnId=[1] token=[6c04e5b7-0525-4553-8666-fc0f32ad7713]} setting up crypto │
│[ 11.255] DEBUG sdk-golang/ziti/edge/network.(*newConnHandler).dialSucceeded: {marker=[XvD6GHr-] parentConnId=[1] token=[6c04e5b7 │
│-0525- │
│4553-8666-fc0f32ad7713] connId=[2147483649]} new connection established │
│[ 11.258] DEBUG sdk-golang/ziti/edge/network.(*newConnHandler).dialSucceeded: {connId=[2147483649] marker=[XvD6GHr-] parentConnId=[1]│
│token=[6c04e5b7-0525-4553-8666-fc0f32ad7713]} sending crypto header │
│[ 11.260] DEBUG sdk-golang/ziti/edge.(*MsgChannel).WriteTraced: {connId=[2147483649] type=[EdgeDataType] chSeq=[-1] edgeSeq=[1]} │
│writing 24 bytes │
│[ 11.263] DEBUG sdk-golang/ziti/edge/network.(*newConnHandler).dialSucceeded: {connId=[2147483649] marker=[XvD6GHr-] parentConnId=[1]│
│token=[6c04e5b7-0525-4553-8666-fc0f32ad7713]} tx crypto established │
│[ 11.396] DEBUG sdk-golang/ziti/edge/network.(*edgeConn).Accept: {type=[EdgeDataType] chSeq=[8] edgeSeq=[1] connId=[2147483649]} │
│received 24 bytes (msg type: 60786) │
│[ 11.451] DEBUG sdk-golang/ziti/edge/network.(*edgeConn).Accept: {edgeSeq=[2] connId=[2147483649] type=[EdgeDataType] chSeq=[9]} │
│received 411 bytes (msg type: 60786) │
│[ 11.451] DEBUG sdk-golang/ziti/edge/network.(*edgeConn).Accept: {chSeq=[10] edgeSeq=[3] connId=[2147483649] type=[EdgeDataType]} │
│received 135 bytes (msg type: 60786) │
│[ 11.455] DEBUG sdk-golang/ziti/edge/network.(*edgeConn).Read: {connId=[2147483649] marker=[XvD6GHr-]} reading 394 bytes │
│[ 11.458] DEBUG sdk-golang/ziti/edge/network.(*edgeConn).Read: {connId=[2147483649] marker=[XvD6GHr-]} reading 118 bytes │
│[ 11.474] DEBUG sdk-golang/ziti/edge.(*MsgChannel).WriteTraced: {chSeq=[-1] edgeSeq=[2] connId=[2147483649] type=[EdgeDataType]} │
│writing 490 bytes │
│[ 11.474] DEBUG sdk-golang/ziti/edge/network.(*edgeConn).Read: {marker=[XvD6GHr-] connId=[2147483649]} unexpected sequencer err (read│
│timed out)
Hi @Fico125, welcome to the community and to zrok (and OpenZiti/BrowZer)!
unexpected sequencer err -- that's a new one for me! Are you able to share exactly the steps you have taken so that one of us can try to reproduce this problem locally? It looks to me like maybe you've discovered some sort of bug for us to work out.
If you can give us the steps to reproduce, that's exceedingly helpful!
Hi @TheLumberjack, thank you for the quick response, I appreciate it 
Oh my, I guess I've started on the wrong foot with Zrok..
The request that goes through Zrok looks like this:
POST /identity/connect/token HTTP/1.1
Host: some_host.com
Content-Type: application/x-www-form-urlencoded
Body:
client_id=some_client_id&client_secret=some_client_secret&username=some_username&password=some_password&grant_type=password
and the response that should be received is looking like this: (I've deleted some parts of the token just for security reasons)
Header:
Content-Type: application/json
Body:
{
"access_token": "eyJhbGciOiJIUzI1NiIsInR5cmllciI6ImZjbW5yaV9jbGllbnRJZCIsImh0dHA6Ly9zY2hlbWFzLm1pY3Jvc29mdC5jb20vssd3MvMjAwOC8wNi9pZGVudGl0eS9jbGFpbXMvYXV0aGVudGljYXRpb25tZXRob2QiOiJwYXNzd29yZCIsssImV4cCI6MTczMDc5NTYwMSwiaXNzIjoidG1zLndzcGF5IiwiYXVkIjoidG1zLndzcGF5In0.sdd-Mb6ZcjpD9PcwZFodRaREBevE9E",
"expires_in": 3600,
"token_type": "Bearer",
"refresh_token": "wU4E8YESWNINDUndus0MXoAknniadqVTr70SoXSUfod7eMHPYm0yE+lkYz0D3kteyXRWqQ=="
}
Is this enough information for you? If not please let me know what else could I provide you.
Hi @Fico125,
That's not quite what I was after. I was thinking more like what steps did you do, more like:
- i ran a simple python (or golang whatever) server:
python -m http.server 1234
- i ran these zrok commands to share the service/set things up
- i used golang/curl/postman/python like "this" to access the server and then "this" happened
I'd like to try to replicate the exact problem you're having. Being able to reproduce it locally will mean I/we will be able to debug what's going on.
Let me try again then:)
I ran the following to get zrok running:
- zrok enable mytoken
- zrok reserve public -n myname https://mypublicsite
- zrok share reserved --insecure myname
I have an device which is used to send a curl request like I've shown above to the public zrok endpoint, and when the request from the device was sent, the response from above was displayed.
Unfortunately, I can't give out more details on how exactly the device sends the request except for the curl/header/body part, as the system is closed and I don't have access to it.
So, I had the device send the curl POST request to zrok with the following header:
Content-Type: application/x-www-form-urlencoded
and the following body:
client_id=some_client_id&client_secret=some_client_secret&username=some_username&password=some_password&grant_type=password
and then I saw this in the zrok output window:
│} │
│looking up listener │
│[ 11.249] DEBUG sdk-golang/ziti/edge/network.(*edgeConn).newChildConnection: {token=[6c04e5b7-0525-4553-8666-fc0f32ad7713] connId=[1]│
│} │
│listener found. checking for router provided connection id │
│[ 11.250] DEBUG sdk-golang/ziti/edge/network.(*edgeConn).newChildConnection: {connId=[1] token=[6c04e5b7-0525-4553-8666-fc0f32ad7713]│
│} │
│using router provided connection id 2147483649 │
│[ 11.253] DEBUG sdk-golang/ziti/edge/network.(*edgeConn).newChildConnection: {circuitId=[6c04e5b7-0525-4553-8666-fc0f32ad7713] │
│marker=[XvD6GHr-] connId=[2147483649] parentConnId=[1] token=[6c04e5b7-0525-4553-8666-fc0f32ad7713]} setting up crypto │
│[ 11.255] DEBUG sdk-golang/ziti/edge/network.(*newConnHandler).dialSucceeded: {marker=[XvD6GHr-] parentConnId=[1] token=[6c04e5b7 │
│-0525- │
│4553-8666-fc0f32ad7713] connId=[2147483649]} new connection established │
│[ 11.258] DEBUG sdk-golang/ziti/edge/network.(*newConnHandler).dialSucceeded: {connId=[2147483649] marker=[XvD6GHr-] parentConnId=[1]│
│token=[6c04e5b7-0525-4553-8666-fc0f32ad7713]} sending crypto header │
│[ 11.260] DEBUG sdk-golang/ziti/edge.(*MsgChannel).WriteTraced: {connId=[2147483649] type=[EdgeDataType] chSeq=[-1] edgeSeq=[1]} │
│writing 24 bytes │
│[ 11.263] DEBUG sdk-golang/ziti/edge/network.(*newConnHandler).dialSucceeded: {connId=[2147483649] marker=[XvD6GHr-] parentConnId=[1]│
│token=[6c04e5b7-0525-4553-8666-fc0f32ad7713]} tx crypto established │
│[ 11.396] DEBUG sdk-golang/ziti/edge/network.(*edgeConn).Accept: {type=[EdgeDataType] chSeq=[8] edgeSeq=[1] connId=[2147483649]} │
│received 24 bytes (msg type: 60786) │
│[ 11.451] DEBUG sdk-golang/ziti/edge/network.(*edgeConn).Accept: {edgeSeq=[2] connId=[2147483649] type=[EdgeDataType] chSeq=[9]} │
│received 411 bytes (msg type: 60786) │
│[ 11.451] DEBUG sdk-golang/ziti/edge/network.(*edgeConn).Accept: {chSeq=[10] edgeSeq=[3] connId=[2147483649] type=[EdgeDataType]} │
│received 135 bytes (msg type: 60786) │
│[ 11.455] DEBUG sdk-golang/ziti/edge/network.(*edgeConn).Read: {connId=[2147483649] marker=[XvD6GHr-]} reading 394 bytes │
│[ 11.458] DEBUG sdk-golang/ziti/edge/network.(*edgeConn).Read: {connId=[2147483649] marker=[XvD6GHr-]} reading 118 bytes │
│[ 11.474] DEBUG sdk-golang/ziti/edge.(*MsgChannel).WriteTraced: {chSeq=[-1] edgeSeq=[2] connId=[2147483649] type=[EdgeDataType]} │
│writing 490 bytes │
│[ 11.474] DEBUG sdk-golang/ziti/edge/network.(*edgeConn).Read: {marker=[XvD6GHr-] connId=[2147483649]} unexpected sequencer err (read│
│timed out)
Ok thank you for that. That does indeed give me a more clear picture. I don't know why, but I thought you were using the zrok sdk for some reason - i was way, way, wayyy off! I appreciate you bearing with me.
So just as test, can you replicate the issue with any other site? is it jsut a problem with this particular site? Could the remote site be denying the traffic from zrok? When I read it again I noticed this:
read timed out
That makes me think that maybe the url isn't accessible from where the zrok share was run?
If you go to machine running the zrok share, are you able to access https://mypublicsite from there?
No worries
Unfortunately, I can't replicate the issue on some public webhook for example because then I do get the response as wanted, so the problem seems to be related strictly with https://mypublicsite.
I too thought that the issue could be network related, but the guys from my "Network team" are telling me that from their side everything works and isn't blocked.
If I try to send the request directly to https://mypublicsite using Postman, I can get the response just fine.
I'm running the zrok share from my own PC, just as I am running Postman.
My network skills aren't great, so if you could maybe tell me some more details which I could provide to my network guys, I would gladly do so. I'm thinking that maybe you can give me some more details about zrok network that would help them trace the issue if it really is on their side.
This is what I was looking for, thanks.
- So just so I understand, you run zrok locally and start the zrok share from your locally running machine.
- The zrok url that's produced is expected to be used by "the device".
- You are then mimicking "what the device will do" by running Postman/curl to the public zrok share URL, to be reasonably sure when you configure "the device" to use this URL it'll be expected to work fine.
- When you emulate the device using curl or postman, you get the read timeout issue.
If that all sounds right, please let me know. I think I understand properly.
Does this read timeout happen 100% of the time? Is it always reproducible? Are yo uwilling to DM me the url here on discourse and let me probe it/try it out or can you DM me and give me maybe a bit more information about the target service?
If it were me, I'd probably move next by installing Wireshark/tcpdump and I would capture the actual packets between your computer and https://mypublicsite to see if it shows me anything.
It still feels like something about the request is not translating properly to the remote machine, leading to it timing out but it's hard for me to have any input than that with the information i have.
I just tested something and will describe it here so that maybe you can help me understand the issue a bit better.
- I pointed Postman, which is replicating "my device" to the zrok address
- I pointed zrok to my locally hosted
https://mypublicsite , so to http://localhost:4999
- I ran the request from Postman, and I DID get the response I wanted but, I also did get this in the zrok console:
│} │
│looking up listener │
│[ 9.864] DEBUG sdk-golang/ziti/edge/network.(*edgeConn).newChildConnection: {connId=[1] token=[8c335b60-22b7-4a4f-9f13-b40713ac7fc9]│
│} │
│listener found. checking for router provided connection id │
│[ 9.867] DEBUG sdk-golang/ziti/edge/network.(*edgeConn).newChildConnection: {connId=[1] token=[8c335b60-22b7-4a4f-9f13-b40713ac7fc9]│
│} │
│using router provided connection id 2147483648 │
│[ 9.871] DEBUG sdk-golang/ziti/edge/network.(*edgeConn).newChildConnection: {connId=[2147483648] parentConnId=[1] marker=[RvTcmjBp] │
│token=[8c335b60-22b7-4a4f-9f13-b40713ac7fc9] circuitId=[8c335b60-22b7-4a4f-9f13-b40713ac7fc9]} setting up crypto │
│[ 9.873] DEBUG sdk-golang/ziti/edge/network.(*newConnHandler).dialSucceeded: {parentConnId=[1] token=[8c335b60-22b7-4a4f-9f13- │
│b40713ac7fc9] connId=[2147483648] marker=[RvTcmjBp]} new connection established │
│[ 9.875] DEBUG sdk-golang/ziti/edge/network.(*newConnHandler).dialSucceeded: {token=[8c335b60-22b7-4a4f-9f13-b40713ac7fc9] │
│connId=[2147483648] marker=[RvTcmjBp] parentConnId=[1]} sending crypto header │
│[ 9.877] DEBUG sdk-golang/ziti/edge.(*MsgChannel).WriteTraced: {type=[EdgeDataType] chSeq=[-1] edgeSeq=[1] connId=[2147483648]} │
│writing 24 bytes │
│[ 9.879] DEBUG sdk-golang/ziti/edge/network.(*newConnHandler).dialSucceeded: {marker=[RvTcmjBp] parentConnId=[1] token=[8c335b60 │
│-22b7- │
│4a4f-9f13-b40713ac7fc9] connId=[2147483648]} tx crypto established │
│[ 10.008] DEBUG sdk-golang/ziti/edge/network.(*edgeConn).Accept: {chSeq=[4] edgeSeq=[1] connId=[2147483648] type=[EdgeDataType]} │
│received 24 bytes (msg type: 60786) │
│[ 10.077] DEBUG sdk-golang/ziti/edge/network.(*edgeConn).Accept: {chSeq=[5] edgeSeq=[2] connId=[2147483648] type=[EdgeDataType]} │
│received 498 bytes (msg type: 60786) │
│[ 10.077] DEBUG sdk-golang/ziti/edge/network.(*edgeConn).Accept: {connId=[2147483648] type=[EdgeDataType] chSeq=[6] edgeSeq=[3]} │
│received 140 bytes (msg type: 60786) │
│[ 10.081] DEBUG sdk-golang/ziti/edge/network.(*edgeConn).Read: {connId=[2147483648] marker=[RvTcmjBp]} reading 481 bytes │
│[ 10.100] DEBUG sdk-golang/ziti/edge/network.(*edgeConn).Read: {connId=[2147483648] marker=[RvTcmjBp]} reading 123 bytes │
│[ 11.949] DEBUG sdk-golang/ziti/edge.(*MsgChannel).WriteTraced: {connId=[2147483648] type=[EdgeDataType] chSeq=[-1] edgeSeq=[2]} │
│writing 716 bytes │
│[ 11.950] DEBUG sdk-golang/ziti/edge/network.(*edgeConn).Read: {connId=[2147483648] marker=[RvTcmjBp]} unexpected sequencer err (read│
│timed out)
which again shows unexpected sequencer err, but what's confusing me is that I indeed did get the response back to Postman.
I think I have to investigate network issues further. I'd like to please ask you if you could tell me how I could find out the network address or some network details from where the zrok is trying to send the payload to https://mypublicsite