Bad Gateway - Self Hosted

zrok
21

One of our users shared their logs files for our app and I see this inside of their log files.

tor_server-1                                 | Dec 01 18:36:37.588 [warn] Unparseable address in hidden service port configuration.
tor_server-1                                 | Dec 01 18:36:37.588 [warn] Failed to parse/validate config: Failed to configure rendezvous options. See logs for details.
tor_server-1                                 | Dec 01 18:36:37.588 [err] Reading config failed--see warnings above.
tor_server-1                                 | Dec 01 18:37:37.762 [notice] Tor 0.4.7.8 running on Linux with Libevent 2.1.12-stable, OpenSSL 1.1.1n, Zlib 1.2.11, Liblzma N/A, Libzstd N/A and Glibc 2.31 as libc.
tor_server-1                                 | Dec 01 18:37:37.762 [notice] Tor can't help you if you use it wrong! Learn how to be safe at 

tor_server-1                                 | Dec 01 18:37:37.762 [notice] Read configuration file "/tmp/torrc".
tor_server-1                                 | Dec 01 18:37:37.772 [warn] Unparseable address in hidden service port configuration.
tor_server-1                                 | Dec 01 18:37:37.772 [warn] Failed to parse/validate config: Failed to configure rendezvous options. See logs for details.
tor_server-1                                 | Dec 01 18:37:37.772 [err] Reading config failed--see warnings above.
mintsolo-platform-mintsolo_zrok-share_1      | [   0.196]    INFO main.(*shareReservedCommand).shareLocal: sharing target: 'http://server:3068'
mintsolo-platform-mintsolo_zrok-share_1      | [   0.196]    INFO main.(*shareReservedCommand).shareLocal: using existing backend target: http://server:3068
mintsolo-platform-mintsolo_zrok-share_1      | [ERROR]: unable to create 'proxy' backend (error listening: failed to listen: no apiSession, authentication attempt failed: Post "https://ziti.shared.mintsolo.app:80/edge/client/v1/authenticate?method=cert": context deadline exceeded (Client.Timeout exceeded while awaiting headers))
mintsolo-platform-mintsolo_zrok-share_1      | WARNING: STATE_DIRECTORY is undefined. Using HOME=/mnt
mintsolo-platform-mintsolo_zrok-share_1      | DEBUG: zrok state directory is /mnt/.zrok
mintsolo-platform-mintsolo_zrok-share_1      | DEBUG: ZROK_SHARE_RESERVED=true
mintsolo-platform-mintsolo_zrok-share_1      | INFO: zrok backend is already reserved: l3c6y9zlv87b
mintsolo-platform-mintsolo_zrok-share_1      | INFO: running: zrok share reserved l3c6y9zlv87b --override-endpoint http://server:3068 --headless
mintsolo-platform-mintsolo_zrok-share_1      | [   0.193]    INFO main.(*shareReservedCommand).shareLocal: sharing target: 'http://server:3068'
mintsolo-platform-mintsolo_zrok-share_1      | [   0.193]    INFO main.(*shareReservedCommand).shareLocal: using existing backend target: http://server:3068
mintsolo-platform-mintsolo_zrok-share_1      | [ERROR]: unable to create 'proxy' backend (error listening: failed to listen: no apiSession, authentication attempt failed: Post "https://ziti.shared.mintsolo.app:80/edge/client/v1/authenticate?method=cert": context deadline exceeded (Client.Timeout exceeded while awaiting headers))
mintsolo-platform-mintsolo_zrok-share_1      | DEBUG: zrok state directory is /mnt/.zrok
mintsolo-platform-mintsolo_zrok-share_1      | DEBUG: ZROK_SHARE_RESERVED=true

From those logs, it appears that we may have erred in creating our server: container due to the following error on their system.

tor_server-1                                 | Dec 01 18:36:37.588 [err] Reading config failed--see warnings above.
22

This right here:

unable to create 'proxy' backend (error listening: failed to listen: no apiSession, authentication attempt failed: Post "https://ziti.shared.mintsolo.app:80/edge/client/v1/authenticate?method=cert": context deadline exceeded (Client.Timeout exceeded while awaiting headers))

I don’t think your ziti controller is accessible to whatever that logging came from… it looks like a zrok share process.

Your zrok share is not able to communicate with the OpenZiti network.

Is it possible that it worked “internally” for testing because your OpenZiti controller is on the same network as the rest of your testing setup, and your external clients cannot reach it?

23

We run our hosted ZROK Service on a VPS. So even within our development environment/testing lab, we are accessing an external network.

However, we believe we may have found the issue that may be related to port 3022 not being available to the public WAN.

Some of our initial testing suggests that opening this port has resolved some of the issues. I’ll monitor for a few days and then provide an update on this issue.

24

That'd definitely make sense to me. Your zrok clients/components wouldn't be able to connect to a router and that would mean nothing would be able to successfully create terminator on any routers and that would lead to the "has no terminators" error.

FWIW the OpenZiti ziti CLI has a command called ziti ops verify traffic. Login to your controller from a remote machine (running on the same machine as the controller can make the test pass when it shouldn't) and then run that comand. you should see something like this:

image
image925×670 106 KB

make sure it succeeds. :slight_smile: I expect it would now.