Newbie no api authentication failed

zrok
1

I'm kind of stumped as to why my Zrok instance does not work so looking for assistance n correcting my thinking. I have a DNS zone, share.domain, and i zrok on a ubuntu box where the hostname is zrok.share.domain. The zrok host is nat'ed behind a the firewall and is using a self hosted instance as per the instructions Self-hosting guide for Docker | Zrok. Is the firewall configuration on the page implying that I need to open these ports to the internet for incoming traffic? It does say that they publish which implies as outgoing traffic i do not need to open incoming holes? My local machine and running ss -lntp i see ports 12080, 3022 on 0.0.0.0 and 18080, 8080, 8081 on localhost (127.0.0.1). I can nslookup ziti.share.domain and it resolves to my wan address. If I try and share files with 'zrok share public --backend-mode web .' i get an error "running http web backend: loading new config: http app module: start: listening │
│on zrok/0765xy5phd1c:0: failed to listen: no apiSession, authentication attempt failed: Post │
│"https://ziti.share.domain:1280/edge/client/v1/authenticate?method=cert": context deadline exceeded (Client.Timeout exceeded while│
│awaiting headers)"

If i look at the logs the only interesting one is the ziti quickstart which appears to have dns certificate issues as below but i am not sure if this is my issue. The host is a clean ubuntu instance specifically for this.

Version of zrok is v0.4.41.

Can anyone suggest what I have wrong?

[ 8.014] INFO ziti/tunnel/dns.NewDnsServer: starting dns server...
[ 8.014] INFO ziti/router/xgress_edge.(*CertExpirationChecker).Run: waiting 8591h59m59.126432325s to renew certificates
[ 8.827] ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:1280]: {remote=[172.19.0.3:53340] error=[remote error: tls: bad certificate]} handshake failed
[ 8.997] INFO ziti/router/state.(*apiSessionAddedHandler).applySync: finished synchronizing api sessions [count: 1, syncId: cm26skcuf000401rsq15el7d7, duration: 27.081µs]
[ 10.016] INFO ziti/tunnel/dns.NewDnsServer: dns server running at 127.0.0.1:53
[ 10.016] INFO ziti/tunnel/dns.(*resolver).AddHostname: adding ziti-tunnel.resolver.test = 19.65.28.94 to resolver
[ 10.030] ERROR ziti/tunnel/dns.NewDnsServer: system resolver test failed: failed to resolve ziti-tunnel.resolver.test: lookup ziti-tunnel.resolver.test on 127.0.0.11:53: no such host

ziti-tunnel runs an internal DNS server which must be first in the host's
resolver configuration. On systems that use NetManager/dhclient, this can
be achieved by adding the following to /etc/dhcp/dhclient.conf:

prepend domain-name-servers 127.0.0.1:53;

[ 10.030] INFO ziti/tunnel/intercept.SetDnsInterceptIpRange: dns intercept IP range: 100.64.0.1 - 100.127.255.255

2

Hi @tommiyau, welcome to the community and to zrok (and OpenZiti and browzer)!

Nat'ed meaning the router is forwarding traffic from "external_ip:port" to "internal_ip:port", right? The firewall isn't trying to terminate TLS is it?

Your zrok frontend and OpenZiti controller and one OpenZiti edge router will need to be exposed to the internet for incoming traffic, yes.

error=[remote error: tls: bad certificate]} handshake failed

These are often what you'll see when you log into the controller using a username/password. I think zrok will do that, so these errors are "probably" ok.

system resolver test failed: failed to resolve ziti-tunnel.resolver.test: lookup ziti-tunnel.resolver.test on 127.0.0.11:53: no such host

This error happens when the OpenZiti router starts up and tries to determine if it would be available to intercept traffic. I don't think this is relevant or causing issues with self-hosting zrok.

It looks to me like the controller is not exposed to the internet correctly or that the controller is not configured with the proper external DNS entry. That's my guess.

The easiest way to test my hypothesis is to test your overlay network by using the ziti CLI. Run the recently released with ziti v1.1.9+.

$ ziti ops verify-traffic --help
A tool to verify traffic can flow over the overlay properly. You must be authenticated to use this tool.

Usage:
  ziti ops verify-traffic [flags]

Flags:
      --allow-multiple-servers   Whether to allows the same server multiple times.
      --cleanup                  Whether to perform cleanup.
  -h, --help                     help for verify-traffic
      --host string              the controller host
  -m, --mode string              [optional, default 'both'] The mode to perform: server, client, both.
  -p, --password string          password to use for authenticating to the Ziti Edge Controller, if -u is supplied and -p is not, a value will be prompted for
      --port string              the controller port
  -x, --prefix string            [optional] The prefix to apply to generated objects, necessary when not using the 'both' role.
  -u, --username string          username to use for authenticating to the Ziti Edge Controller
      --verbose                  Show additional output.

Pass the host, port, username and password and verify you see no errors. for example, after i started a local instance, i ran:

ziti ops verify-traffic --host localhost --port 1280 --username admin --password admin

results below

image
image1380×439 28.1 KB

Let's make sure your overlay is setup properly. Make sure you pass the externally addressable dns name and port

3

Great thanks. Having never hard zrok run previously I'm not sure what is correct and what maybe ignore so your response is terrific as it gives me some where to focus on.

To answer your question yes there is port forwarding in place and the firewall is not terminating the TLS.

I'll try this asap later today. Thanks alot, much appreciated.

4

ahh by passed work figured i can get to it...

I'm going to make the assumption that the username and pw is that of zrok and the hostname is ziti on the share? I hope that is correct?

I did a tcpdump at the same time on port 1280 comming into the linux box and i got to see nothing so I am assuming that the forwarding is not working....? Is that correct?

ziti ops verify-traffic --host ziti.share.domain --port 1280 --username <zrok_username> --password <zrok_pw>
WARNING no prefix and mode is not 'both'. default prefix of 2024-10-12-1912 will be used
INFO connecting with user <zrok_username> to https://ziti.share.domain:1280
FATAL Get "https://ziti.share.domain:1280/.well-known/est/cacerts": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

5

actually i just went and looked at the firewall when running the ziti ops command and there is nothing trying to hit port 1280. yet i can ping ziti.share.domain successfully and nslookup resolves it to the correct address. So now I am scratching my head.....

6

To test the OpenZiti overlay using ziti ops verify-traffic, you need to know the url of the controller and the port of the controller. The default port will be 1280, yes. The username and password is whatever the OpenZiti Controller username and password are. Looking at that page you referenced, I would expect it to be this value:

ZITI_PWD=whatever

The username is likely admin unless you changed ZITI_USER.

Pinging the domain is not the same as hitting port 1280. Based on FATAL Get "https://ziti.share.domain:1280/.well-known/est/cacerts": context deadline exceeded, I'm certain you don't have the controller exposed correctly.

You will need https://ziti.share.domain:1280 to resolve to your controller. You need to expose your controller on some url and you need to make sure it's resolvable.

For example I have a demo zrok located at https://api.clint.demo.openziti.org:8445/ and my controller is at https://ctrl.clint.demo.openziti.org:8441/

7

yes does appear to be some thing wrong there as i have tried a curl from a different network to port 1280 and it resolves to the right address but yet I am not seeing it on the firewall.....interesting....almost like it is blocked else where which explains some thing....not sure what. I'll work on this thanks alot for your time and knowl;edge much appreciated.

8

I have found the issue and my Service Provider thanks you. Seems that there is a bug in there service and for some strange reason i was not receiving packets I should have been getting....rather weird and odd but to cut the long story short my zrok now works. PS. never seen anything like it as it was only traffic incoming as well......maybe zero trust :wink: