Setting up Zrok (self hosting)

Support General Questions
1

Hello, I am testing zrok for my mqtt tunnelling application. my test setup is two hardware boards running on linux one will act as a server and other as a client(no public ip). I succesfully installed the controller and when i am trying to install the router at the bootstraping step i am getting the following error.

 sudo /opt/openziti/etc/router/bootstrap.bash
ERROR: something went wrong during bootstrapping; set DEBUG=1
WARN: set VERBOSE=1 or DEBUG=1 for more output
WARN: see output in '/tmp/tmp.3DO35AbZdG'

i dont know wether the token path i am giving is invalid or something else can any one help with what i am doing wrong here??

2

That sounds like it will work.

You'll have a Ziti controller and router running on Linux, then you can follow the zrok self hosting guide for Linux.

Look for clues in the log file mentioned in the output. Please re-run your command after export DEBUG=1 and share (or DM if it contains anything confidential) the output.

3

{"endpoint":"tls:raspi.local:3022","error":"error connecting ctrl (remote error: tls: internal error)","file":"github.com/openziti/ziti/router/env/

i have successfully started ziti router and controller but this error is being shown how can i solve this

4

I think the router's config.yml has the wrong address for the controller in ctrl.endpoint. Try "raspi.local:1280" to match the port where the controller is listening for the router to call.

1 Like
5

image
image1246×205 11.3 KB

this is the error now iam getting the service is not running now

6

Why didn't the router start? Please check the router's log.

journalctl -lfu ziti-router.service

If it's unclear why it didn't start, you can paste the error messages here in Discourse as formatted text like this.

```text
this is some formatted text
```
7 
Sep 13 10:31:42 raspberrypi systemd[1]: ziti-router.service: Scheduled restart job, restart counter is at 15860.
Sep 13 10:31:42 raspberrypi systemd[1]: Stopped ziti-router.service - OpenZiti Router.
Sep 13 10:31:42 raspberrypi systemd[1]: Starting ziti-router.service - OpenZiti Router...
Sep 13 10:31:42 raspberrypi entrypoint.bash[312421]: WARN: set VERBOSE=1 or DEBUG=1 for more output
Sep 13 10:31:42 raspberrypi entrypoint.bash[312421]: WARN: see output in '/tmp/tmp.t2U8UV6oKh'
Sep 13 10:31:42 raspberrypi systemd[1]: Started ziti-router.service - OpenZiti Router.
Sep 13 10:31:42 raspberrypi ziti[312426]: {"arch":"arm64","build-date":"2024-07-16T13:15:40Z","configFile":"config.yml","file":"github.com/openziti/ziti/ziti/router/run.go:71","func":"github.com/openziti/ziti/ziti/router.run","go-version":"go1.22.5","level":"info","msg":"starting ziti router","os":"linux","revision":"94013fe4af89","routerId":"b8q4XvfJg","time":"2024-09-13T10:31:42.755Z","version":"v1.1.7"}
Sep 13 10:31:42 raspberrypi ziti[312426]: {"file":"github.com/openziti/ziti/common/metrics/pool_metrics.go:50","func":"github.com/openziti/ziti/common/metrics.ConfigureGoroutinesPoolMetrics.GoroutinesPoolMetricsConfigF.func1.1","idleTime":30000000000,"level":"info","maxQueueSize":1000,"maxWorkers":32,"minWorkers":0,"msg":"starting goroutine pool","poolType":"pool.link.dialer","time":"2024-09-13T10:31:42.756Z"}
Sep 13 10:31:42 raspberrypi ziti[312426]: {"file":"github.com/openziti/ziti/router/forwarder/faulter.go:78","func":"github.com/openziti/ziti/router/forwarder.(*Faulter).run","level":"info","msg":"started","time":"2024-09-13T10:31:42.756Z"}
Sep 13 10:31:42 raspberrypi ziti[312426]: {"file":"github.com/openziti/ziti/router/forwarder/scanner.go:52","func":"github.com/openziti/ziti/router/forwarder.(*Scanner).run","level":"info","msg":"started","time":"2024-09-13T10:31:42.756Z"}
Sep 13 10:31:42 raspberrypi ziti[312426]: {"file":"github.com/openziti/ziti/common/metrics/pool_metrics.go:50","func":"github.com/openziti/ziti/common/metrics.ConfigureGoroutinesPoolMetrics.GoroutinesPoolMetricsConfigF.func1.1","idleTime":30000000000,"level":"info","maxQueueSize":1000,"maxWorkers":128,"minWorkers":0,"msg":"starting goroutine pool","poolType":"pool.route.handler","time":"2024-09-13T10:31:42.756Z"}
Sep 13 10:31:42 raspberrypi ziti[312426]: {"file":"github.com/openziti/ziti/common/metrics/pool_metrics.go:50","func":"github.com/openziti/ziti/common/metrics.ConfigureGoroutinesPoolMetrics.GoroutinesPoolMetricsConfigF.func1.1","idleTime":30000000000,"level":"info","maxQueueSize":1,"maxWorkers":50,"minWorkers":0,"msg":"starting goroutine pool","poolType":"pool.terminator_validation","time":"2024-09-13T10:31:42.757Z"}
Sep 13 10:31:42 raspberrypi ziti[312426]: {"file":"github.com/openziti/ziti/router/internal/edgerouter/config.go:154","func":"github.com/openziti/ziti/router/internal/edgerouter.(*Config).LoadConfigFromMap","level":"info","msg":"cached data model file set to: config.yml.json.gzip","time":"2024-09-13T10:31:42.757Z"}
Sep 13 10:31:42 raspberrypi ziti[312426]: {"file":"github.com/openziti/ziti/router/internal/edgerouter/config.go:171","func":"github.com/openziti/ziti/router/internal/edgerouter.(*Config).LoadConfigFromMap","level":"warning","msg":"Invalid heartbeat interval [0] (min: 60, max: 10), setting to default [60]","time":"2024-09-13T10:31:42.757Z"}
Sep 13 10:31:42 raspberrypi ziti[312426]: {"file":"github.com/openziti/ziti/router/router.go:346","func":"github.com/openziti/ziti/router.(*Router).showOptions","level":"info","msg":"ctrl = {\"OutQueueSize\":4,\"MaxQueuedConnects\":1,\"MaxOutstandingConnects\":16,\"ConnectTimeout\":5000000000,\"DelayRxStart\":false,\"WriteTimeout\":0}","time":"2024-09-13T10:31:42.758Z"}
Sep 13 10:31:42 raspberrypi ziti[312426]: {"file":"github.com/openziti/ziti/router/router.go:352","func":"github.com/openziti/ziti/router.(*Router).showOptions","level":"info","msg":"metrics = {\"ReportInterval\":60000000000,\"IntervalAgeThreshold\":0,\"MessageQueueSize\":10}","time":"2024-09-13T10:31:42.759Z"}
Sep 13 10:31:42 raspberrypi ziti[312426]: {"file":"github.com/openziti/ziti/common/metrics/pool_metrics.go:50","func":"github.com/openziti/ziti/common/metrics.ConfigureGoroutinesPoolMetrics.GoroutinesPoolMetricsConfigF.func1.1","idleTime":30000000000,"level":"info","maxQueueSize":5000,"maxWorkers":15,"minWorkers":0,"msg":"starting goroutine pool","poolType":"pool.rate_limiter","time":"2024-09-13T10:31:42.759Z"}
Sep 13 10:31:42 raspberrypi ziti[312426]: {"file":"github.com/openziti/ziti/router/router.go:660","func":"github.com/openziti/ziti/router.(*Router).initializeHealthChecks","level":"info","msg":"starting health check with ctrl ping initially after 15s, then every 30s, timing out after 15s","time":"2024-09-13T10:31:42.759Z"}
Sep 13 10:31:42 raspberrypi ziti[312426]: {"file":"github.com/openziti/ziti/router/router.go:481","func":"github.com/openziti/ziti/router.(*Router).startXlinkDialers","level":"info","msg":"started Xlink dialer with binding [transport]","time":"2024-09-13T10:31:42.760Z"}
Sep 13 10:31:42 raspberrypi ziti[312426]: {"addr":"tls:0.0.0.0:3022","error":"no network interface found for 0.0.0.0","file":"github.com/openziti/ziti/router/xlink_transport/config.go:76","func":"github.com/openziti/ziti/router/xlink_transport.loadListenerConfig","level":"warning","msg":"unable to get interface for address","time":"2024-09-13T10:31:42.761Z"}
Sep 13 10:31:42 raspberrypi ziti[312426]: {"file":"github.com/openziti/ziti/common/metrics/pool_metrics.go:50","func":"github.com/openziti/ziti/router/xlink_transport.(*listener).Listen.GoroutinesPoolMetricsConfigF.func1.1","idleTime":10000000000,"level":"info","maxQueueSize":1,"maxWorkers":16,"minWorkers":1,"msg":"starting goroutine pool","poolType":"pool.listener.link","time":"2024-09-13T10:31:42.761Z"}
Sep 13 10:31:42 raspberrypi ziti[312426]: {"file":"github.com/openziti/ziti/router/router.go:506","func":"github.com/openziti/ziti/router.(*Router).startXlinkListeners","level":"info","msg":"started Xlink listener with binding [transport] advertising [tls:192.168.1.11:3022]","time":"2024-09-13T10:31:42.762Z"}
Sep 13 10:31:42 raspberrypi ziti[312426]: {"address":{},"file":"github.com/openziti/ziti/router/xgress_edge/listener.go:87","func":"github.com/openziti/ziti/router/xgress_edge.(*listener).Listen","level":"info","msg":"starting channel listener","time":"2024-09-13T10:31:42.762Z"}
Sep 13 10:31:42 raspberrypi ziti[312426]: {"file":"github.com/openziti/ziti/common/metrics/pool_metrics.go:50","func":"github.com/openziti/ziti/router/xgress_edge.(*listener).Listen.GoroutinesPoolMetricsConfigF.func1.1","idleTime":10000000000,"level":"info","maxQueueSize":1,"maxWorkers":16,"minWorkers":1,"msg":"starting goroutine pool","poolType":"pool.listener.xgress_edge","time":"2024-09-13T10:31:42.763Z"}
Sep 13 10:31:42 raspberrypi ziti[312426]: {"file":"github.com/openziti/ziti/router/router.go:544","func":"github.com/openziti/ziti/router.(*Router).startXgressListeners","level":"info","msg":"created xgress listener [edge] at [tls:0.0.0.0:3022]","time":"2024-09-13T10:31:42.763Z"}
Sep 13 10:31:42 raspberrypi ziti[312426]: {"file":"github.com/openziti/ziti/router/router.go:544","func":"github.com/openziti/ziti/router.(*Router).startXgressListeners","level":"info","msg":"created xgress listener [tunnel] at []","time":"2024-09-13T10:31:42.763Z"}
Sep 13 10:31:42 raspberrypi ziti[312426]: {"file":"github.com/openziti/ziti/router/router.go:722","func":"github.com/openziti/ziti/router.(*Router).getInitialCtrlEndpoints","level":"info","msg":"controller endpoints file [endpoints] doesn't exist. Using initial endpoints from config","time":"2024-09-13T10:31:42.763Z"}
Sep 13 10:31:42 raspberrypi ziti[312426]: {"file":"github.com/openziti/ziti/router/router.go:555","func":"github.com/openziti/ziti/router.(*Router).startControlPlane","level":"info","msg":"router configured with 1 controller endpoints","time":"2024-09-13T10:31:42.763Z"}
Sep 13 10:31:42 raspberrypi ziti[312426]: {"file":"github.com/openziti/ziti/router/xgress_edge/accept.go:126","func":"github.com/openziti/ziti/router/xgress_edge.(*Acceptor).Run","level":"info","msg":"starting","time":"2024-09-13T10:31:42.763Z"}
Sep 13 10:31:42 raspberrypi ziti[312426]: {"endpoint":{"tls:raspi.local:1280":{}},"file":"github.com/openziti/ziti/router/env/ctrls.go:95","func":"github.com/openziti/ziti/router/env.(*networkControllers).UpdateControllerEndpoints","level":"info","msg":"adding new ctrl endpoint","time":"2024-09-13T10:31:42.764Z"}
Sep 13 10:31:42 raspberrypi ziti[312426]: {"endpoint":"tls:raspi.local:1280","file":"github.com/openziti/ziti/router/env/ctrls.go:134","func":"github.com/openziti/ziti/router/env.(*networkControllers).connectToControllerWithBackoff","level":"info","msg":"starting connection attempts","time":"2024-09-13T10:31:42.764Z"}
Sep 13 10:31:42 raspberrypi ziti[312426]: {"endpoint":"tls:raspi.local:1280","file":"github.com/openziti/ziti/router/env/ctrls.go:140","func":"github.com/openziti/ziti/router/env.(*networkControllers).connectToControllerWithBackoff.func3","level":"info","msg":"successfully connected to controller","time":"2024-09-13T10:31:42.887Z"}
Sep 13 10:31:42 raspberrypi ziti[312426]: {"file":"github.com/openziti/ziti/router/xgress_edge/factory.go:77","func":"github.com/openziti/ziti/router/xgress_edge.(*Factory).NotifyOfReconnect","level":"info","msg":"control channel reconnected, re-establishing hosted services","time":"2024-09-13T10:31:42.887Z"}
Sep 13 10:31:42 raspberrypi ziti[312426]: {"file":"github.com/openziti/ziti/router/xgress_edge_tunnel/factory.go:56","func":"github.com/openziti/ziti/router/xgress_edge_tunnel.(*Factory).NotifyOfReconnect","level":"info","msg":"control channel reconnected, re-establishing hosted services","time":"2024-09-13T10:31:42.887Z"}
Sep 13 10:31:42 raspberrypi ziti[312426]: {"ctrlId":"NetFoundry Inc. Client y4hgZwfqy","file":"github.com/openziti/ziti/router/link/link_registry.go:306","func":"github.com/openziti/ziti/router/link.(*linkRegistryImpl).NotifyOfReconnect","level":"info","msg":"resending link states after reconnect","time":"2024-09-13T10:31:42.887Z"}
Sep 13 10:31:42 raspberrypi ziti[312426]: {"file":"github.com/openziti/ziti/router/handler_edge_ctrl/hello.go:82","func":"github.com/openziti/ziti/router/handler_edge_ctrl.(*helloHandler).HandleReceive.func1","level":"info","msg":"received server hello, replying","time":"2024-09-13T10:31:42.889Z"}
Sep 13 10:31:42 raspberrypi ziti[312426]: {"file":"github.com/openziti/ziti/router/state/manager.go:604","func":"github.com/openziti/ziti/router/state.(*ManagerImpl).StartHeartbeat","level":"info","msg":"heartbeat starting","time":"2024-09-13T10:31:42.928Z"}
Sep 13 10:31:42 raspberrypi ziti[312426]: {"file":"github.com/openziti/ziti/router/xgress_edge_tunnel/tunneler.go:71","func":"github.com/openziti/ziti/router/xgress_edge_tunnel.(*tunneler).Start","level":"info","mode":"host","msg":"creating interceptor","time":"2024-09-13T10:31:42.928Z"}
Sep 13 10:31:42 raspberrypi ziti[312426]: {"file":"github.com/openziti/ziti/router/xgress_edge/certchecker.go:124","func":"github.com/openziti/ziti/router/xgress_edge.(*CertExpirationChecker).Run","level":"info","msg":"waiting 8263h11m51.071376858s to renew certificates","time":"2024-09-13T10:31:42.928Z"}
Sep 13 10:31:42 raspberrypi ziti[312426]: {"error":"exec: \"resolvectl\": executable file not found in $PATH","file":"github.com/openziti/ziti/tunnel/dns/server.go:49","func":"github.com/openziti/ziti/tunnel/dns.flushDnsCaches","level":"warning","msg":"unable to find systemd-resolve or resolvectl in path, consider adding a dns flush to your restart process","time":"2024-09-13T10:31:42.928Z"}
Sep 13 10:31:42 raspberrypi ziti[312426]: {"file":"github.com/openziti/ziti/tunnel/dns/server.go:89","func":"github.com/openziti/ziti/tunnel/dns.NewDnsServer","level":"info","msg":"starting dns server...","time":"2024-09-13T10:31:42.928Z"}
Sep 13 10:31:42 raspberrypi ziti[312426]: {"error":"dns server failed to start: listen udp 127.0.0.1:53: bind: permission denied","file":"github.com/openziti/ziti/router/xgress_edge_tunnel/tunneler.go:75","func":"github.com/openziti/ziti/router/xgress_edge_tunnel.(*tunneler).Start","level":"error","msg":"failed to start DNS resolver. using dummy resolver","time":"2024-09-13T10:31:42.929Z"}
Sep 13 10:31:42 raspberrypi ziti[312426]: {"file":"github.com/openziti/ziti/tunnel/dns/dummy.go:37","func":"github.com/openziti/ziti/tunnel/dns.NewDummyResolver","level":"warning","msg":"dummy resolver does not store hostname/ip mappings","time":"2024-09-13T10:31:42.929Z"}
Sep 13 10:31:42 raspberrypi ziti[312426]: {"file":"github.com/openziti/ziti/tunnel/intercept/iputils.go:51","func":"github.com/openziti/ziti/tunnel/intercept.SetDnsInterceptIpRange","level":"info","msg":"dns intercept IP range: 100.64.0.1 - 100.127.255.255","time":"2024-09-13T10:31:42.929Z"}
Sep 13 10:31:42 raspberrypi ziti[312426]: {"ctrlId":"NetFoundry Inc. Client y4hgZwfqy","error":"tunneling not enabled","file":"github.com/openziti/ziti/router/xgress_edge_tunnel/fabric.go:190","func":"github.com/openziti/ziti/router/xgress_edge_tunnel.(*fabricProvider).authenticate.func1","level":"error","msg":"failed to authenticate","time":"2024-09-13T10:31:42.930Z"}
Sep 13 10:31:42 raspberrypi ziti[312426]: {"error":"tunneling not enabled","file":"github.com/openziti/ziti/router/xgress_edge_tunnel/servicepoll.go:105","func":"github.com/openziti/ziti/router/xgress_edge_tunnel.(*servicePoller).pollServices","level":"fatal","msg":"xgress_edge_tunnel unable to authenticate to controller. ensure tunneler mode is enabled for this router or disable tunnel listener. exiting ","time":"2024-09-13T10:31:42.930Z"}
Sep 13 10:31:42 raspberrypi systemd[1]: ziti-router.service: Main process exited, code=exited, status=1/FAILURE
Sep 13 10:31:42 raspberrypi systemd[1]: ziti-router.service: Failed with result 'exit-code'.
8

The router failed to start because of a mismatch in the local and central configurations.

The local configuration requires tunneling to be enabled, but tunneling wasn't enabled when it was created. You can update the router to allow tunneling like this.

If the name of your router is "router1", then execute:

ziti edge update edge-router "router1" --tunneler-enabled
9

thank you @qrkourier and sorry for the trouble the router is up and running. I

10

hesy @qrkourier i stumbled upon something again . I think you can help me fix. While setting up zrok i setup both ZROK_ADMIN_TOKEN and ZROK_API_ENDPOINT but when i try to enable frontend using command zrok admin create frontend ma48lFMsV public http://{token}.raspi.local:18080 i got error [ERROR]: create frontend request failed ([POST /frontend][401] createFrontendUnauthorized) what might have possibly gone wrong??

11

It's probably the incorrect value for admin token variable because you received an active rejection from the correct API URL. Compare the value with the zrok controller configuration file.

If that's not it, then audit the precise syntax and order of the admin command against the built-in usage hints to ensure you're sending legal opts and args. It looks correct to me at a glance.

12

i exported a token using export ZROK_ADMIN_TOKEN="" is this the correct method or any other method is there to set admin toker or is it something that is provided by the programme??

13

I believe you are following the zrok self-hosting guide for Linux and have arrived at the "Create Frontend" step.

The problem you encountered is that the zrok admin command is configured for the correct API endpoint but is not configured with the correct API token.

The correct token is any string from the list admin.secrets in the zrok controller's configuration YAML file.

If you happened to follow the self-hosting guide precisely, then your zrok controller's config file is named etc/ctrl.yml.

Look in that file for the correct API token in the list named secrets in the admin section.

It looks like this in the configuration example:

admin:
  secrets:
    -                             77623cad-1847-4d6d-8ffe-37defc33c909

You must assign one of the values from the list named secrets to ZROK_ADMIN_TOKEN.

14

i tried that and it gave the same authorisation error. is it becuase i created the yml file??? i created this from the example given in the guide

15

It's unclear what's wrong from the info provided. Please verify the value of ZROK_ADMIN_TOKEN in the environment where you are running the zrok admin command matches the admin token in the zrok controller's configuration file.

(if /bin/grep -qE "\b${ZROK_ADMIN_TOKEN:-noop}\b" etc/ctrl.yml
then
    echo "OK: ZROK_ADMIN_TOKEN match"
else
    echo "FAIL: ZROK_ADMIN_TOKEN mismatch"
fi)

Note: I've enclosed the script in () to ensure we are testing for an exported value of ZROK_ADMIN_TOKEN (i.e., inherited by child processes of the current shell).

16 
#   |_  / '__/ _ \| |/ /
#    / /| | | (_) |   <
#   /___|_|  \___/|_|\_\
# controller configuration

v:                  4

admin:
  # generate these admin tokens from a source of randomness, e.g.
  #  LC_ALL=C tr -dc _A-Z-a-z-0-9 < /dev/urandom | head -c32
  secrets:
    -               Q8V0LqnNb5wNX9kE1fgQ0H6VlcvJybB1  # be sure to change this!

endpoint:
  host:             0.0.0.0
  port:             18080

invites:
  invites_open:    true

store:
  path:             zrok.db
  type:             sqlite3

ziti:
  api_endpoint:     "https://127.0.0.1:1280"
  username:         calixto_admin
  password:         calixto123

# you can use certbot to renew the wildcard cert for the controller with a DNS provider API token or front this `zrok` # controller with Caddy
#tls:
#  cert_path: "/Path/To/Cert/zrok.crt"
#  key_path:  "/Path/To/Cert/zrok.key"

this is my ctrl.yml which doesnt have an zrok_admin_token entry which is the sample config provided in the guide
i tried running the above shell script and it gave a token mismatch

17

OK, let's use that zrok controller configuration YAML example from the Linux self-hosting guide.

Suppose your zrok controller's API is available at http://zrok.example.com:18080, your admin token is Q8V0LqnNb5wNX9kE1fgQ0H6VlcvJybB1 (from the example), and bootstrapping the OpenZiti network resulted in a frontend identifier ma48lFMsV, and your zrok frontend will be available at http://*.zrok.example.com:8080.

Your admin shell must be configured with these values before running zrok admin commands, including the command to create or update a frontend.

ZROK_ADMIN_TOKEN='Q8V0LqnNb5wNX9kE1fgQ0H6VlcvJybB1' \
ZROK_API_ENDPOINT='http://zrok.example.com:18080' \
zrok admin create frontend "ma48lFMsV" public 'http://{token}.zrok.example.com:8080'

Here's another way of expressing the same shell environment (exporting to all future child processes instead of a specific child process).

export ZROK_ADMIN_TOKEN='Q8V0LqnNb5wNX9kE1fgQ0H6VlcvJybB1';
export ZROK_API_ENDPOINT='http://zrok.example.com:18080';
zrok admin create frontend "ma48lFMsV" public 'http://{token}.zrok.example.com:8080';
18

Thanks a ton @qrkourier for the support now the frontend is running perfectly one more question Ken i am running this frontend and the controller in a seperate terminal which is not efficient in a production environment do i need to make them as a systemd service??

19

That's what I would do! No one has created zrok-controller or zrok-frontend Linux packages yet, unfortunately.

1 Like