Quickstart doesn't work

1

I am new to Ziti.
I have installed ziti-controller, ziti-router, ziti-host for homelab.
I follow the Kubernetes Service | OpenZiti

But I can't get access to http://hello.ziti.internal/

curl -v hello.ziti.internal
* Host hello.ziti.internal:80 was resolved.
* IPv6: (none)
* IPv4: 100.64.0.3
*   Trying 100.64.0.3:80...
* connect to 100.64.0.3 port 80 from 100.64.0.1 port 60997 failed: Connection refused
* Failed to connect to hello.ziti.internal port 80 after 70 ms: Couldn't connect to server
* Closing connection
curl: (7) Failed to connect to hello.ziti.internal port 80 after 70 ms: Couldn't connect to server

My Pods

ziti-controller-844f58bcbc-7hnjw 1/1 Running 0 4d1h
ziti-host-76799885b-zpfrn 1/1 Running 0 26h
ziti-router-65d664dfff-2dhs4 1/1 Running 17 (8h ago) 4d

kg po -n hello-toy
NAME READY STATUS RESTARTS AGE
hello-toy-5f586cb9c5-2h2z4 1/1 Running 0 4h13m

telegram-cloud-photo-size-2-5371044697902018254-y
telegram-cloud-photo-size-2-5371044697902018254-y1280×445 28.5 KB

│ ID         │ NAME          │ TYPE    │ ATTRIBUTES    │ AUTH-POLICY │
├────────────┼───────────────┼─────────┼───────────────┼─────────────┤
│ 2hhw-oWCj  │ ziti-host     │ Default │               │ Default     │
│ PKQauDhCjD │ hello-client  │ Default │ hello-clients │ Default     │
│ bIk.8HgCL  │ diamon        │ Default │ hello-clients │ Default     │
│ dsWg6GWCjD │ 002           │ Router  │ hello-hosts   │ Default     │
│ gbuzDKPhO  │ Default Admin │ Default │               │ Default     │
╰────────────┴───────────────┴─────────┴───────────────┴─────────────╯

ziti edge policy-advisor services hello-service -q
OKAY : hello-client (1) -> hello-service (1) Common Routers: (1/1) Dial: Y Bind: N
OKAY : diamon (1) -> hello-service (1) Common Routers: (1/1) Dial: Y Bind: N
OKAY : 002 (1) -> hello-service (1) Common Routers: (1/1) Dial: N Bind: Y


[ziggy@ziti-router-65d664dfff-2dhs4 ~]$ curl hello.hello-toy.svc
<pre>
Hello World


			::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
			:::::::::::::::::::,::$77777777777777,:,::::::::::::::::::::
			::::::::::::::::::77777777777777777777777~,:::::::::::::::::
			:::::::::::::::77777777777777II7777777777777,:::::::::::::::
			::::::::::::$777777777777777I.:7777777777777777,::::::::::::
			::::::::::77777777777777777I...I7777777777777777I:::::::::::
			:::::::::77777777777777777I....?777777777777777777::::::::::
			:::::::$77777777777777777I......77777777777777777777::::::::
			::::::777777777777777777I.......I77777777777777777777,::::::
			:::::777777777777777777I....?...?777777777777777777777::::::
			:::,777777777777777777I....I7?...777777777777777777777$:::::
			:::777777777777777777I....I77I...I777777777777777777777$::::
			:::77777777777777777I....I7777...?7777777777777777777777::::
			::77777777777777777I....I77777?..,77777777777777777777777:::
			::7777777777777777I....I777777I...I77777777777777$7$$$$7$,::
			:$777777777777777I....I77777777...?7777777777777$$77777777::
			:777777777777777I ...I777II7777?...I.I7777777$777777777777::
			:77777777777777I....I777I..7777I.......?I777$$$$$77$$$$7$$::
			:7777777777777I....?I77I...I7777..........I777777$$$$$7$$$,:
			:77777777777777?..  .??.   ?7777?  ..??.   .?7$7$$$7$$$$$7::
			,7777777777777777I..........I$77I...I777?....77777$7$$$$$$,,
			:7777777777777777777?.......I7$$7..I777I....7$$$$$$$$$$$$$::
			:777777777777777777777I.I=..?77777777$7....77$$$$$$$$7$$$$::
			:777777777777777777777777I...I$7777777....77$$$$$$$$$$$$$$::
			::77777777777777$7$7$$$$$I...?7$$7$77....7$$$$$$$$$$$$$$$:::
			::777777777777777777$$$777+..~77$$7I....77$$$$$$$$$$$$$$$:::
			:::77777777777777777777$$7I...7$$$I....7$7$$$$$$$$$$$$$$::::
			:::Z77777777$7777777777$77I...?$77....I$$$$$$$$$$$$$$$$$::::
			::::77777$$$$$7777$$$$$$$$7:..+77....I$$$$$$$$$$$$$$$$$:::::
			:::::77777$777$$$$777$$$$77I...I....I$$$$$$$$$$$$$$$$$::::::
			::::::$7777777$7777$$$7$$$$I...... I$$$$$$$$$$$$$$$$7:::::::
			:::::::?$$$$$$$$$$$$$$$$$$$7=.....I$$$$$$$$$$$$$$$$=::::::::
			:::::::::7$$$$$7$$$$$$$$$$$$?....77$$$$$$$$$$$$$$$::::::::::
			::::::::::,7$$7$$$$$$$$$$$$$7...I$$$$$$$$$$$$$$$::::::::::::
			::::::::::::~$$$$$$$$$$$$$$$7?.I$$$$$$$$$$$$$$::::::::::::::
			:::::::::::::::$$$$$$$$$$$$$$77$$$$$$$$$$$$$::::::::::::::::
			::::::::::::::::::7$$$$$$$$$$$$$$$$$$$$$$:::::::::::::::::::
			:::::::::::::::::::::::$$$$$$$$$$$$$::::::::::::::::::::::::
			::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::


</pre>

What am I doing wrong?

2

Welcome! My first guess is the identity named "002" is not running or cannot reach the target address in the host.v1 config for hello-service. You already confirmed it has permission to host/bind the Ziti service, so the last piece, if it's running, is to ensure the real target address you verified, hello.hello-toy.svc:80, matches the host.v1 config for hello-service.

3

@qrkourier, Excuse me, as you can see, I demonstrated it above.

I duplicate it here

ziti-router-65d664dfff-2dhs4 1/1 Running 17 (8h ago) 4d

# inside the pod ziti-router-65d664dfff-2dhs4
curl hello.hello-toy.svc
<pre>
Hello World

ziti edge policy-advisor services hello-service -q
OKAY : hello-client (1) -> hello-service (1) Common Routers: (1/1) Dial: Y Bind: N
OKAY : diamon (1) -> hello-service (1) Common Routers: (1/1) Dial: Y Bind: N
OKAY : 002 (1) -> hello-service (1) Common Routers: (1/1) Dial: N Bind: Y

002           │ Router  │ hello-hosts   │ Default

What about host.v1 I use, like it was in the quickstart guide.

ziti edge create config "hello-intercept-config" intercept.v1 \
    '{"protocols":["tcp"],"addresses":["hello.ziti.internal"], "portRanges":[{"low":80, "high":80}]}'

ziti edge create config "hello-host-config" host.v1 \
    '{"protocol":"tcp", "address":"hello.hello-toy.svc","port":80}'

ziti edge create service "hello-service" \
    --configs hello-intercept-config,hello-host-config

image
image1606×672 51.8 KB

Could you @qrkourier clarify for me what I should demonstrate to investigate the problem?

4

Yes, let's work backwards from your Ziti Desktop Edge with identity "diamon," i.e., the tunneler working as a client proxy for macOS, where you attempted to visit http://hello.ziti.internal. In the terminal, you confirmed that Ziti DNS resolves the domain name in macOS, but the connection is refused.

In most cases, the client initiating the connection will log a helpful message. You can also inspect the logs for your Ziti controller running in Kubernetes.

First, let's identify the current log file for Ziti Desktop Edge in macOS. I don't have access to a macOS device, so I'm guessing at the correct path from old notes. Please double check if no results!

The last line of output should be the current tunneler log file, where you need to look for an error message when you try to connect to hello-service. You can open it in your prefered log viewer and try hello-service.

ls -ltrAh ~/Library/Group\ Containers/MN5S649TXM.ZitiPacketTunnel.group/logs/TUN_*.log

:backhand_index_pointing_up: if that finds the log file, you can watch it for new messages in your terminal while you try to connect in your browser.

ls -1t ~/Library/Group\ Containers/MN5S649TXM.ZitiPacketTunnel.group/logs/TUN_*.log | head -1 | xargs -r tail -F

If you don't find an interesting error message in the tunneler log then look in the controller log.

kubectl --namespace ziti logs --selector app.kubernetes.io/name=ziti-controller --container=ziti-controller --follow --tail=-1

Good luck!

5

appex.log rather than TUN_*.log

6

Well, I found the problem. ziti-router didn't create a terminator for hello-toy.

{"_context":"ch{edge}-\u003eu{classic}-\u003ei{ziti-sdk-c[0]@mac.local/l8nK}","chSeq":8,"connId":3,"edgeSeq":0,"error":"service 5kqph5pYy4BBYWCzp19SDv has no terminators","file":"github.com/openziti/ziti/router/xgress_edge/listener.go:199","func":"github.com/openziti/ziti/router/xgress_edge.(*edgeClientConn).processConnect","level":"warning","msg":"failed to dial fabric","time":"2025-06-17T14:15:38.372Z","token":"zzzzzzzzz","type":"EdgeConnectType"}

I spent a few days investigating all available pages on the current documentation.
Then I switched to watching YouTube videos, again and again.
Well, after a few failed attempts.
I found the video https://www.youtube.com/watch?v=Ocia-KdPBXM

I saw an old version of the documentation. And the previous documentation was clearer and useful.

  1. I was creating yet another identity
    ziti edge create identity "hello-host" --role-attributes hello-hosts --jwt-output-file hello-host.jwt

  2. ziti-host chart has a bug with PVC jwt. It means you can't pass jwt directly. You must enroll it manually.
    ziti edge enroll --jwt hello-host.jwt --out hello-host.json

  3. install ziti-host with identity=hello-host
    helm upgrade -i ziti-host openziti/ziti-host --set-file zitiIdentity=hello-host.json

  4. Profit

    curl http://hello.ziti.internal/
<pre>
Hello World

Unfortunately, I dunno why this command should work.

ziti edge update identity "router1" --role-attributes hello-hosts # for me = "002"

Should I change ziti-router tunneler mode or something else?

7

Nice going! It was a journey. Let's see if we can remove those speedbumps for the next person.

The first thing I checked was the PVC issue you mentioned in the ziti-host chart. This worked for me:

helm upgrade --install ziti-host openziti/ziti-host --set-file zitiEnrollToken=hello-host.jwt

Which distribution of K8S did you try? Does it have a default storage class? The ziti-host chart's PVC template is helm-charts/charts/ziti-host/templates/identity-pvc.yaml at main · openziti/helm-charts · GitHub, which doesn't specify a storage class, so it depends on the cluster fulfilling the claim with the default class. We could add an input to set the class if needed. The only other possibility that comes to mind is the default size of 2GiB. It's conceivable your distribution, like some managed distributions of K8S, won't fulfill a PVC that requests less than some higher size. You should see a relevant event in the describe output if that's the case.

8

Does this summary capture the path you took?

First, you followed the first k8s service tutorial, which guided you to host the "hello-service" with the built-in Ziti identity on the Ziti router running inside the cluster. That didn't work for some reason, but you went back to the old version of the guide and found it had been written to instead host the service on a "ziti-host" pod, which can be created by deploying the Helm Chart of the same name (link to doc about this).

There's nothing wrong with hosting services inside the cluster this way. It's very similar to the approach used in the first K8S service guide, but it uses a dedicated tunneler running in host mode instead of a router's host mode. I just switched it over to router-based hosting to save a step.

Zooming in on the problem you encountered with the current version of the first K8S service tutorial, can you clarify whether you updated the router's identity with the hosting role attribute like this?

ziti edge update identity "002" --role-attributes hello-hosts

I'm pretty sure you did this, but just making certain! Now I see you did show the identity of type "Router" named "002" has this role, so it must be something else!

Within a few seconds, the router named "002" should show that it started hosting "hello-service." Then the service should begin working end-to-end since you already confirmed your client identity has dial permission and the web server on the cluster IP is reachable by the 002 router pod.

Yes, that's the next thing. The router's config.yml was written to its persistence volume during first startup. You can verify it has a tunnel mode binding. It should be in host mode for hosting services.

kubectl -n ziti get pods --selector app.kubernetes.io/name=ziti-router -o jsonpath="{.items[0].metadata.name}" \
| xargs -rI POD kubectl exec -n ziti POD -- cat /etc/ziti/config/ziti-router.yaml

It should have lines like these.

listeners:
  - binding: edge
    address: tls:0.0.0.0:3022
    options:
      advertise: 002.ziti.example.com:443
  - binding: tunnel
    options:
        mode: host
9

@qrkourier do you have any plans for ziti-k8s-operator? (Should I ask about ziti-k8s-operator in another topic :sweat_smile: ?)

I found a netfoundry/ziti-k8s-agent, but it's not what I want.

K8s-friendly operator

apiVersion: v1
kind: Service
metadata:
  annotations:
    openziti.io/bla-bla: foobar

and/or

I have taken a look at https://github.com/openziti/ngx_ziti_module, but it is not for beginners. It's too complicated to implement in Kubernetes.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: nginx
spec:
  defaultBackend:
    service:
      name: nginx
      port:
        number: 80
  ingressClassName: openziti
10

Yep, ziti-host chart PVC doesn't have the metadata.labels.
My bad, looks like a problem with my identity. Even though the token was generated and immediately passed to helm install ziti-host, the time it took to deploy the PVC and deployment was too long, and the token expired. I didn't find a way to extend the expiration time.

kubectl logs ziti-host-58bdc95c75-r29m7  
Defaulted container "ziti-host" out of: ziti-host, chown-identity-dir (init)
WARN: clobbering non-empty Ziti enrollment token file /ziti-edge-tunnel/ziti-host-identity.jwt with contents of env var ZITI_ENROLL_TOKEN
DEBUG: waiting 3s for /ziti-edge-tunnel/ziti-host-identity.json (or token) to appear
DEBUG: identity file /ziti-edge-tunnel/ziti-host-identity.json not found
DEBUG: /var/run/secrets/netfoundry.io/enrollment-token/ziti-host-identity.jwt not found
DEBUG: /enrollment-token/ziti-host-identity.jwt not found
INFO: enrolling /ziti-edge-tunnel/ziti-host-identity.jwt
(6)[        0.000]    INFO ziti-sdk:utils.c:198 ziti_log_set_level() set log level: root=3/INFO
  (6)[        0.000]    INFO ziti-sdk:utils.c:169 ziti_log_init() Ziti C SDK version 1.6.1 @g6057d76(HEAD) starting at (2025-06-19T13:37:56.725)
  (6)[        0.000]    INFO ziti-sdk:ziti_enroll.c:114 ziti_enroll() Ziti C SDK version 1.6.1 @g6057d76(HEAD) starting enrollment at (2025-06-19T13:37:56.725)
  (6)[        0.000]    INFO ziti-sdk:ziti_ctrl.c:637 ziti_ctrl_init() ctrl[https://ziti-edge.example.com:443] controller initialized
  (6)[        0.000]    INFO ziti-sdk:ziti_ctrl.c:637 ziti_ctrl_init() ctrl[https://ziti-edge.example.com:443] controller initialized
  (6)[        0.000]   ERROR ziti-sdk:ziti_ctrl.c:525 ctrl_body_cb() ctrl[https://ziti-edge.example.com:443] API request[/enroll] failed code[INVALID_ENROLLMENT_TOKEN] message[The supplied token is not valid]
(6)[        0.000]   ERROR ziti-sdk:ziti_enroll.c:419 enroll_cb() failed to enroll with controller: https://ziti-edge.example.com:443 INVALID_ENROLLMENT_TOKEN[The supplied token is not valid] reason[]
(6)[        0.000]   ERROR ziti-edge-tunnel:ziti-edge-tunnel.c:1644 enroll_cb() enrollment failed: JWT not accepted by controller(-3)
ERROR: failed to enroll with token from /ziti-edge-tunnel/ziti-host-identity.jwt (1034B)
11

My current router config:

listeners:
# bindings of edge and tunnel requires an "edge" section below
  - binding: edge
    address: tls:0.0.0.0:3022
    options:
      advertise: ziti-router1.example.com:443

Should I use the next helm values with "002 --role-attributes hello-hosts"?

# helm-values for ziti-router
tunnel:
  mode: proxy
  proxyServices:
    # This will be bound on the "default" proxy Kubernetes service, see below
    - zitiService: ziti.svc
      containerPort: 10443
      advertisedPort: 10443
  proxyDefaultK8sService:
    enabled: true
    type: ClusterIP
12

Yes, I'm happy to discuss and ideate on a K8S Operator in another topic. That's preferred, thank you.

13

Good sleuthing on that enrollment issue. Still, the hosting with the router's built-in identity should have worked the way the tutorial is written. If we can figure out what happened then maybe we can make that tutorial more resistant to the same problem.

14

Well, followed by quickstart. I have only 1 symptom: ziti-router doesn't have a terminator. I was using debug=1, verbose=1. I dunno what happened. How can I help investigate it?

1 Like