It appears my issue is related to specifying the 'ZITI_CTRL_ADVERTISED_ADDRESS' in the .env file after removing that line, the server seems to build fine. So now I'm up with the edge-management, fabric, and the zac on 1281 (open to only my home ip), while the edge-client is on 1280 (open to public)
ZITI_CTRL_ADVERTISED_ADDRESS=my_server_public_ipaddress
Yay! Is everything working then?
It appears so. I was able to add an identity for my android phone and for my self-hosted adguard-home server (ubuntu 24.04) and both identities appear as "on-line" in the zac.
Now I can try and figure out the rest of the overlay setting/configurations. Any good quick guide you recommend for the next step?
Great!
Here's an overview of Ziti services.
The ziti CLI has built-in demonstrations for various concepts, including creating your first Ziti service. You can run the demos, which will guide you to run REAL commands, like this.
# run ziti CLI inside the container
docker compose exec quickstart ziti demo first-service
The demo will prompt you to log in with your set password (default is ZITI_PWD=admin).
You can log in the CLI with the same private address where you published the management API and console, e.g., ziti edge login https://127.0.0.1:1281 -u admin -p $ZITI_PWD by running the ziti CLI from anywhere. I suggested running it inside the container because it's already installed there, but it's also readily installed on the Linux Docker host.
curl -sS https://get.openziti.io/install.bash | sudo bash -s "openziti"
Link to more Linux install options.
Been playing around with adding identities from a couple of LXC servers (hosting different things) in my local Proxmox using this method to install the edge-tunnel. It appears the /dev/net/tun device can't be shared by more than 1 LXC at the sametime. So I was planning on enrolling the edge-tunnel on just one LXC and use that to proxy to other servers in my network. But it looks like I'm running into the same issue as this post here only my version combinations are different than his.
My ziti-edge-tunnel version is ziti-edge-tunnel version v1.2.2
My in zac, my versions displayed are: Controller: v1.1.15 ZAC: 3.4.3
I'm getting the same error as the other gentleman when I start the ziti-edge-tunnel.service with a fresh jwt file in the /opt/openziti/etc/identity/ folder:
INFO ziti-sdk:utils.c:167 ziti_log_init() Ziti C SDK version 1.1.2
INFO: enrolled home-proxy.jwt in /opt/openziti/etc/identities/home-proxy.json
WARN ziti-sdk:model_support.c:202 model_parse() json parse error: expected comment
ERROR ziti-sdk:channel.c:943 on_tls_connect() ch[0] failed to connect to ER[quickstart-router]
Do you have any suggestions on how to resolve issue?
Do you want the lxc container running ziti-edge-tunnel to configure the lxc host's IP routes, DNS resolver, etc. for intercepting Ziti service traffic on the lxc host as well as any other lxc containers?
Alternatively, do you want the lxc container to host Ziti services and provide only an exit point to your private network from the Ziti network?
I think I'm aiming for option B. I just need the LXC container to provide an exit point to my private network from the Ziti network... Assuming I can use the Services in zac to route specific identities/attributes to other servers using the hosting configuration...
Cool. That will work if the LXC container's network interface can initiate connections to the target servers inside your network. The tunneler service will begin hosting Ziti services you have created when you grant permission in a Bind Service Policy.
I believe the error you encountered is due to file permissions on the directory /opt/openziti/etc/identities or the JWT file you placed there.
The best way to add an identity because it avoids file permission issues:
sudo ziti-edge-tunnel add --jwt "$(< ./in-file.jwt)" --identity myIdentityName
If you prefer to place the JWT file in the identities dir, ensure you have corrected permissions before restarting the tunneler service.
sudo chown -cR :ziti /opt/openziti/etc/identities
sudo chmod -cR ug=rwX,o-rwx /opt/openziti/etc/identities
Group "ziti" must have read-write-list permission on the directory and its contents, including the JWT file, to delete it after successful enrollment.
If you change your mind later and want to connect to Ziti services from ProxMox, you'll probably need to install Ziti on the LXC host. I'm unaware of how you might configure the LXC host from inside an LXC container.
This command didn't work for me... So I had to manually add the identity because that command results in this error:
{
"Success":false,
"Error":"enrollment failed",
"Code":500
}
I wonder if it has anything to do with running this inside a privileged LXC and permissions on the /dev/net/tun device...
It would be awesome if the error message indicated why enrollment failed. We'll have to investigate. Is the JWT file corrupted? Does the file contain a JWT (three period-separated base64 encodings, etc.)? Is the JWT inside expired? Is it a Ziti identity JWT (em: ott) or a Ziti router JWT (em: erott)?
You can analyze the contents by pasting in jwt.io or by feeding the token file path as a parameter to this script.
If the JWT is valid, then enrollment could still fail because:
iss (issuer) claim is not reachable by the LXC containerziti-edge-tunnel add command are invalid, it may be necessary to provide an arg for the --identity optionEverytime I do this I generate a fresh JWT so it shouldn't be expired. Here's the contents from the most recent file that errored out from jwt.io output.
"alg": "RS256",
"kid": "1df9f1dc7b2ba4767076819aca80275f3555e014",
"typ": "JWT"
"exp": 1728962828,
"jti": "d9f45185-2e57-4dd2-9287-c3e3be0ad98e",
"em": "ott",
"ctrls": null
If I curl -k URL I get this:
{"data":{"apiVersions":{"edge":{"v1":{"apiBaseUrls":["https://vps_public_ip:1280/edge/client/v1"],"path":"/edge/client/v1"}},"edge-client":{"v1":{"apiBaseUrls":["https://vps_public_ip:1280/edge/client/v1"],"path":"/edge/client/v1"}},"edge-management":{"v1":{"apiBaseUrls":["https://127.0.0.1:1281/edge/management/v1"],"path":"/edge/management/v1"}}},"buildDate":"2024-10-02T13:03:23Z","capabilities":[],"revision":"0eec47ce3c80","runtimeVersion":"go1.23.1","version":"v1.1.15"},"meta":{}}
I'm not sure how that could've happened, or to check that. I generate the jwt, immediately winscp the file to the target LXC and the error pops up.
I don't think I understand this one...
I just realized after I wrote this, that I probably need to change winscp transfer mode to binary rather than default. After transfer in binary mode, the enrollment on the LXC using the suggested method succeded.
sudo ziti-edge-tunnel add --jwt "$(< ./in-file.jwt)" --identity
{
"Success":true,
"Code":0
}
But when I check the tunnel service, I'm still getting this error.
sudo systemctl status ziti-edge-tunnel.service
WARN ziti-sdk:model_support.c:202 model_parse() json parse error: expected comment
ERROR ziti-sdk:channel.c:943 on_tls_connect() ch[0] failed to connect to ER[quickstart-router] [-3008/unknown node or service]
ERROR ziti-sdk:channel.c:943 on_tls_connect() ch[0] failed to connect to ER[quickstart-router] [-3008/unknown node or service]
ERROR ziti-sdk:channel.c:943 on_tls_connect() ch[0] failed to connect to ER[quickstart-router] [-3008/unknown node or service]
When I check the tunnel_status, in addition to the current identity (test-identity-2), I find an old identity that I tried to enroll on this machine previously (test-identity-1). But instead of using the ziti-edge-tunnel to delete that identity, I simply deleted that json file in /opt/openziti/etc/identities/ folder. So now I don't know how to delete that identity.
I wonder if the fail to connect error above may be related to this old identity? How do I start from a clean slate for the ziti-edge-tunnel on this LXC?
sudo ziti-edge-tunnel tunnel_status
{
"Success":true,
"Data":{
"Active":false,
"Duration":171179,
"StartTime":"2024-10-16T05:34:18.245000Z",
"Identities":[
{
"Name":"test-identity-1",
"Identifier":"\/opt\/openziti\/etc\/identities\/test-identity-1.json",
"FingerPrint":"test-identity-1",
"Active":true,
"Loaded":false,
"IdFileStatus":false,
"NeedsExtAuth":false,
"MfaEnabled":false,
"MfaNeeded":false,
"Metrics":{
"Up":0,
"Down":0
},
"MfaMinTimeout":0,
"MfaMaxTimeout":0,
"MfaMinTimeoutRem":0,
"MfaMaxTimeoutRem":0,
"MinTimeoutRemInSvcEvent":0,
"MaxTimeoutRemInSvcEvent":0,
"Deleted":false,
"Notified":false
},
{
"Name":"test-identity-2",
"Identifier":"\/opt\/openziti\/etc\/identities\/test-identity-2.json",
"FingerPrint":"test-identity-2",
"Active":true,
"Loaded":true,
"Config":{
"ztAPI":"https:\/\/vps-public-ip:1280"
},
"ControllerVersion":"v1.1.15",
"IdFileStatus":false,
"NeedsExtAuth":false,
"MfaEnabled":false,
"MfaNeeded":false,
"Metrics":{
"Up":0,
"Down":0
},
"MfaMinTimeout":0,
"MfaMaxTimeout":0,
"MfaMinTimeoutRem":0,
"MfaMaxTimeoutRem":0,
"MinTimeoutRemInSvcEvent":0,
"MaxTimeoutRemInSvcEvent":0,
"Deleted":false,
"Notified":false
}
],
"IpInfo":{
"Ip":"100.64.0.1",
"Subnet":"255.192.0.0",
"MTU":65535,
"DNS":"100.64.0.2"
},
"LogLevel":"warn",
"ServiceVersion":{
"Version":"v1.2.2",
"BuildDate":"Sun-10\/13\/2024-17:09:13-UTC"
},
"TunIpv4":"100.64.0.1",
"TunIpv4Mask":10,
"AddDns":false,
"ApiPageSize":25
},
"Code":0
}
Excellent sleuthing finding the JWT was corrupted by the winscp transfer mode that isn't binary. Now you suspect there's a lingering, contradictory tunneler state from a prior enrollment attempt.
To clean the tunneler's Linux service state:
sudo systemctl disable --now ziti-edge-tunnel.service
sudo systemctl reset-failed ziti-edge-tunnel.service
sudo systemctl clean --what=state ziti-edge-tunnel.service
sudo rm -rv /var/lib/ziti/config.json /tmp/.ziti/
Additionally, you may wish to reset your identities dir. This is destructive, and the identities will need to be re-enrolled.
sudo rm -rv /opt/openziti/etc/identities
Finally, purge the package files with apt, or the yum/dnf equivalent.
sudo apt purge ziti-edge-tunnel
This should be a clean state. After re-installing the package, enable and start the service.
sudo systemctl enable --now ziti-edge-tunnel.service
Then, begin adding identities. The argument to the --identity option determines the basename of the resulting JSON file, e.g., some-filename.json in the identity directory.
sudo ziti-edge-tunnel add --jwt "$(< ./in-file.jwt)" --identity some-filename
Another user reported this error that's not actually an error in another thread
...but, not I think this is the error you were referring to. I think the device where you installed ziti-edge-tunnel is unable to lookup the DNS name your router is configured to advertise "quickstart-router."
You're following the all-in-one quickstart instructions, so you need to set ZITI_CTRL_ADVERTISED_ADDRESS=ziti.example.com where the value is an FQDN that resolves to the IP where your quickstart container is listening. It's probably easiest to start with a clean slate for the quickstart and identity enrollments if this value needs to change.
Okay did all of the steps on the LXC, restarted the tunnel and the status was clear of any identities.
Then:
Started from scratch on the VPS, recreated the all-in-one specifying an FQDN for ZITI_CTRL_ADVERTISED_ADDRESS. I didn't realize it had to be FQDN (in the beginning of this thread, I had used the public IP address and that would fail during the docker compose up building stage. Resulting in those missing quickstart-router.* files).
With this fresh setup I create a new identity jwt file, upload via winscp (binary mode), then add the identity jwt using:
sudo ziti-edge-tunnel add --jwt "$(< ./in-file.jwt)" --identity some-filename
{
"Success":true,
"Code":0
}
But now systemctl status shows a different error:
sudo systemctl status ziti-edge-tunnel.service
ERROR ziti-sdk:channel.c:735 ch_connect_timeout() ch[0] connect timeout
And the tunnel_status still shows Active: false
sudo ziti-edge-tunnel tunnel_status
{
"Success":true,
"Data":{
"Active":false,
"Duration":19828,
"StartTime":"2024-10-17T03:18:32.895775Z",
"Identities":[
{
"Name":"test-identity-1",
"Identifier":"\/opt\/openziti\/etc\/identities\/test-identity-1.json",
"FingerPrint":"test-identity-1",
"Active":true,
"Loaded":true,
"Config":{
"ztAPI":"https:\/\/fqdn-of-vps-server:1280"
},
"ControllerVersion":"v1.1.15",
"IdFileStatus":false,
"NeedsExtAuth":false,
"MfaEnabled":false,
"MfaNeeded":false,
"Metrics":{
"Up":0,
"Down":0
},
"MfaMinTimeout":0,
"MfaMaxTimeout":0,
"MfaMinTimeoutRem":0,
"MfaMaxTimeoutRem":0,
"MinTimeoutRemInSvcEvent":0,
"MaxTimeoutRemInSvcEvent":0,
"Deleted":false,
"Notified":false
}
],
"IpInfo":{
"Ip":"100.64.0.1",
"Subnet":"255.192.0.0",
"MTU":65535,
"DNS":"100.64.0.2"
},
"LogLevel":"warn",
"ServiceVersion":{
"Version":"v1.2.2",
"BuildDate":"Sun-10\/13\/2024-17:09:13-UTC"
},
"TunIpv4":"100.64.0.1",
"TunIpv4Mask":10,
"AddDns":false,
"ApiPageSize":25
},
"Code":0
}
curl -k https://fqdn-of-vps-server:1280
shows:
{"data":{"apiVersions":{"edge":{"v1":{"apiBaseUrls":["https://fqdn-of-vps-server:1280/edge/client/v1"],"path":"/edge/client/v1"}},"edge-client":{"v1":{"apiBaseUrls":["https://fqdn-of-vps-server:1280/edge/client/v1"],"path":"/edge/client/v1"}},"edge-management":{"v1":{"apiBaseUrls":["https://127.0.0.1:1281/edge/management/v1"],"path":"/edge/management/v1"}}},"buildDate":"2024-10-02T13:03:23Z","capabilities":[],"revision":"0eec47ce3c80","runtimeVersion":"go1.23.1","version":"v1.1.15"},"meta":{}}
I couldn't find a spec for the tunnel_status object. Still, one maintainer said, .Data.Active is not relevant, only the per-identity property. Data.Identities[].Active represents the status, and that one is Active: true, so I think we're OK for the active/inactive state.
It would be helpful to see more context as surrounding log lines and to increase verbosity to understand which connection timed out.
ziti-edge-tunnel set_log_level --loglevel DEBUG
It must be either the client API or the router edge. You can confirm the client API URL by parsing the identity file. I noted that your tunnel_status output does report a ztAPI: https://fqdn-of-vps-server:1280, but it contains escaping \ for some reason, which is unexpected. Do those also appear verbatim inside the identity JSON?
jq '.ztAPI' /opt/openziti/etc/identities/test-identity-1.json
The router edge listener address it would be attempting to connect is determined by the router's advertise option on the edge listener.
listeners:
- binding: edge
address: tls:0.0.0.0:3022
options:
advertise: fqdn-of-vps-server:3022
The escape must be a typo. running the jq code, shows no escape.
Hmm, it sounds like I'm supposed to open port 3022 to the world. which I haven't done...
Yep! At a minimum, you must publish two ports on an IP address that's reachable by all identities and routers:
Identities talk to controllers and routers, and routers talk to controllers and other routers.
You can have additional routers not advertising a public address. At least one router must be public to allow others to form mesh links.
Got it. Now both of my identitie appear to be connected to the ziti server.
An update. I added an identity for my Windows11 laptop in the ZAC. Added that identity to the immich-dial-policy like this:
And I can access my immich through the tunnel. But still no luck through my android phone. Any hints on how to troubleshoot this connection issue with the phone?
Also just noticed after deleting and recreating the service for my immich, that the CPU usage for ziti-edge-tunnel on the proxy is very high.
