After deleting/recreating the immich service, I no longer can access that service from my laptop or my phone through the tunnel.
I went through the troubleshooting steps.
The output of: sudo -u ziti ziti-edge-tunnel dump -p /tmp/ziti-dump-dir/ sanitized and formatted for readability is:
But this step does not produce any outputs: dig +noall +answer immich.localdomain @100.64.0.2
I speculated that perhaps my LXC proxy tunnel isn't working properly and that it might be related to passing the /dev/net/tun device into the unprivileged container, I removed the tunnel from the LXC.
Then created a Ubuntu 24.04 VM (instead of LXC), and installed ziti-edge-tunnel in that VM. The VM behaves the same as the LXC. I still do not get any output on the above step. Any otherway of troubleshooting this tunnel problem?
Now I'm getting the 100% CPU utilization on my VM as well with the error below:
sudo journalctl -u ziti-edge-tunnel.service
WARN ziti-sdk:ziti_ctrl.c:180 ctrl_resp_cb() ctrl[fqdn-of-vps-server:1280] request failed: -111(connection refused)
ERROR ziti-sdk:ziti.c:1289 edge_routers_cb() ztx[0] failed to get current edge routers: code[0] CONTROLLER_UNAVAILABLE/connection refused
WARN ziti-sdk:ziti_ctrl.c:180 ctrl_resp_cb() ctrl[fqdn-of-vps-server:1280] request failed: -111(connection refused)
WARN ziti-sdk:ziti_ctrl.c:566 ctrl_next_ep() ctrl[fqdn-of-vps-server:1280] no controllers are online
WARN ziti-sdk:ziti.c:1237 check_service_update() ztx[0] failed to poll service updates: code[0] err[-16/connection refused]
WARN ziti-sdk:ziti_ctrl.c:180 ctrl_resp_cb() ctrl[fqdn-of-vps-server:1280] request failed: -111(connection refused)
ERROR ziti-sdk:ziti.c:1289 edge_routers_cb() ztx[0] failed to get current edge routers: code[0] CONTROLLER_UNAVAILABLE/connection refused
WARN ziti-sdk:ziti_ctrl.c:180 ctrl_resp_cb() ctrl[fqdn-of-vps-server:1280] request failed: -111(connection refused)
WARN ziti-sdk:ziti_ctrl.c:566 ctrl_next_ep() ctrl[fqdn-of-vps-server:1280] no controllers are online
WARN ziti-sdk:ziti.c:1237 check_service_update() ztx[0] failed to poll service updates: code[0] err[-16/connection refused]
WARN ziti-sdk:bind.c:234 list_routers_cb() server[0.0] failed to list routers: The resource requested was not found or is no longer available
WARN ziti-sdk:bind.c:204 session_cb() server[0.0] failed to get session for service[immich]: -29/NOT_FOUND
ERROR tunnel-cbs:ziti_hosting.c:575 on_hosted_client_connect() hosted_service[immich] incoming connection failed: service not available
I am having extremely difficult time accomplishing this... Using either LXC or VM to enroll an identity through installing the ziti-edge-tunnel results in 100% cpu usage in that LXC or VM. And routing a service to a target server in my network fails.
Instead of having the all-in-one image hosted on oracle vps, then enrolling a LXC or VM from inside my network to that vps, can I accomplish the same by avoiding the vps all together and just host the all-in-one image inside a VM in my network and just configuring the "hosting configuration" in a Service to @quickstart-router and send the traffic to an IP address inside my network? eg:
One possible issue here is the use of the LOCALDOMAIN. TLD. This doesn't have a universal meaning, and I understand this TLD to conventionally mean bypassing DNS and relying on the local host's database. I think it always means the name should resolve to a loopback address, so there's a good chance that it has a special meaning like this on Android and so the DNS query is never sent to the Ziti nameserver.
TL;DR Please try it with a DNS-friendly intercept address like immich.ziti.internal
More context:
When inventing Ziti intercept addresses, the ideal DNS zone is a public zone you control and have delegated to Ziti services. This gives you the flexibility to do two things in the future without necessarily updating the Ziti intercept addresses:
forward queries to a Ziti nameserver by matching the zone
obtain publicly trusted certificates for Ziti services by solving DNS01 challenges, e.g., Caddy, Cert Manager, CertBot, etc.
If those will never apply to your case, then it's also fine to invent Ziti service intercepts with a private zone. ICANN finally settled on INTERNAL. for this purpose, so that's the safest TLD because it's guaranteed to never exist in the global root zone ..
Still, you may use any TLD whether or not you control it. Just be aware that a TLD you don't control could eventually be controlled by another party, allowing them to obtain server certificates for the same DNS zone you use with Ziti.
You're asking if you can run the all-in-one quickstart in a container in your private network instead of a public VPS. Yes! These are the required network flows, i.e., firewall exceptions:
all identities and all routers must reach the controller's advertised address:port
all identities must reach the router's advertised address:port
The advertised address:port must be reachable by all devices running Ziti, so if the address is a domain name then all devices must also be able to resolve that domain name to the correct IP address.
I'll try to reproduce this. Which LXC image and ZET version did you try? For example, Ubuntu Noble image w/ ZET v1.2.3?
EDIT: I followed the install instructions for the ziti-edge-tunnel Debian package in an Ubuntu Noble LXC, though not an LXC on Proxmox. I created an identity from a publicly-available Ziti network with Dial permission for an httpbin service. I added the identity to the running tunneler and was able to send a POST to the httpbin service and got the expected response (link to terminal recording).
Great! I think I got this working now for a few of my service hosted internally.
My LXC or VM was on Ubuntu Noble w ZET 1.2.2. Perhaps my issue w high CPU was a combination of that trying to connect to an arm64 based all-in-one image on the Oracle vps? In any case, bypassing that (ie hosting the all-in-one in a Noble based VM in my network) seem to be working. I have been slowing setting up Services through ziti and making those service on my nginx reverse proxy accessible to my internal network only.
Is intercepting all mobile devices (cellphone, laptop) DNS queries through a ziti tunnel to an Adguard dns server hosted privately, a use case for ziti? If so, do you have a write-up/tutorial on how that is setup?
Was the Ziti container image arm64 or the Oracle VPS or both? It's a good guess that the cause of high CPU was emulation...if indeed you were emulating.
The Ziti container images are multi-platform (arm64, amd64), so if you don't specify a platform in the container runtime, it will use the image of the native platform.
That sounds similar to something I do, too. I like to use a proxy like Caddy to manage trusted certificates for my Ziti services that are web applications. It's an extra step for each Ziti service to configure caddy to handle requests for some domain name and forward them to the right port or container. It means several Ziti services point to the same reverse proxy, so it's a tradeoff. It's nice to have trusted certs for Ziti services, but it also means that if someone has Ziti access to any of the services they could spoof the SNI or HOST header or both to gain access to a different backend app. Not an issue if you're the only authorized party.
I looked up Adguard DNS server.
It only filters DNS queries, not HTTP requests, correct?
Is the Adguard nameserver reachable by the client tunnelers directly, or would it be necessary to tunnel their DNS queries to the hosting/server side that has access to the Adguard nameserver?
Let me ask around. It seems too intrusive to interdict all your clients' resolvers, but maybe someone has a way.
UPDATE: I didn't find a recipe for this specific goal of tunneling only (all) DNS queries. You can tunnel all traffic, including DNS after the tunnel is created, but the nameserver that resolves the Ziti controllers' and routers' addresses must be reachable without the tunnel.
