Using OpenZiti in distributed surveillance system

papiris · February 23, 2024, 10:28am

Hi good folks,

I'm building a thing , and find myself in need of a way to connect video streaming webservices in multiple locations, to eachother as well as a dynamic selection of strongly authenticated and group-isolated clients.

In more human terms, I'm making an open source, privacy-first, camera surveillance (NVR) system for farms; designed for scalability, security, ease of deployment and use.

At first I tried using Nebula, but provisioning new clients to the network turned out to be not so simple. There's a community maintained solution, but the docs are somewhat lackluster.

Then I remembered hearing about OpenZiti on FLOSS Weekly a while ago; and figured I'd check it out. OpenZiti seems a little overkill for my project currently, but I think it's better to grow into a solution, rather than grow out of one.

OpenZiti consists of a multitude of components, and uses terms that are a little unfamiliar to me. For that reason, I hope you can provide guidance on which parts of OpenZiti I need, and where I should put them

Here's a simplified overview of the deployment plan of the surveillance system:

I'm thinking of using k3s, containerized software, immutable OSes and gitops as much as possible, to ease deployment and maintenance.

A few points to keep in mind:

Each farm may consist of multiple physical locations, on different networks.
Only one location per farm should need to deploy the actual surveillance software.
No camera should be able to communicate with anything except for the surveillance software.
No cross-talk between farms, no information leakage.
The NVR system has both mobile apps and a webapp for mobile, and both should be accessible.
Getting access to the NVR UI from a new client should be trivial for non-technical users.
The mesh controller should be centralized by default, but users should have a path to transition from managed to self hosted.

If I understand OpenZiti correctly, I would need

a controller accessible to all participants in the mesh (in my case, deployed in a public cloud)
a containerized tunneler in each farm's k3s cluster
a (edge?) router in each cluster?
Browzer-bootstrapper for PWA
tunnelers on each client device (when not using PWA)

I'm uncertain about how I should make this project work with OpenZiti, so any and all advice are welcome

Cheers,
Jacob

TheLumberjack · February 23, 2024, 1:11pm

Hi @papiris, welcome to the community and to OpenZiti!

Thanks for the details in your question. That sounds like a neat project™ and a perfect use of OpenZiti, if you're building the camera software. If you do, then you can build an sdk into the camera software and you can accomodate reqruirements 3 and 4 easily. If you don't control that software, well then it's going to be hard to verify/guarantee #3/#4 using OpenZiti alone... I expect you won't be using OpenZiti for local traffic, relying on trusting that private network. It wasn't clear to me from your diagram exactly what you meant by 'No cross-talk between farms, no information leakage'. But I think that's what you meant?

Your overview of what you need sounds right, yes. I'd probably lean towards an edge router at each location so that the clients would be able to connect to the overlay using the LAN when they're on the LAN.

I'd start by:

get your overlay network working and just get familiar with the zero trust concepts (which are somewhat different than the classic, IP-based rules most people are familiar with), identities, services, configs, service-policy, edge router policy... Those are the ones you'll need to understand.
look at the helm/kubernetes stuff we have out there
you mentioned provisioning new clients -- you'll have to decide how you'll want to do that. I'd start by using the Ziti Desktop Edge/Ziti Mobile Edge clients and emailing people a jwt then possibly transition to making your own (whitelabled) app someday. You'll also want to learn how to automate the setting up of the k3s cluster/identity
once you have identities provisioned, you'll then use services, service policies to see everything working
after you get familiar with that, I'd layer on browzer at the end.

hope that helps (or at least starts a discussion)

papiris · February 26, 2024, 8:54am

Thanks a lot for your detailed reply, @TheLumberjack!
I tried building the camera software myself last year, and it turned out to be way more trouble than its worth. That project didn't make it past the prototyping stage.

This time I'll be relying on other open source offerings, but I haven't made a choice with regards to which project to integrate yet. Viseron, Zoneminder and Frigate are all in the running; with Moonfire-NVR as a strong contender if it only were a bit further along in its development.
Since they're all open source, contributing zero-trust networking capabilities upstream using the various SDKs seems like the way to go in the long term. (PS. It would be super nice to have a native Rust SDK, so Rusts strong safety guarantees would encompass the Ziti code as well )

At least as a start, I'll be happy with securing traffic outside of the local network. Users will have to configure their routers to forward all traffic from the local cameras to the box running Ziti; which can, but isn't necessarily the same box running their copy of the NVR software. This would neuter cameras phoning home; and make them less vulnerable to exploitation. Going a step further, routers could be configured to disallow any traffic to and from the cameras, except communication with the Ziti box.
For "real" zero trust from the camera, I suppose one could intercept the camera radio circuits with an ESP32 flashed with an OpenZiti app. I'll leave that to as an ambition for now

By "No cross-talk between farms, no information leakage", I meant that apart from the information necessary for me to operate the network, no user information should be available to other users or myself. There should be guarantees that third-parties won't be able to access user information. Having now gone through the local, no docker expressInstall and read a bit more about the auth system, I think OpenZiti fills that need.

That looks like a step-by-step I can follow. I'll keep you all posted about progress and bumps in the road

papiris · March 30, 2024, 10:39pm

Update!
Because I couldn't get any of the other open source NVR projects working to my satisfaction, I picked up my past project, Predalert, and will try to implement OpenZiti in it going forwards, using the Python SDK.
On that note, I've met a couple snags in the road, which I'll just list here.

edit: Solved point 5. Turns out that having group ziti own the identities directory is really important. By actually following the tutorial and doing:

$ sudo chown -cR :ziti        /etc/openziti/identities
$ sudo chmod -cR ug=rwX,o-rwx /etc/openziti/identities
eier av '/etc/openziti/identities/laptopClient1.json' er endret fra root:root til :ziti
eier av '/etc/openziti/identities' er endret fra root:root til :ziti
'/etc/openziti/identities' har endret modus fra 0755 (rwxr-xr-x) til 0770 (rwxrwx---)
'/etc/openziti/identities/laptopClient1.json' har endret modus fra 0600 (rw-------) til 0660 (rw-rw----)

My laptop can now reach the example/test services just fine

Ziti Edge Developer Sandbox docs/tour links to https://openziti.github.io/ziti/config-store/overview.html which doesn't exist. Is it supposed to be "Builtin Config Types | OpenZiti"?
Ziti Edge Tunneler for Linux seems to put a lot of useful stuff in directories that are read-only on "immutable" distros like Fedora Atomic Workstation, e.g /opt and /usr. Since the systemd service file (when enabled) in /etc/systemd/system/multi-user.target.wants/ziti-edge-tunnel.service is just a symlink to /usr/lib/systemd/system/ziti-edge-tunnel.service, changing the systemd service must be done by creating a drop-in snippet in /etc/systemd/system/ziti-edge-tunnel.service.d/. This drop-in file loads identities from a read-write directory (make sure it exists first) instead of the read-only (and therefore broken) default location of /opt/openziti/etc/identities:

[Service]
ExecStart=
ExecStart=/opt/openziti/bin/ziti-edge-tunnel run --verbose=${ZITI_VERBOSE} --dns-ip-range=${ZITI_DNS_IP_RANGE} --identity-dir=/etc/openziti/identities

I've had bad experiences with running rootful containers on a system that primarily runs rootless podman. Permissions issues has led me to nuke all containers (podman system reset) a few times. Because of this I'm not all that keen on using the containerized ziti-edge-tunnel on my daily+dev laptop, since it AFAICT requires root privileges to do its magic on the host's network.
My webapp project uses a client-side serviceworker to handle webpush notifications to the client. This requires the website and its contents be served in a secure context, i.e. SSL encryption with a cert from a trusted CA. I haven't gotten a chance to look into it yet, but I surmise that internally zitified apps can, in addition to the mTLS stuff of ziti, also encrypt traffic using regular SSL certs.
Even with the drop-in file, spinning up a network in ZEDS, enrolling laptopClient1.jwt and placing laptopClient1.json in /etc/openziti/identities/; my laptop can't reach any of the *.ziti test services provided in the ZEDS tour. With loglevel=debug, these are the journalctl logs:

mars 30 23:33:03 fedora systemd[1]: Stopping ziti-edge-tunnel.service - Ziti Edge Tunnel...
░░ Subject: A stop job for unit ziti-edge-tunnel.service has begun execution
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░
░░ A stop job for unit ziti-edge-tunnel.service has begun execution.
░░
░░ The job identifier is 380375.
mars 30 23:33:03 fedora systemd[1]: ziti-edge-tunnel.service: Deactivated successfully.
░░ Subject: Unit succeeded
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░
░░ The unit ziti-edge-tunnel.service has successfully entered the 'dead' state.
mars 30 23:33:03 fedora systemd[1]: Stopped ziti-edge-tunnel.service - Ziti Edge Tunnel.
░░ Subject: A stop job for unit ziti-edge-tunnel.service has finished
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░
░░ A stop job for unit ziti-edge-tunnel.service has finished.
░░
░░ The job identifier is 380375 and the job result is done.
mars 30 23:33:03 fedora systemd[1]: ziti-edge-tunnel.service: Consumed 5.300s CPU time.
░░ Subject: Resources consumed by unit runtime
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░
░░ The unit ziti-edge-tunnel.service completed and consumed the indicated resources.
mars 30 23:33:03 fedora systemd[1]: Starting ziti-edge-tunnel.service - Ziti Edge Tunnel...
░░ Subject: A start job for unit ziti-edge-tunnel.service has begun execution
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░
░░ A start job for unit ziti-edge-tunnel.service has begun execution.
░░
░░ The job identifier is 380375.
mars 30 23:33:03 fedora ziti-edge-tunnel.sh[56012]: NOTICE: no new JWT files in /opt/openziti/etc/identities/*.jwt
mars 30 23:33:03 fedora systemd[1]: Started ziti-edge-tunnel.service - Ziti Edge Tunnel.
░░ Subject: A start job for unit ziti-edge-tunnel.service has finished successfully
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░
░░ A start job for unit ziti-edge-tunnel.service has finished successfully.
░░
░░ The job identifier is 380375.
mars 30 23:33:03 fedora ziti-edge-tunnel[56020]: (56020)[        0.000]    INFO ziti-sdk:utils.c:199 ziti_log_set_level() set log level: root=3/INFO
mars 30 23:33:03 fedora ziti-edge-tunnel[56020]: (56020)[        0.000]    INFO tunnel-sdk:ziti_tunnel.c:60 create_tunneler_ctx() Ziti Tunneler SDK (v0.22.26-local)
mars 30 23:33:03 fedora ziti-edge-tunnel[56020]: (56020)[        0.000]    INFO tunnel-cbs:ziti_dns.c:164 seed_dns() DNS configured with range 100.64.0.0 - 100.127.255.255 (4194302 ips)
mars 30 23:33:03 fedora ziti-edge-tunnel[56020]: (56020)[        0.000]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:1716 run_tunneler_loop() Loading identity files from /etc/openziti/identities
mars 30 23:33:03 fedora ziti-edge-tunnel[56020]: (56020)[        0.000]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:1120 load_identities() loading identity file: laptopClient1.json
mars 30 23:33:03 fedora ziti-edge-tunnel[56020]: (56020)[        0.000]    WARN ziti-edge-tunnel:instance.c:40 find_tunnel_identity() Identity ztx[/etc/openziti/identities/laptopClient1.json] is not loaded yet or already removed.
mars 30 23:33:03 fedora ziti-edge-tunnel[56020]: (56020)[        0.000]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:1642 make_socket_path() effective group set to 'ziti' (gid=965)
mars 30 23:33:03 fedora ziti-edge-tunnel[56020]: (56020)[        0.056]    INFO ziti-edge-tunnel:resolvers.c:68 init_libsystemd() Initializing libsystemd
mars 30 23:33:03 fedora ziti-edge-tunnel[56020]: (56020)[        0.057]   ERROR ziti-sdk:model_support.c:67 parse_tokens() jsmn_parse() failed: -2
mars 30 23:33:03 fedora ziti-edge-tunnel[56020]: (56020)[        0.057]   ERROR ziti-edge-tunnel:instance-config.c:136 save_tunnel_status_to_file() Could not copy config file [/var/lib/ziti/config.json] to backup config file, the config might not exists at the moment
mars 30 23:33:03 fedora ziti-edge-tunnel[56020]: (56020)[        0.057]   ERROR ziti-edge-tunnel:instance-config.c:142 save_tunnel_status_to_file() Could not open config file /var/lib/ziti/config.json to store the tunnel status data
mars 30 23:33:03 fedora ziti-edge-tunnel[56020]: (56020)[        0.057]    INFO ziti-edge-tunnel:resolvers.c:356 try_libsystemd_resolver() systemd-resolved selected as DNS resolver manager

Looking at the logs, maybe only new, not already enrolled *.jwt files should go in the identity dir?
logs without any files in /etc/openziti/identities/:

░░ Subject: A stop job for unit ziti-edge-tunnel.service has begun execution
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░
░░ A stop job for unit ziti-edge-tunnel.service has begun execution.
░░
░░ The job identifier is 388518.
mars 30 23:37:31 fedora systemd[1]: ziti-edge-tunnel.service: Deactivated successfully.
░░ Subject: Unit succeeded
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░
░░ The unit ziti-edge-tunnel.service has successfully entered the 'dead' state.
mars 30 23:37:31 fedora systemd[1]: Stopped ziti-edge-tunnel.service - Ziti Edge Tunnel.
░░ Subject: A stop job for unit ziti-edge-tunnel.service has finished
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░
░░ A stop job for unit ziti-edge-tunnel.service has finished.
░░
░░ The job identifier is 388518 and the job result is done.
mars 30 23:37:32 fedora systemd[1]: Starting ziti-edge-tunnel.service - Ziti Edge Tunnel...
░░ Subject: A start job for unit ziti-edge-tunnel.service has begun execution
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░
░░ A start job for unit ziti-edge-tunnel.service has begun execution.
░░
░░ The job identifier is 388518.
mars 30 23:37:32 fedora ziti-edge-tunnel.sh[56708]: NOTICE: no new JWT files in /opt/openziti/etc/identities/*.jwt
mars 30 23:37:32 fedora systemd[1]: Started ziti-edge-tunnel.service - Ziti Edge Tunnel.
░░ Subject: A start job for unit ziti-edge-tunnel.service has finished successfully
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░
░░ A start job for unit ziti-edge-tunnel.service has finished successfully.
░░
░░ The job identifier is 388518.
mars 30 23:37:32 fedora ziti-edge-tunnel[56719]: (56719)[        0.000]    INFO ziti-sdk:utils.c:199 ziti_log_set_level() set log level: root=3/INFO
mars 30 23:37:32 fedora ziti-edge-tunnel[56719]: (56719)[        0.000]    INFO tunnel-sdk:ziti_tunnel.c:60 create_tunneler_ctx() Ziti Tunneler SDK (v0.22.26-local)
mars 30 23:37:32 fedora ziti-edge-tunnel[56719]: (56719)[        0.000]    INFO tunnel-cbs:ziti_dns.c:164 seed_dns() DNS configured with range 100.64.0.0 - 100.127.255.255 (4194302 ips)
mars 30 23:37:32 fedora ziti-edge-tunnel[56719]: (56719)[        0.000]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:1716 run_tunneler_loop() Loading identity files from /etc/openziti/identities
mars 30 23:37:32 fedora ziti-edge-tunnel[56719]: (56719)[        0.000]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:1642 make_socket_path() effective group set to 'ziti' (gid=965)
mars 30 23:37:32 fedora ziti-edge-tunnel[56719]: (56719)[        0.057]    INFO ziti-edge-tunnel:resolvers.c:68 init_libsystemd() Initializing libsystemd
mars 30 23:37:32 fedora ziti-edge-tunnel[56719]: (56719)[        0.057]   ERROR ziti-edge-tunnel:instance-config.c:136 save_tunnel_status_to_file() Could not copy config file [/var/lib/ziti/config.json] to backup config file, the config might not exists at the moment
mars 30 23:37:32 fedora ziti-edge-tunnel[56719]: (56719)[        0.057]   ERROR ziti-edge-tunnel:instance-config.c:142 save_tunnel_status_to_file() Could not open config file /var/lib/ziti/config.json to store the tunnel status data
mars 30 23:37:32 fedora ziti-edge-tunnel[56719]: (56719)[        0.058]    INFO ziti-edge-tunnel:resolvers.c:356 try_libsystemd_resolver() systemd-resolved selected as DNS resolver manager

We can see that ziti gets registered by systemd-resolved as a DNS resolver:

resolvectl status
Global
         Protocols: LLMNR=resolve -mDNS -DNSOverTLS DNSSEC=no/unsupported
  resolv.conf mode: stub

Link 2 (enp4s0)
    Current Scopes: none
         Protocols: -DefaultRoute LLMNR=resolve -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 3 (wlp3s0)
    Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6
         Protocols: +DefaultRoute LLMNR=resolve -mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: 192.168.10.1
       DNS Servers: 192.168.10.1
        DNS Domain: lan

Link 19 (ziti0)
    Current Scopes: DNS
         Protocols: +DefaultRoute -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: 100.64.0.2
       DNS Servers: 100.64.0.2

but the .ziti domain doesn't get resolved:

 $ wget http://httpbin.ziti/image
Failed to resolve 'httpbin.ziti' (Name or service not known)
Failed to resolve 'httpbin.ziti' (Name or service not known)

                          [Files: 0  Bytes: 0  [0 B/s] Redirects: 0  Todo: 1  Errors: 0]

$ curl http://httpbin.ziti/image
curl: (6) Could not resolve host: httpbin.ziti

some more troubleshooting info:

$ readlink -f /etc/resolv.conf
/run/systemd/resolve/stub-resolv.conf

$ cat /etc/resolv.conf
# This is /run/systemd/resolve/stub-resolv.conf managed by man:systemd-resolved(8).
# Do not edit.
#
# This file might be symlinked as /etc/resolv.conf. If you're looking at
# /etc/resolv.conf and seeing this text, you have followed the symlink.
#
# This is a dynamic resolv.conf file for connecting local clients to the
# internal DNS stub resolver of systemd-resolved. This file lists all
# configured search domains.
#
# Run "resolvectl status" to see details about the uplink DNS servers
# currently in use.
#
# Third party programs should typically not access this file directly, but only
# through the symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a
# different way, replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.

nameserver 127.0.0.53
options edns0 trust-ad
search lan

$ cat /etc/systemd/resolved.conf
nameserver 100.64.0.2

$ inxi -FZy
System:
  Host: fedora Kernel: 6.8.1-300.fc40.x86_64 arch: x86_64 bits: 64
  Desktop: KDE Plasma v: 6.0.2 Distro: Fedora Linux 40.20240329.n.1 (Kinoite
    Prerelease)
Machine:
  Type: Laptop System: Notebook product: P95xER v: N/A
    serial: <superuser required>
  Mobo: Notebook model: P95xER serial: <superuser required>
    UEFI: American Megatrends v: 1.07.15 date: 11/16/2018
Battery:
  ID-1: BAT0 charge: 33.8 Wh (80.9%) condition: 41.8/53.2 Wh (78.7%)
CPU:
  Info: 6-core model: Intel Core i7-8750H bits: 64 type: MT MCP cache:
    L2: 1.5 MiB
  Speed (MHz): avg: 805 min/max: 800/4100 cores: 1: 800 2: 800 3: 800 4: 800
    5: 800 6: 800 7: 800 8: 800 9: 800 10: 800 11: 800 12: 865
Graphics:
  Device-1: Intel CoffeeLake-H GT2 [UHD Graphics 630] driver: i915 v: kernel
  Device-2: NVIDIA GP104M [GeForce GTX 1070 Mobile] driver: nvidia v: 550.67
  Device-3: Chicony USB 2.0 Camera driver: uvcvideo type: USB
  Display: wayland server: X.org v: 1.20.14 with: Xwayland v: 23.2.4
    compositor: kwin_wayland driver: X: loaded: modesetting unloaded: fbdev,vesa
    dri: iris gpu: i915,nvidia resolution: 1920x1080
  API: EGL v: 1.5 drivers: iris,nvidia,swrast
    platforms: wayland,x11,surfaceless,device
  API: OpenGL v: 4.6.0 compat-v: 4.5 vendor: intel mesa v: 24.0.0
    renderer: Mesa Intel UHD Graphics 630 (CFL GT2)
  API: Vulkan v: 1.3.275 drivers: N/A surfaces: xcb,xlib,wayland
Audio:
  Device-1: Intel Cannon Lake PCH cAVS driver: snd_hda_intel
  Device-2: NVIDIA GP104 High Definition Audio driver: snd_hda_intel
  API: ALSA v: k6.8.1-300.fc40.x86_64 status: kernel-api
  Server-1: PipeWire v: 1.0.4 status: active
Network:
  Device-1: Intel Wi-Fi 5 Wireless-AC 9x6x [Thunder Peak] driver: iwlwifi
  IF: wlp3s0 state: up mac: c2:76:21:9f:9e:20
  Device-2: Realtek RTL8111/8168/8211/8411 PCI Express Gigabit Ethernet
    driver: r8169
  IF: enp4s0 state: down mac: 80:fa:5b:59:46:51
  IF-ID-1: ziti0 state: unknown speed: 10000 Mbps duplex: full mac: N/A
Bluetooth:
  Device-1: Intel Wireless-AC 9260 Bluetooth Adapter driver: btusb type: USB
  Report: btmgmt ID: hci0 rfk-id: 2 state: down bt-service: enabled,running
    rfk-block: hardware: no software: yes address: 74:E5:F9:89:55:2F bt-v: 5.1
Drives:
  Local Storage: total: 2.27 TiB used: 1.39 TiB (61.2%)
  ID-1: /dev/nvme0n1 vendor: Samsung model: SSD 980 500GB size: 465.76 GiB
  ID-2: /dev/sda vendor: Seagate model: ST2000LX001-1RG174 size: 1.82 TiB
Partition:
  ID-1: /boot size: 973.4 MiB used: 287.3 MiB (29.5%) fs: ext4
    dev: /dev/nvme0n1p2
  ID-2: /boot/efi size: 598.8 MiB used: 11.6 MiB (1.9%) fs: vfat
    dev: /dev/nvme0n1p1
  ID-3: /var size: 464.16 GiB used: 330.3 GiB (71.2%) fs: btrfs
    dev: /dev/dm-0
Swap:
  ID-1: swap-1 type: zram size: 8 GiB used: 0 KiB (0.0%) dev: /dev/zram0
Sensors:
  System Temperatures: cpu: 42.0 C pch: 57.0 C mobo: N/A
  Fan Speeds (rpm): N/A
Info:
  Memory: total: 32 GiB note: est. available: 31.07 GiB used: 7.7 GiB (24.8%)
  Processes: 411 Uptime: 12h 32m Shell: Zsh inxi: 3.3.33

$ resolvectl domain
Global:
Link 2 (enp4s0):
Link 3 (wlp3s0): lan
Link 19 (ziti0):

$ resolvectl dns
Global:
Link 2 (enp4s0):
Link 3 (wlp3s0): 192.168.10.1
Link 19 (ziti0): 100.64.0.2

Oh and happy holidays to those who celebrate

papiris · April 1, 2024, 9:14am

I use Tailscale to access a few remote systems (for now). Realized today that running tailscale at least partially conflicts with running OpenZiti, as none of the example ziti services are reachable from my system (with any browser or tool); as long as the Tailscale daemon tailscaled is running.
The issue persists with and without Tailscale's Magic DNS feature enabled.
To work around this issue, i run sudo systemctl stop tailscaled && sudo systemctl restart ziti-edge-tunnel before testing any ziti functions, and all is well. It's a bit janky for a tailscale/ziti mixed environment though, so I'll need to look into a more permanent solution.

Without MagicDNS:

$ resolvectl domain
Global:
Link 2 (enp4s0):
Link 3 (wlp3s0): lan
Link 10 (tailscale0):
Link 11 (ziti0):

$ resolvectl dns
Global:
Link 2 (enp4s0):
Link 3 (wlp3s0): 192.168.10.1
Link 10 (tailscale0):
Link 11 (ziti0): 100.64.0.2

With MagicDNS:

$ resolvectl domain
Global:
Link 2 (enp4s0):
Link 3 (wlp3s0): lan
Link 10 (tailscale0): tailea25.ts.net ~0.e.1.a.c.5.1.1.a.7.d.f.ip6.arpa ~100.100.in-addr.arpa ~101.100.in-addr.arpa ~102.100.in-addr.arpa ~103.100.in-addr.arpa ~104.100.in-addr.arpa ~105.100.in-addr.arpa
        ~106.100.in-addr.arpa ~107.100.in-addr.arpa ~108.100.in-addr.arpa ~109.100.in-addr.arpa ~110.100.in-addr.arpa ~111.100.in-addr.arpa ~112.100.in-addr.arpa ~113.100.in-addr.arpa ~114.100.in-addr.arpa
        ~115.100.in-addr.arpa ~116.100.in-addr.arpa ~117.100.in-addr.arpa ~118.100.in-addr.arpa ~119.100.in-addr.arpa ~120.100.in-addr.arpa ~121.100.in-addr.arpa ~122.100.in-addr.arpa ~123.100.in-addr.arpa
        ~124.100.in-addr.arpa ~125.100.in-addr.arpa ~126.100.in-addr.arpa ~127.100.in-addr.arpa ~64.100.in-addr.arpa ~65.100.in-addr.arpa ~66.100.in-addr.arpa ~67.100.in-addr.arpa ~68.100.in-addr.arpa
        ~69.100.in-addr.arpa ~70.100.in-addr.arpa ~71.100.in-addr.arpa ~72.100.in-addr.arpa ~73.100.in-addr.arpa ~74.100.in-addr.arpa ~75.100.in-addr.arpa ~76.100.in-addr.arpa ~77.100.in-addr.arpa
        ~78.100.in-addr.arpa ~79.100.in-addr.arpa ~80.100.in-addr.arpa ~81.100.in-addr.arpa ~82.100.in-addr.arpa ~83.100.in-addr.arpa ~84.100.in-addr.arpa ~85.100.in-addr.arpa ~86.100.in-addr.arpa
        ~87.100.in-addr.arpa ~88.100.in-addr.arpa ~89.100.in-addr.arpa ~90.100.in-addr.arpa ~91.100.in-addr.arpa ~92.100.in-addr.arpa ~93.100.in-addr.arpa ~94.100.in-addr.arpa ~95.100.in-addr.arpa
        ~96.100.in-addr.arpa ~97.100.in-addr.arpa ~98.100.in-addr.arpa ~99.100.in-addr.arpa ~ts.net
Link 11 (ziti0):

$ resolvectl dns
Global:
Link 2 (enp4s0):
Link 3 (wlp3s0): 192.168.10.1
Link 10 (tailscale0): 100.100.100.100
Link 11 (ziti0): 100.64.0.2

TheLumberjack · April 1, 2024, 11:42am

Howdy @papiris, lots to go overe here! Nice to see you making progress. I'll try to tackle everything in one big post but if i miss something, let me know...

Ziti Edge Developer Sandbox docs/tour links to https://openziti.github.io/ziti/config-store/overview.html which doesn't exist. Is it supposed to be "Builtin Config Types | OpenZiti"?

Yes that looks likely. I didn't find the place where that was linked yet, could you share the page that's incorrect so someone can fix the link?

Ziti Edge Tunneler for Linux seems to put a lot of useful stuff in directories that are read-only on "immutable" distros like Fedora Atomic Workstation

Thanks for this feedback. That seems like useful information to have captured in an issue. I'll get someone to look at this.

Because of this I'm not all that keen on using the containerized ziti-edge-tunnel on my daily+dev laptop, since it AFAICT requires root privileges to do its magic on the host's network.

Yes, when performing any kind of intercept-based approach, it doesn't really make sense in my opinion to run the tunneler in docker. however, if you're using the ziti-edge-tunnel with the run-host flag, it doesn't perform interception and doesn't require root/CAP_NET_ADMIN since it's really then only working as as the offload location. So, if you are using ziti-edge-tunnel for offload into the docker network space, maybe consider using run-host and then it'll make sense? If you want intercept mode -- I don't see the benefits either.

... I surmise that internally zitified apps can, in addition to the mTLS stuff of ziti, also encrypt traffic using regular SSL certs.

one of the beautiful things of ziti is that it allows you to tunnel any protocol you want. If you have a cert/key and can make a TLS or mTLS connection, you can do that over ziti. If you're interested, I made a presetnation for QCon 2023 that does exactly this using SPIFFE/SPIRE. You can find the code on GitHub and a video recoding of the presentation on YouTube here

... my laptop can't reach any of the *.ziti test services provided in the ZEDS tour

If you try to resolve the name directly using dig and the server what happens? Usually, that will be something like this:

dig @100.64.0.2 inovato.dovanet +short

You mention tailscale in the next post - yes that can definitely get in the way... Looks like disabling tailscale did get things working. You might try moving your IP range for ziti or tailscale. I expect both are using 100.64.0.0/10 (or at least starting there). For example, I just started my tunneler on the other half of the subnet by using:

sudo ./ziti-edge-tunnel run -i ./dovanet-cd.json --dns-ip-range 100.96.0.1/11

Once I do that, I can then resolve one of my services:

dig @100.96.0.2 inovato.dovanet +short
100.96.0.4

You sound like you're making great progress, let us know if/how we can help as you need it.

papiris · April 1, 2024, 11:04pm

Thanks for your quick and comprehensive reply

The page in ZEDS is this: Ziti Edge Developer Sandbox
I've switched over to using the free trial of Netfoundry's NAAS for now, since the user interface of nfconsole is easier to understand and more polished than ZEDS.
I'll make an issue about tunneler hijinx on immutable distros tomorrow
Once I get my app to production, I'll probably try to containerize and zitify as much on the server-side as possible. Components that can't be zitified might use the ziti-edge-tunnel container. For now, since my laptop works as both the server and primary client, intercept mode is necessary.
That's neat! I've had your presentation on in the background this evening while working with the zitification of my app; definitely something to try implementing as well.
Yup, right on the money. What are these 100.x.y.z addresses? · Tailscale Docs

I managed to communicate with (part of) my webapp over ziti! The bindings stuff for the decorator was really finicky to get right, maybe due to some peculiarities of my code. Fixed that, but got another issue:
My app dynamically adds routes to reach corresponding StreamingResponses when a request reaches the / route, and I think this might mess with openziti somehow?
In any case, via the browser I can reach static and predefined resources of the webapp just fine, but all video routes return "endpoint not connected"...

Relevant code in predalert.web.web_ui.py:

@app.route("/")
async def homepage(request):
    """
    Serves the homepage, with semi-dynamically defined
    video streams.

    Parameters
    ----------
        request (web request): This function triggers when
            the homepage is visited, i.e. when a request is run.
    """
    # Get routes for videos
    video_urls, video_stream_objects = await generate_video_urls(web_config)
    for url, video_stream in zip(video_urls, video_stream_objects):
        print(url, video_stream)
        app.add_route(url, video_stream)

    # Render the template with dynamic content
    return templates.TemplateResponse(
        "index.html", {"request": request, "video_urls": video_urls})

Relevant code in predalert.web.videos.py:


async def frame_producer_dynamic(q_result: Queue, web_config: dict):
    """
    Custom frame producer for FastAPI.
    Transcodes images from q_result to bytes,
    and yields them to the web process.
    Due to peculiarities with web frameworks (django, flask, uvicorn)
    when called as the target of multiprocessing,
    arguments apart from q_result must be passed through the <config> object.

    Parameters
    ----------
        q_img_grid_web (mp.Manager.Queue): Images to show on the web UI.
        q_result (mp.Manager.Queue): Analyzed, annotated images to display.
            One queue per analytics process.
        q_logging (mp.Manager.Queue): Log messages to be consumed
            by log_handler process.
        event (mp.Manager.Event): Cross-process flag, to initiate termination

    Returns
    ----------
        byte-formatted responses to facilitate network transfer,
            either a plaintext errorcode, or an encoded image.
    """
    q_logging = web_config.read_key('q_logging')
    event = web_config.read_key('event')
    while not event.is_set():
        try:
            # read frame from queue
            unencoded_image = q_result.get_nowait()
        except queue.Empty:
            await asyncio.sleep(0.1)
            continue

        except Exception as e:
            q_logging.put(("error", f"web_ui.get_frames has error: {e}"))
            await asyncio.sleep(0.001)
            yield (b"--frame\r\n"
                b"Content-Type: text/plain\r\n\r\n\r\n")

        else:
            try:
                # cv.imencode() returns ret (boolean) as well as the encoded image.
                # To use the .tobytes() function directly, only return the second.
                encoded_image = cv.imencode(".jpg", unencoded_image)[1].tobytes()
            except Exception as e:
                q_logging.put(("error", f"web_ui.get_frames has error: {e}"))
                await asyncio.sleep(0.001)
            # yield frame in byte format
            await asyncio.sleep(0.001)
            yield (b"--frame\r\n"
                b"Content-Type:image/jpeg\r\n\r\n" + encoded_image + b"\r\n")


async def dynamic_video_response(q_result: Queue) -> StreamingResponse:
    """
    Return an async video streaming response for frame_producer_dynamic().

    Parameters
    ----------
        q_result (mp.Manager.Queue): Contains analyzed,
            annotated images to be displayed.
            One queue per analytics process.

    Returns
    ----------
        StreamingResponse that continuously replaces the previous image
            in a frame with the newest one in the queue.
    """
    # assert scope["type"] in ["http", "https"]
    await asyncio.sleep(0.001)
    return StreamingResponse(
        (frame_producer_dynamic(q_result)),
        media_type="multipart/x-mixed-replace; boundary=frame",
    )


async def generate_video_urls(web_config: dict) -> list[str]:
    """
    Makes routes and streamingResponses
    for the image queue objects within
    q_result_dict.

    Parameters
    ----------
        web_config (dict):
            convenience structure to pass objects and variables between
                different classes, functions and processes.
        q_result_dict (dict of q_result):
            q_result (mp.Manager.Queue): Contains analyzed,
                annotated images to be displayed.
                One queue per analytics process.

    Returns
    ----------
        video_urls (list): Contains urls for where frontend html can GET
            analyzed, annotated image streams.
    """
    q_result_dict = web_config.read_key("q_result_dict")
    q_logging = web_config.read_key('q_logging')
    event = web_config.read_key('event')
    video_urls = []
    video_stream_objects = []
    for index, q_result in enumerate(q_result_dict.values()):
        async def video_stream(request, q_result=q_result):
            return StreamingResponse(
                frame_producer_dynamic(q_result, web_config),
                media_type='multipart/x-mixed-replace; boundary=frame')

        video_urls.append(f"/video{index}")
        video_stream_objects.append(video_stream)

    return video_urls, video_stream_objects

error log when visiting http://predalert.ziti:8443

04/01/2024 22:18:22 - DEBUG - Predalert -   video read queue for rtsp://192.168.10.12:554/12 has 3 elements
04/01/2024 22:18:22 - ERROR - asyncio -   Exception in callback BaseSelectorEventLoop._accept_connection(<function Ser...x7f99f6fbc860>, <openziti.dec...27.0.0.1', 0)>, None, <Server socke...0.0.1', 0)>,)>, 2048, None, None)
handle: <Handle BaseSelectorEventLoop._accept_connection(<function Ser...x7f99f6fbc860>, <openziti.dec...27.0.0.1', 0)>, None, <Server socke...0.0.1', 0)>,)>, 2048, None, None)>
Traceback (most recent call last):
  File "/usr/lib64/python3.11/asyncio/events.py", line 84, in _run
    self._context.run(self._callback, *self._args)
  File "/usr/lib64/python3.11/asyncio/selector_events.py", line 165, in _accept_connection
    conn, addr = sock.accept()
                 ^^^^^^^^^^^^^
  File "/var/mnt/data/jacob/git/farm-animal-detection/.env/lib64/python3.11/site-packages/openziti/zitisock.py", line 119, in accept
    fd, peer = zitilib.accept(self.fileno())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/mnt/data/jacob/git/farm-animal-detection/.env/lib64/python3.11/site-packages/openziti/zitilib.py", line 276, in accept
    check_error(clt)
  File "/var/mnt/data/jacob/git/farm-animal-detection/.env/lib64/python3.11/site-packages/openziti/zitilib.py", line 219, in check_error
    raise Exception(err, msg)
Exception: (11, 'unexpected error')
04/01/2024 22:18:23 - ERROR - asyncio -   Exception in callback BaseSelectorEventLoop._accept_connection(<function Ser...x7f99f6fbc860>, <openziti.dec...27.0.0.1', 0)>, None, <Server socke...0.0.1', 0)>,)>, 2048, None, None)
handle: <Handle BaseSelectorEventLoop._accept_connection(<function Ser...x7f99f6fbc860>, <openziti.dec...27.0.0.1', 0)>, None, <Server socke...0.0.1', 0)>,)>, 2048, None, None)>
Traceback (most recent call last):
  File "/usr/lib64/python3.11/asyncio/events.py", line 84, in _run
    self._context.run(self._callback, *self._args)
  File "/usr/lib64/python3.11/asyncio/selector_events.py", line 165, in _accept_connection
    conn, addr = sock.accept()
                 ^^^^^^^^^^^^^
  File "/var/mnt/data/jacob/git/farm-animal-detection/.env/lib64/python3.11/site-packages/openziti/zitisock.py", line 119, in accept
    fd, peer = zitilib.accept(self.fileno())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/mnt/data/jacob/git/farm-animal-detection/.env/lib64/python3.11/site-packages/openziti/zitilib.py", line 276, in accept
    check_error(clt)
  File "/var/mnt/data/jacob/git/farm-animal-detection/.env/lib64/python3.11/site-packages/openziti/zitilib.py", line 219, in check_error
    raise Exception(err, msg)
Exception: (11, 'unexpected error')
/video0 <function generate_video_urls.<locals>.video_stream at 0x7f99f6fbd3a0>
/video1 <function generate_video_urls.<locals>.video_stream at 0x7f99f6fbd440>
INFO:      - "GET / HTTP/1.1" 200 OK
04/01/2024 22:18:23 - DEBUG - Predalert -   video read queue for rtsp://192.168.10.11:554/12 has 3 elements
04/01/2024 22:18:23 - DEBUG - Predalert -   video read queue for rtsp://192.168.10.12:554/12 has 3 elements
INFO:      - "GET /video1 HTTP/1.1" 200 OK
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/var/mnt/data/jacob/git/farm-animal-detection/.env/lib64/python3.11/site-packages/starlette/responses.py", line 264, in __call__
    await wrap(partial(self.listen_for_disconnect, receive))
  File "/var/mnt/data/jacob/git/farm-animal-detection/.env/lib64/python3.11/site-packages/starlette/responses.py", line 260, in wrap
    await func()
  File "/var/mnt/data/jacob/git/farm-animal-detection/.env/lib64/python3.11/site-packages/starlette/responses.py", line 237, in listen_for_disconnect
    message = await receive()
              ^^^^^^^^^^^^^^^
  File "/var/mnt/data/jacob/git/farm-animal-detection/.env/lib64/python3.11/site-packages/uvicorn/protocols/http/h11_impl.py", line 538, in receive
    await self.message_event.wait()
  File "/usr/lib64/python3.11/asyncio/locks.py", line 213, in wait
    await fut
asyncio.exceptions.CancelledError: Cancelled by cancel scope 7f99f6e2f350

During handling of the above exception, another exception occurred:

  + Exception Group Traceback (most recent call last):
  |   File "/var/mnt/data/jacob/git/farm-animal-detection/.env/lib64/python3.11/site-packages/uvicorn/protocols/http/h11_impl.py", line 408, in run_asgi
  |     result = await app(  # type: ignore[func-returns-value]
  |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/var/mnt/data/jacob/git/farm-animal-detection/.env/lib64/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 69, in __call__
  |     return await self.app(scope, receive, send)
  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/var/mnt/data/jacob/git/farm-animal-detection/.env/lib64/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__
  |     await super().__call__(scope, receive, send)
  |   File "/var/mnt/data/jacob/git/farm-animal-detection/.env/lib64/python3.11/site-packages/starlette/applications.py", line 123, in __call__
  |     await self.middleware_stack(scope, receive, send)
  |   File "/var/mnt/data/jacob/git/farm-animal-detection/.env/lib64/python3.11/site-packages/starlette/middleware/errors.py", line 186, in __call__
  |     raise exc
  |   File "/var/mnt/data/jacob/git/farm-animal-detection/.env/lib64/python3.11/site-packages/starlette/middleware/errors.py", line 164, in __call__
  |     await self.app(scope, receive, _send)
  |   File "/var/mnt/data/jacob/git/farm-animal-detection/.env/lib64/python3.11/site-packages/starlette/middleware/cors.py", line 83, in __call__
  |     await self.app(scope, receive, send)
  |   File "/var/mnt/data/jacob/git/farm-animal-detection/.env/lib64/python3.11/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
  |     await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  |   File "/var/mnt/data/jacob/git/farm-animal-detection/.env/lib64/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
  |     raise exc
  |   File "/var/mnt/data/jacob/git/farm-animal-detection/.env/lib64/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
  |     await app(scope, receive, sender)
  |   File "/var/mnt/data/jacob/git/farm-animal-detection/.env/lib64/python3.11/site-packages/starlette/routing.py", line 758, in __call__
  |     await self.middleware_stack(scope, receive, send)
  |   File "/var/mnt/data/jacob/git/farm-animal-detection/.env/lib64/python3.11/site-packages/starlette/routing.py", line 778, in app
  |     await route.handle(scope, receive, send)
  |   File "/var/mnt/data/jacob/git/farm-animal-detection/.env/lib64/python3.11/site-packages/starlette/routing.py", line 299, in handle
  |     await self.app(scope, receive, send)
  |   File "/var/mnt/data/jacob/git/farm-animal-detection/.env/lib64/python3.11/site-packages/starlette/routing.py", line 79, in app
  |     await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  |   File "/var/mnt/data/jacob/git/farm-animal-detection/.env/lib64/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
  |     raise exc
  |   File "/var/mnt/data/jacob/git/farm-animal-detection/.env/lib64/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
  |     await app(scope, receive, sender)
  |   File "/var/mnt/data/jacob/git/farm-animal-detection/.env/lib64/python3.11/site-packages/starlette/routing.py", line 77, in app
  |     await response(scope, receive, send)
  |   File "/var/mnt/data/jacob/git/farm-animal-detection/.env/lib64/python3.11/site-packages/starlette/responses.py", line 257, in __call__
  |     async with anyio.create_task_group() as task_group:
  |   File "/var/mnt/data/jacob/git/farm-animal-detection/.env/lib64/python3.11/site-packages/anyio/_backends/_asyncio.py", line 678, in __aexit__
  |     raise BaseExceptionGroup(
  | ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
  +-+---------------- 1 ----------------
    | Traceback (most recent call last):
    |   File "/usr/lib64/python3.11/multiprocessing/managers.py", line 814, in _callmethod
    |     conn = self._tls.connection
    |            ^^^^^^^^^^^^^^^^^^^^
    | AttributeError: 'ForkAwareLocal' object has no attribute 'connection'
    | 
    | During handling of the above exception, another exception occurred:
    | 
    | Traceback (most recent call last):
    |   File "/var/mnt/data/jacob/git/farm-animal-detection/.env/lib64/python3.11/site-packages/starlette/responses.py", line 260, in wrap
    |     await func()
    |   File "/var/mnt/data/jacob/git/farm-animal-detection/.env/lib64/python3.11/site-packages/starlette/responses.py", line 249, in stream_response
    |     async for chunk in self.body_iterator:
    |   File "/var/mnt/data/jacob/git/farm-animal-detection/predalert/web/video.py", line 39, in frame_producer_dynamic
    |     while not event.is_set():
    |               ^^^^^^^^^^^^^^
    |   File "/usr/lib64/python3.11/multiprocessing/managers.py", line 1091, in is_set
    |     return self._callmethod('is_set')
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/usr/lib64/python3.11/multiprocessing/managers.py", line 818, in _callmethod
    |     self._connect()
    |   File "/usr/lib64/python3.11/multiprocessing/managers.py", line 805, in _connect
    |     conn = self._Client(self._token.address, authkey=self._authkey)
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/usr/lib64/python3.11/multiprocessing/connection.py", line 524, in Client
    |     answer_challenge(c, authkey)
    |   File "/usr/lib64/python3.11/multiprocessing/connection.py", line 768, in answer_challenge
    |     message = connection.recv_bytes(256)         # reject large message
    |               ^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/usr/lib64/python3.11/multiprocessing/connection.py", line 216, in recv_bytes
    |     buf = self._recv_bytes(maxlength)
    |           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/usr/lib64/python3.11/multiprocessing/connection.py", line 430, in _recv_bytes
    |     buf = self._recv(4)
    |           ^^^^^^^^^^^^^
    |   File "/usr/lib64/python3.11/multiprocessing/connection.py", line 395, in _recv
    |     chunk = read(handle, remaining)
    |             ^^^^^^^^^^^^^^^^^^^^^^^
    | OSError: [Errno 107] Transport endpoint is not connected
    +------------------------------------

If you'd like to take a closer look (I'm not sure if I missed anything myself), the webapp is installable like this:

git clone https://gitlab.com/papiris/predator-detect-and-notify.git
cd predator-detect-and-notify
git checkout feat-zitify
python -m venv .env
source .env/bin/activate
pip install .
vapid -g

You'll need to change values in config/config.toml to suit your environment.
Specifically, change values for alert.push.vapid_private_key to ./private_key.pem and alert.push.vapid_public_key to ./public_key.pem; and web_ui.zero_trust{service_name, identity} to a service name and the associated identity.
There're also some hardcoded paths to http://predalert.ziti:8443 in the javascripts in predalert/web/static/js/ for now.

TheLumberjack · April 2, 2024, 12:44pm

very cool! congrats!

Interesting. I'll definitely take a look as to why it was perhaps finicky (and point @ekoby here to do the same).

Assuming we're talking "http route" and not IP route, I can't imagine how but I mean, things happen... It does depend on the what/why/how's though, so maybe.

I expect some other error is happening. It kind of looks like maybe something isn't wired in correctly but I can't tell what.

Looking at your logs, I see these relevant bits:

    | Traceback (most recent call last):
    |   File "/usr/lib64/python3.11/multiprocessing/managers.py", line 814, in _callmethod
    |     conn = self._tls.connection
    |            ^^^^^^^^^^^^^^^^^^^^
    | AttributeError: 'ForkAwareLocal' object has no attribute 'connection'

due to:

OSError: [Errno 107] Transport endpoint is not connected

That last "transport endpoint is not connected" makes me think the OpenZit connection failed in some way. I dunno if you can enable more OpenZiti logging, but that'd be what I'd go for. Again I'll point @ekoby here to look at this too.

papiris · April 2, 2024, 1:44pm

hmm, I believe we're talking http routes indeed.
Here are debug ziti logs from when the web server is started, through visiting the index webpage, to shutting down the server:

april 02 15:39:08 fedora ziti-edge-tunnel[1753]: (1753)[    20928.482]   DEBUG tunnel-sdk:tunnel_udp.c:269 recv_udp() intercepted address[udp:100.64.0.2:53] client[udp:100.64.0.1:33713] service[ziti:dns-resolver]
april 02 15:39:08 fedora ziti-edge-tunnel[1753]: (1753)[    20928.482]   DEBUG tunnel-cbs:ziti_dns.c:234 on_dns_client() new DNS client
april 02 15:39:08 fedora ziti-edge-tunnel[1753]: (1753)[    20928.482]   DEBUG tunnel-sdk:ziti_tunnel.c:221 ziti_tunneler_dial_completed() ziti dial succeeded: client[udp:100.64.0.1:33713] service[ziti:dns-resolver]
april 02 15:39:08 fedora ziti-edge-tunnel[1753]: (1753)[    20928.482]   DEBUG tunnel-sdk:ziti_tunnel.c:434 ziti_tunneler_close() closing connection: client[udp:100.64.0.1:33713] service[ziti:dns-resolver]
april 02 15:39:08 fedora ziti-edge-tunnel[1753]: (1753)[    20928.482]   DEBUG tunnel-sdk:tunnel_udp.c:126 tunneler_udp_close() closing src[udp:100.64.0.1:33713] dst[udp:100.64.0.2:53] service[ziti:dns-resolver]
april 02 15:39:08 fedora ziti-edge-tunnel[1753]: (1753)[    20928.482]   DEBUG tunnel-sdk:tunnel_udp.c:269 recv_udp() intercepted address[udp:100.64.0.2:53] client[udp:100.64.0.1:54726] service[ziti:dns-resolver]
april 02 15:39:08 fedora ziti-edge-tunnel[1753]: (1753)[    20928.482]   DEBUG tunnel-cbs:ziti_dns.c:234 on_dns_client() new DNS client
april 02 15:39:08 fedora ziti-edge-tunnel[1753]: (1753)[    20928.482]   DEBUG tunnel-sdk:ziti_tunnel.c:221 ziti_tunneler_dial_completed() ziti dial succeeded: client[udp:100.64.0.1:54726] service[ziti:dns-resolver]
april 02 15:39:08 fedora ziti-edge-tunnel[1753]: (1753)[    20928.482]   DEBUG tunnel-sdk:ziti_tunnel.c:434 ziti_tunneler_close() closing connection: client[udp:100.64.0.1:54726] service[ziti:dns-resolver]
april 02 15:39:08 fedora ziti-edge-tunnel[1753]: (1753)[    20928.482]   DEBUG tunnel-sdk:tunnel_udp.c:126 tunneler_udp_close() closing src[udp:100.64.0.1:54726] dst[udp:100.64.0.2:53] service[ziti:dns-resolver]
april 02 15:39:09 fedora ziti-edge-tunnel[1753]: (1753)[    20928.895]   DEBUG ziti-sdk:ziti_ctrl.c:722 ctrl_paging_req() ctrl[96df698b-b55b-4064-a79b-108624c35043.production.netfoundry.io] starting paging request GET[/current-identity/edge-routers]
april 02 15:39:09 fedora ziti-edge-tunnel[1753]: (1753)[    20928.895]   DEBUG tunnel-sdk:tunnel_udp.c:269 recv_udp() intercepted address[udp:100.64.0.2:53] client[udp:100.64.0.1:39191] service[ziti:dns-resolver]
april 02 15:39:09 fedora ziti-edge-tunnel[1753]: (1753)[    20928.895]   DEBUG tunnel-cbs:ziti_dns.c:234 on_dns_client() new DNS client
april 02 15:39:09 fedora ziti-edge-tunnel[1753]: (1753)[    20928.895]   DEBUG tunnel-sdk:ziti_tunnel.c:221 ziti_tunneler_dial_completed() ziti dial succeeded: client[udp:100.64.0.1:39191] service[ziti:dns-resolver]
april 02 15:39:09 fedora ziti-edge-tunnel[1753]: (1753)[    20928.895]   DEBUG tunnel-sdk:ziti_tunnel.c:434 ziti_tunneler_close() closing connection: client[udp:100.64.0.1:39191] service[ziti:dns-resolver]
april 02 15:39:09 fedora ziti-edge-tunnel[1753]: (1753)[    20928.895]   DEBUG tunnel-sdk:tunnel_udp.c:126 tunneler_udp_close() closing src[udp:100.64.0.1:39191] dst[udp:100.64.0.2:53] service[ziti:dns-resolver]
april 02 15:39:09 fedora ziti-edge-tunnel[1753]: (1753)[    20928.895]   DEBUG tunnel-sdk:tunnel_udp.c:269 recv_udp() intercepted address[udp:100.64.0.2:53] client[udp:100.64.0.1:32903] service[ziti:dns-resolver]
april 02 15:39:09 fedora ziti-edge-tunnel[1753]: (1753)[    20928.895]   DEBUG tunnel-cbs:ziti_dns.c:234 on_dns_client() new DNS client
april 02 15:39:09 fedora ziti-edge-tunnel[1753]: (1753)[    20928.895]   DEBUG tunnel-sdk:ziti_tunnel.c:221 ziti_tunneler_dial_completed() ziti dial succeeded: client[udp:100.64.0.1:32903] service[ziti:dns-resolver]
april 02 15:39:09 fedora ziti-edge-tunnel[1753]: (1753)[    20928.895]   DEBUG tunnel-sdk:ziti_tunnel.c:434 ziti_tunneler_close() closing connection: client[udp:100.64.0.1:32903] service[ziti:dns-resolver]
april 02 15:39:09 fedora ziti-edge-tunnel[1753]: (1753)[    20928.895]   DEBUG tunnel-sdk:tunnel_udp.c:126 tunneler_udp_close() closing src[udp:100.64.0.1:32903] dst[udp:100.64.0.2:53] service[ziti:dns-resolver]
april 02 15:39:09 fedora ziti-edge-tunnel[1753]: (1753)[    20929.443]   DEBUG ziti-sdk:ziti_ctrl.c:330 ctrl_body_cb() ctrl[96df698b-b55b-4064-a79b-108624c35043.production.netfoundry.io] completed GET[/current-identity/edge-routers?limit=25&offset=0] in 0.548 s
april 02 15:39:09 fedora ziti-edge-tunnel[1753]: (1753)[    20929.443]   DEBUG ziti-sdk:ziti_ctrl.c:347 ctrl_body_cb() ctrl[96df698b-b55b-4064-a79b-108624c35043.production.netfoundry.io] received 1/1 for paging request GET[/current-identity/edge-routers]
april 02 15:39:09 fedora ziti-edge-tunnel[1753]: (1753)[    20929.443]   DEBUG ziti-sdk:ziti_ctrl.c:359 ctrl_body_cb() ctrl[96df698b-b55b-4064-a79b-108624c35043.production.netfoundry.io] completed paging request GET[/current-identity/edge-routers] in 0.548 s
april 02 15:39:10 fedora ziti-edge-tunnel[1753]: (1753)[    20929.536]   DEBUG tunnel-sdk:tunnel_udp.c:269 recv_udp() intercepted address[udp:100.64.0.2:53] client[udp:100.64.0.1:49793] service[ziti:dns-resolver]
april 02 15:39:10 fedora ziti-edge-tunnel[1753]: (1753)[    20929.536]   DEBUG tunnel-cbs:ziti_dns.c:234 on_dns_client() new DNS client
april 02 15:39:10 fedora ziti-edge-tunnel[1753]: (1753)[    20929.536]   DEBUG tunnel-sdk:ziti_tunnel.c:221 ziti_tunneler_dial_completed() ziti dial succeeded: client[udp:100.64.0.1:49793] service[ziti:dns-resolver]
april 02 15:39:10 fedora ziti-edge-tunnel[1753]: (1753)[    20929.536]   DEBUG tunnel-sdk:ziti_tunnel.c:434 ziti_tunneler_close() closing connection: client[udp:100.64.0.1:49793] service[ziti:dns-resolver]
april 02 15:39:10 fedora ziti-edge-tunnel[1753]: (1753)[    20929.536]   DEBUG tunnel-sdk:tunnel_udp.c:126 tunneler_udp_close() closing src[udp:100.64.0.1:49793] dst[udp:100.64.0.2:53] service[ziti:dns-resolver]
april 02 15:39:10 fedora ziti-edge-tunnel[1753]: (1753)[    20929.536]   DEBUG tunnel-sdk:tunnel_udp.c:269 recv_udp() intercepted address[udp:100.64.0.2:53] client[udp:100.64.0.1:48894] service[ziti:dns-resolver]
april 02 15:39:10 fedora ziti-edge-tunnel[1753]: (1753)[    20929.536]   DEBUG tunnel-cbs:ziti_dns.c:234 on_dns_client() new DNS client
april 02 15:39:10 fedora ziti-edge-tunnel[1753]: (1753)[    20929.536]   DEBUG tunnel-sdk:ziti_tunnel.c:221 ziti_tunneler_dial_completed() ziti dial succeeded: client[udp:100.64.0.1:48894] service[ziti:dns-resolver]
april 02 15:39:10 fedora ziti-edge-tunnel[1753]: (1753)[    20929.536]   DEBUG tunnel-sdk:ziti_tunnel.c:434 ziti_tunneler_close() closing connection: client[udp:100.64.0.1:48894] service[ziti:dns-resolver]
april 02 15:39:10 fedora ziti-edge-tunnel[1753]: (1753)[    20929.536]   DEBUG tunnel-sdk:tunnel_udp.c:126 tunneler_udp_close() closing src[udp:100.64.0.1:48894] dst[udp:100.64.0.2:53] service[ziti:dns-resolver]
april 02 15:39:10 fedora ziti-edge-tunnel[1753]: (1753)[    20929.567]   DEBUG ziti-sdk:ziti_ctrl.c:330 ctrl_body_cb() ctrl[96df698b-b55b-4064-a79b-108624c35043.production.netfoundry.io] completed GET[/current-api-session/service-updates] in 0.672 s
april 02 15:39:10 fedora ziti-edge-tunnel[1753]: (1753)[    20929.788]   DEBUG tunnel-sdk:tunnel_udp.c:269 recv_udp() intercepted address[udp:100.64.0.2:53] client[udp:100.64.0.1:41825] service[ziti:dns-resolver]
april 02 15:39:10 fedora ziti-edge-tunnel[1753]: (1753)[    20929.788]   DEBUG tunnel-cbs:ziti_dns.c:234 on_dns_client() new DNS client
april 02 15:39:10 fedora ziti-edge-tunnel[1753]: (1753)[    20929.788]   DEBUG tunnel-sdk:ziti_tunnel.c:221 ziti_tunneler_dial_completed() ziti dial succeeded: client[udp:100.64.0.1:41825] service[ziti:dns-resolver]
april 02 15:39:10 fedora ziti-edge-tunnel[1753]: (1753)[    20929.788]   DEBUG tunnel-sdk:ziti_tunnel.c:434 ziti_tunneler_close() closing connection: client[udp:100.64.0.1:41825] service[ziti:dns-resolver]
april 02 15:39:10 fedora ziti-edge-tunnel[1753]: (1753)[    20929.788]   DEBUG tunnel-sdk:tunnel_udp.c:126 tunneler_udp_close() closing src[udp:100.64.0.1:41825] dst[udp:100.64.0.2:53] service[ziti:dns-resolver]
april 02 15:39:10 fedora ziti-edge-tunnel[1753]: (1753)[    20929.788]   DEBUG tunnel-sdk:tunnel_udp.c:269 recv_udp() intercepted address[udp:100.64.0.2:53] client[udp:100.64.0.1:56727] service[ziti:dns-resolver]
april 02 15:39:10 fedora ziti-edge-tunnel[1753]: (1753)[    20929.788]   DEBUG tunnel-cbs:ziti_dns.c:234 on_dns_client() new DNS client
april 02 15:39:10 fedora ziti-edge-tunnel[1753]: (1753)[    20929.788]   DEBUG tunnel-sdk:ziti_tunnel.c:221 ziti_tunneler_dial_completed() ziti dial succeeded: client[udp:100.64.0.1:56727] service[ziti:dns-resolver]
april 02 15:39:10 fedora ziti-edge-tunnel[1753]: (1753)[    20929.788]   DEBUG tunnel-sdk:ziti_tunnel.c:434 ziti_tunneler_close() closing connection: client[udp:100.64.0.1:56727] service[ziti:dns-resolver]
april 02 15:39:10 fedora ziti-edge-tunnel[1753]: (1753)[    20929.788]   DEBUG tunnel-sdk:tunnel_udp.c:126 tunneler_udp_close() closing src[udp:100.64.0.1:56727] dst[udp:100.64.0.2:53] service[ziti:dns-resolver]
april 02 15:39:12 fedora ziti-edge-tunnel[1753]: (1753)[    20932.414]   DEBUG tunnel-sdk:tunnel_tcp.c:116 new_tcp_pcb() snd_wnd: 30845, snd_snd_max: 30845, mss: 1460
april 02 15:39:12 fedora ziti-edge-tunnel[1753]: (1753)[    20932.414]   DEBUG tunnel-sdk:tunnel_tcp.c:429 recv_tcp() intercepted address[tcp:100.64.0.8:8443] client[tcp:100.64.0.1:43386] service[predalert_ui]
april 02 15:39:12 fedora ziti-edge-tunnel[1753]: (1753)[    20932.414]   DEBUG tunnel-cbs:ziti_tunnel_cbs.c:349 ziti_sdk_c_dial() service[predalert_ui] app_data_json[172]='{"connType":null,"dst_protocol":"tcp","dst_hostname":"predalert.ziti","dst_ip":"100.64.0.8","dst_port":"8443","src_protocol":"tcp","src_ip":"100.64.0.1","src_port":"43386"}'
april 02 15:39:12 fedora ziti-edge-tunnel[1753]: (1753)[    20932.414]   DEBUG ziti-sdk:connect.c:413 connect_get_service_cb() conn[0.13/p_SseXsZ/Connecting] got service[predalert_ui] id[7RCDLQwl6oe2rqTR8mi4Ty]
april 02 15:39:12 fedora ziti-edge-tunnel[1753]: (1753)[    20932.414]   DEBUG ziti-sdk:posture.c:210 ziti_send_posture_data() ztx[0] posture checks must_send set to TRUE, new_session_id[FALSE], must_send_every_time[TRUE], new_controller_instance[FALSE]
april 02 15:39:12 fedora ziti-edge-tunnel[1753]: (1753)[    20932.414]   DEBUG ziti-sdk:connect.c:534 process_connect() conn[0.13/p_SseXsZ/Connecting] starting Dial connection for service[predalert_ui] with session[cluib1wzz1ie2copict271v1z]
april 02 15:39:12 fedora ziti-edge-tunnel[1753]: (1753)[    20932.414]   DEBUG ziti-sdk:connect.c:390 ziti_connect() conn[0.13/p_SseXsZ/Connecting] selected ch[AWS us-east-1 Edge Router 1@tls://389c3da8-4278-44d1-905c-b342c1fd8c84.production.netfoundry.io:443] for best latency(179 ms)
april 02 15:39:12 fedora ziti-edge-tunnel[1753]: (1753)[    20932.414]   DEBUG ziti-sdk:channel.c:246 ziti_channel_add_receiver() ch[2] added receiver[13]
april 02 15:39:13 fedora ziti-edge-tunnel[1753]: (1753)[    20932.664]   DEBUG tunnel-sdk:tunnel_tcp.c:116 new_tcp_pcb() snd_wnd: 30845, snd_snd_max: 30845, mss: 1460
april 02 15:39:13 fedora ziti-edge-tunnel[1753]: (1753)[    20932.664]   DEBUG tunnel-sdk:tunnel_tcp.c:429 recv_tcp() intercepted address[tcp:100.64.0.8:8443] client[tcp:100.64.0.1:43390] service[predalert_ui]
april 02 15:39:13 fedora ziti-edge-tunnel[1753]: (1753)[    20932.664]   DEBUG tunnel-cbs:ziti_tunnel_cbs.c:349 ziti_sdk_c_dial() service[predalert_ui] app_data_json[172]='{"connType":null,"dst_protocol":"tcp","dst_hostname":"predalert.ziti","dst_ip":"100.64.0.8","dst_port":"8443","src_protocol":"tcp","src_ip":"100.64.0.1","src_port":"43390"}'
april 02 15:39:13 fedora ziti-edge-tunnel[1753]: (1753)[    20932.664]   DEBUG ziti-sdk:connect.c:413 connect_get_service_cb() conn[0.14/24bC_V11/Connecting] got service[predalert_ui] id[7RCDLQwl6oe2rqTR8mi4Ty]
april 02 15:39:13 fedora ziti-edge-tunnel[1753]: (1753)[    20932.664]   DEBUG ziti-sdk:posture.c:210 ziti_send_posture_data() ztx[0] posture checks must_send set to TRUE, new_session_id[FALSE], must_send_every_time[TRUE], new_controller_instance[FALSE]
april 02 15:39:13 fedora ziti-edge-tunnel[1753]: (1753)[    20932.664]   DEBUG ziti-sdk:connect.c:534 process_connect() conn[0.14/24bC_V11/Connecting] starting Dial connection for service[predalert_ui] with session[cluib1wzz1ie2copict271v1z]
april 02 15:39:13 fedora ziti-edge-tunnel[1753]: (1753)[    20932.664]   DEBUG ziti-sdk:connect.c:390 ziti_connect() conn[0.14/24bC_V11/Connecting] selected ch[AWS us-east-1 Edge Router 1@tls://389c3da8-4278-44d1-905c-b342c1fd8c84.production.netfoundry.io:443] for best latency(179 ms)
april 02 15:39:13 fedora ziti-edge-tunnel[1753]: (1753)[    20932.664]   DEBUG ziti-sdk:channel.c:246 ziti_channel_add_receiver() ch[2] added receiver[14]
april 02 15:39:13 fedora ziti-edge-tunnel[1753]: (1753)[    20932.700]   DEBUG tunnel-sdk:ziti_tunnel.c:221 ziti_tunneler_dial_completed() ziti dial succeeded: client[tcp:100.64.0.1:43386] service[predalert_ui]
april 02 15:39:13 fedora ziti-edge-tunnel[1753]: (1753)[    20932.701]   DEBUG tunnel-sdk:tunnel_tcp.c:66 on_accept() on_accept: 0
april 02 15:39:13 fedora ziti-edge-tunnel[1753]: (1753)[    20932.955]   DEBUG tunnel-sdk:ziti_tunnel.c:221 ziti_tunneler_dial_completed() ziti dial succeeded: client[tcp:100.64.0.1:43390] service[predalert_ui]
april 02 15:39:13 fedora ziti-edge-tunnel[1753]: (1753)[    20932.957]   DEBUG tunnel-sdk:tunnel_tcp.c:66 on_accept() on_accept: 0
april 02 15:39:14 fedora ziti-edge-tunnel[1753]: (1753)[    20933.709]   DEBUG tunnel-cbs:ziti_tunnel_cbs.c:126 on_ziti_data() ziti connection sent EOF (ziti_eof=0, tnlr_eof=0)
april 02 15:39:14 fedora ziti-edge-tunnel[1753]: (1753)[    20933.709]   DEBUG tunnel-sdk:ziti_tunnel.c:465 ziti_tunneler_close_write() closing write connection: client[tcp:100.64.0.1:43386] service[predalert_ui]
april 02 15:39:14 fedora ziti-edge-tunnel[1753]: (1753)[    20933.709]   DEBUG tunnel-sdk:tunnel_tcp.c:235 tunneler_tcp_close_write() closing write src[tcp:100.64.0.1:43386] dst[tcp:100.64.0.8:8443] state[4/ESTABLISHED] flags[0x100] service[predalert_ui]
april 02 15:39:14 fedora ziti-edge-tunnel[1753]: (1753)[    20933.709]   DEBUG tunnel-sdk:tunnel_tcp.c:241 tunneler_tcp_close_write() closed write src[tcp:100.64.0.1:43386] dst[tcp:100.64.0.8:8443] state[5/FIN_WAIT_1] flags[0x120] service[predalert_ui]
april 02 15:39:14 fedora ziti-edge-tunnel[1753]: (1753)[    20933.709]   DEBUG tunnel-sdk:tunnel_tcp.c:145 on_tcp_client_data() client sent FIN: client=tcp:100.64.0.1:43386, service=predalert_ui
april 02 15:39:14 fedora ziti-edge-tunnel[1753]: (1753)[    20933.709]   DEBUG tunnel-sdk:tunnel_tcp.c:146 on_tcp_client_data() FIN received src[tcp:100.64.0.1:43386] dst[tcp:100.64.0.8:8443] state[10/TIME_WAIT] flags[0x123] service[predalert_ui]
april 02 15:39:14 fedora ziti-edge-tunnel[1753]: (1753)[    20933.709]   DEBUG tunnel-cbs:ziti_tunnel_cbs.c:157 ziti_sdk_c_close_write() closing ziti_conn tnlr_eof=0, ziti_eof=1
april 02 15:39:14 fedora ziti-edge-tunnel[1753]: (1753)[    20933.709]   DEBUG tunnel-cbs:ziti_tunnel_cbs.c:160 ziti_sdk_c_close_write() closing ziti_conn tnlr_eof=1, ziti_eof=1
april 02 15:39:14 fedora ziti-edge-tunnel[1753]: (1753)[    20933.709]   DEBUG ziti-sdk:channel.c:253 ziti_channel_rem_receiver() ch[2] removed receiver[13]
april 02 15:39:14 fedora ziti-edge-tunnel[1753]: (1753)[    20933.709]   DEBUG ziti-sdk:connect.c:172 close_conn_internal() conn[0.13/p_SseXsZ/Closed] removing
april 02 15:39:14 fedora ziti-edge-tunnel[1753]: (1753)[    20933.709]   DEBUG tunnel-sdk:ziti_tunnel.c:434 ziti_tunneler_close() closing connection: client[tcp:100.64.0.1:43386] service[predalert_ui]
april 02 15:39:14 fedora ziti-edge-tunnel[1753]: (1753)[    20933.709]   DEBUG tunnel-sdk:tunnel_tcp.c:251 tunneler_tcp_close() closing src[tcp:100.64.0.1:43386] dst[tcp:100.64.0.8:8443] state[10/TIME_WAIT] flags[0x120] service[predalert_ui]
april 02 15:39:14 fedora ziti-edge-tunnel[1753]: (1753)[    20933.709]   DEBUG tunnel-sdk:tunnel_tcp.c:267 tunneler_tcp_close() closed src[] dst[] state[10/TIME_WAIT] flags[0x130] service[]
april 02 15:39:14 fedora ziti-edge-tunnel[1753]: (1753)[    20933.709]   DEBUG ziti-sdk:ziti.c:1749 grim_reaper() ztx[0] reaped 1 closed (out of 2 total) connections
april 02 15:39:14 fedora ziti-edge-tunnel[1753]: (1753)[    20933.822]   DEBUG tunnel-cbs:ziti_tunnel_cbs.c:126 on_ziti_data() ziti connection sent EOF (ziti_eof=0, tnlr_eof=0)
april 02 15:39:14 fedora ziti-edge-tunnel[1753]: (1753)[    20933.822]   DEBUG tunnel-sdk:ziti_tunnel.c:465 ziti_tunneler_close_write() closing write connection: client[tcp:100.64.0.1:43390] service[predalert_ui]
april 02 15:39:14 fedora ziti-edge-tunnel[1753]: (1753)[    20933.822]   DEBUG tunnel-sdk:tunnel_tcp.c:235 tunneler_tcp_close_write() closing write src[tcp:100.64.0.1:43390] dst[tcp:100.64.0.8:8443] state[4/ESTABLISHED] flags[0x100] service[predalert_ui]
april 02 15:39:14 fedora ziti-edge-tunnel[1753]: (1753)[    20933.822]   DEBUG tunnel-sdk:tunnel_tcp.c:241 tunneler_tcp_close_write() closed write src[tcp:100.64.0.1:43390] dst[tcp:100.64.0.8:8443] state[5/FIN_WAIT_1] flags[0x120] service[predalert_ui]
april 02 15:39:14 fedora ziti-edge-tunnel[1753]: (1753)[    20933.823]   DEBUG tunnel-sdk:tunnel_tcp.c:145 on_tcp_client_data() client sent FIN: client=tcp:100.64.0.1:43390, service=predalert_ui
april 02 15:39:14 fedora ziti-edge-tunnel[1753]: (1753)[    20933.823]   DEBUG tunnel-sdk:tunnel_tcp.c:146 on_tcp_client_data() FIN received src[tcp:100.64.0.1:43390] dst[tcp:100.64.0.8:8443] state[10/TIME_WAIT] flags[0x123] service[predalert_ui]
april 02 15:39:14 fedora ziti-edge-tunnel[1753]: (1753)[    20933.823]   DEBUG tunnel-cbs:ziti_tunnel_cbs.c:157 ziti_sdk_c_close_write() closing ziti_conn tnlr_eof=0, ziti_eof=1
april 02 15:39:14 fedora ziti-edge-tunnel[1753]: (1753)[    20933.823]   DEBUG tunnel-cbs:ziti_tunnel_cbs.c:160 ziti_sdk_c_close_write() closing ziti_conn tnlr_eof=1, ziti_eof=1
april 02 15:39:14 fedora ziti-edge-tunnel[1753]: (1753)[    20933.823]   DEBUG ziti-sdk:channel.c:253 ziti_channel_rem_receiver() ch[2] removed receiver[14]
april 02 15:39:14 fedora ziti-edge-tunnel[1753]: (1753)[    20933.823]   DEBUG ziti-sdk:connect.c:172 close_conn_internal() conn[0.14/24bC_V11/Closed] removing
april 02 15:39:14 fedora ziti-edge-tunnel[1753]: (1753)[    20933.823]   DEBUG tunnel-sdk:ziti_tunnel.c:434 ziti_tunneler_close() closing connection: client[tcp:100.64.0.1:43390] service[predalert_ui]
april 02 15:39:14 fedora ziti-edge-tunnel[1753]: (1753)[    20933.823]   DEBUG tunnel-sdk:tunnel_tcp.c:251 tunneler_tcp_close() closing src[tcp:100.64.0.1:43390] dst[tcp:100.64.0.8:8443] state[10/TIME_WAIT] flags[0x120] service[predalert_ui]
april 02 15:39:14 fedora ziti-edge-tunnel[1753]: (1753)[    20933.823]   DEBUG tunnel-sdk:tunnel_tcp.c:267 tunneler_tcp_close() closed src[] dst[] state[10/TIME_WAIT] flags[0x130] service[]
april 02 15:39:14 fedora ziti-edge-tunnel[1753]: (1753)[    20933.823]   DEBUG ziti-sdk:ziti.c:1749 grim_reaper() ztx[0] reaped 1 closed (out of 1 total) connections
april 02 15:39:15 fedora ziti-edge-tunnel[1753]: (1753)[    20935.347]   DEBUG ziti-sdk:ziti_ctrl.c:722 ctrl_paging_req() ctrl[7ce7e424-6a92-4ff2-9459-ebbba32346fa.production.netfoundry.io] starting paging request GET[/current-identity/edge-routers]
april 02 15:39:15 fedora ziti-edge-tunnel[1753]: (1753)[    20935.348]   DEBUG tunnel-sdk:tunnel_udp.c:269 recv_udp() intercepted address[udp:100.64.0.2:53] client[udp:100.64.0.1:52793] service[ziti:dns-resolver]
april 02 15:39:15 fedora ziti-edge-tunnel[1753]: (1753)[    20935.348]   DEBUG tunnel-cbs:ziti_dns.c:234 on_dns_client() new DNS client
april 02 15:39:15 fedora ziti-edge-tunnel[1753]: (1753)[    20935.348]   DEBUG tunnel-sdk:ziti_tunnel.c:221 ziti_tunneler_dial_completed() ziti dial succeeded: client[udp:100.64.0.1:52793] service[ziti:dns-resolver]
april 02 15:39:15 fedora ziti-edge-tunnel[1753]: (1753)[    20935.348]   DEBUG tunnel-sdk:ziti_tunnel.c:434 ziti_tunneler_close() closing connection: client[udp:100.64.0.1:52793] service[ziti:dns-resolver]
april 02 15:39:15 fedora ziti-edge-tunnel[1753]: (1753)[    20935.348]   DEBUG tunnel-sdk:tunnel_udp.c:126 tunneler_udp_close() closing src[udp:100.64.0.1:52793] dst[udp:100.64.0.2:53] service[ziti:dns-resolver]
april 02 15:39:15 fedora ziti-edge-tunnel[1753]: (1753)[    20935.348]   DEBUG tunnel-sdk:tunnel_udp.c:269 recv_udp() intercepted address[udp:100.64.0.2:53] client[udp:100.64.0.1:35674] service[ziti:dns-resolver]
april 02 15:39:15 fedora ziti-edge-tunnel[1753]: (1753)[    20935.348]   DEBUG tunnel-cbs:ziti_dns.c:234 on_dns_client() new DNS client
april 02 15:39:15 fedora ziti-edge-tunnel[1753]: (1753)[    20935.348]   DEBUG tunnel-sdk:ziti_tunnel.c:221 ziti_tunneler_dial_completed() ziti dial succeeded: client[udp:100.64.0.1:35674] service[ziti:dns-resolver]
april 02 15:39:15 fedora ziti-edge-tunnel[1753]: (1753)[    20935.348]   DEBUG tunnel-sdk:ziti_tunnel.c:434 ziti_tunneler_close() closing connection: client[udp:100.64.0.1:35674] service[ziti:dns-resolver]
april 02 15:39:15 fedora ziti-edge-tunnel[1753]: (1753)[    20935.348]   DEBUG tunnel-sdk:tunnel_udp.c:126 tunneler_udp_close() closing src[udp:100.64.0.1:35674] dst[udp:100.64.0.2:53] service[ziti:dns-resolver]
april 02 15:39:16 fedora ziti-edge-tunnel[1753]: (1753)[    20935.890]   DEBUG ziti-sdk:ziti_ctrl.c:330 ctrl_body_cb() ctrl[7ce7e424-6a92-4ff2-9459-ebbba32346fa.production.netfoundry.io] completed GET[/current-identity/edge-routers?limit=25&offset=0] in 0.542 s
april 02 15:39:16 fedora ziti-edge-tunnel[1753]: (1753)[    20935.890]   DEBUG ziti-sdk:ziti_ctrl.c:347 ctrl_body_cb() ctrl[7ce7e424-6a92-4ff2-9459-ebbba32346fa.production.netfoundry.io] received 2/2 for paging request GET[/current-identity/edge-routers]
april 02 15:39:16 fedora ziti-edge-tunnel[1753]: (1753)[    20935.890]   DEBUG ziti-sdk:ziti_ctrl.c:359 ctrl_body_cb() ctrl[7ce7e424-6a92-4ff2-9459-ebbba32346fa.production.netfoundry.io] completed paging request GET[/current-identity/edge-routers] in 0.543 s
april 02 15:39:16 fedora ziti-edge-tunnel[1753]: (1753)[    20936.013]   DEBUG ziti-sdk:ziti_ctrl.c:330 ctrl_body_cb() ctrl[7ce7e424-6a92-4ff2-9459-ebbba32346fa.production.netfoundry.io] completed GET[/current-api-session/service-updates] in 0.666 s
april 02 15:39:16 fedora ziti-edge-tunnel[1753]: (1753)[    20936.321]   DEBUG tunnel-sdk:tunnel_udp.c:269 recv_udp() intercepted address[udp:100.64.0.2:53] client[udp:100.64.0.1:52397] service[ziti:dns-resolver]
april 02 15:39:16 fedora ziti-edge-tunnel[1753]: (1753)[    20936.321]   DEBUG tunnel-cbs:ziti_dns.c:234 on_dns_client() new DNS client
april 02 15:39:16 fedora ziti-edge-tunnel[1753]: (1753)[    20936.321]   DEBUG tunnel-sdk:ziti_tunnel.c:221 ziti_tunneler_dial_completed() ziti dial succeeded: client[udp:100.64.0.1:52397] service[ziti:dns-resolver]
april 02 15:39:16 fedora ziti-edge-tunnel[1753]: (1753)[    20936.321]   DEBUG tunnel-sdk:ziti_tunnel.c:434 ziti_tunneler_close() closing connection: client[udp:100.64.0.1:52397] service[ziti:dns-resolver]
april 02 15:39:16 fedora ziti-edge-tunnel[1753]: (1753)[    20936.321]   DEBUG tunnel-sdk:tunnel_udp.c:126 tunneler_udp_close() closing src[udp:100.64.0.1:52397] dst[udp:100.64.0.2:53] service[ziti:dns-resolver]
april 02 15:39:16 fedora ziti-edge-tunnel[1753]: (1753)[    20936.321]   DEBUG tunnel-sdk:tunnel_udp.c:269 recv_udp() intercepted address[udp:100.64.0.2:53] client[udp:100.64.0.1:60785] service[ziti:dns-resolver]
april 02 15:39:16 fedora ziti-edge-tunnel[1753]: (1753)[    20936.321]   DEBUG tunnel-cbs:ziti_dns.c:234 on_dns_client() new DNS client
april 02 15:39:16 fedora ziti-edge-tunnel[1753]: (1753)[    20936.321]   DEBUG tunnel-sdk:ziti_tunnel.c:221 ziti_tunneler_dial_completed() ziti dial succeeded: client[udp:100.64.0.1:60785] service[ziti:dns-resolver]
april 02 15:39:16 fedora ziti-edge-tunnel[1753]: (1753)[    20936.321]   DEBUG tunnel-sdk:ziti_tunnel.c:434 ziti_tunneler_close() closing connection: client[udp:100.64.0.1:60785] service[ziti:dns-resolver]
april 02 15:39:16 fedora ziti-edge-tunnel[1753]: (1753)[    20936.321]   DEBUG tunnel-sdk:tunnel_udp.c:126 tunneler_udp_close() closing src[udp:100.64.0.1:60785] dst[udp:100.64.0.2:53] service[ziti:dns-resolver]
april 02 15:39:17 fedora ziti-edge-tunnel[1753]: (1753)[    20937.418]   DEBUG ziti-sdk:posture.c:210 ziti_send_posture_data() ztx[2] posture checks must_send set to TRUE, new_session_id[FALSE], must_send_every_time[TRUE], new_controller_instance[FALSE]
april 02 15:39:17 fedora ziti-edge-tunnel[1753]: (1753)[    20937.475]   DEBUG ziti-sdk:posture.c:210 ziti_send_posture_data() ztx[0] posture checks must_send set to TRUE, new_session_id[FALSE], must_send_every_time[TRUE], new_controller_instance[FALSE]
april 02 15:39:20 fedora ziti-edge-tunnel[1753]: (1753)[    20939.567]   DEBUG ziti-sdk:ziti_ctrl.c:722 ctrl_paging_re

TheLumberjack · April 2, 2024, 2:33pm

I feel like I'm a little bit lost at the moment and could use a refreshed architectural diagram for what you're actually doing right now, if it's easy to produce? I went back and referenced the one from before (in the OP) but I'm not sure it's still up to date, is it? It's probalby close, I just want to know where your tunneler is etc. You're running a tunneler locally, but you're also running whatever client locally (is it purely underlay based or is it also application embedded?) and you're running the openziti-enabled python server somewhere too... Right? I'm not sure I fully understand the lay of the land with this particular issue...

Web Browser <-- standard TCP underlay --> OpenZiti enabled Python Server <-- remote cam resource --> ziti-edge-tunnel

is that the general view/flow? I'm wondering if maybe the ziti edge tunnel is fighting with the python server somehow? Do you have a real basic diagram? Or is "server with k3s/NVR/UI" the python server?

papiris · April 2, 2024, 4:22pm

Apologies if I'm not being clear enough, I hope this updated architectural diagram showing how things are connected during development, helps

The connection basically does a round trip from the client (navigating browser to predalert.ziti domain, intercepted by edge-tunneler), via the Ziti fabric, to the python webapp deployed as a ziti service using its embedded ziti Python SDK. Client and server are both on the same device, just using different identities.

The application uses the Python SDK, and to access the webUI via the browser, the laptop also runs an edge-tunneler.

TheLumberjack · April 2, 2024, 10:10pm

Yes that helps a lot. So your local browser is trying to access a "StreamingResponses", which I expect is the camera stream, right? So in your browser you go to "http:predalert.ziti" or do you go to "http://predalert.ziti/stream1"? That request is intercepted by the ziti-edge-tunnel, sent to the edge router (provisioned by NetFoundry) out on the internet, which is then routed back to your network space to the zitified webapp which is a wholly different identity... I think I got it/most of it.

At that point, i assume your webapp turns around and makes an underlay connection to the streaming resource (the ip camera) and pipes the response back through ziti, returning towards your browser where the browser will display the video. Assuming I have all that straight, it makes sense...

When you access static content on the web server, it works fine, but accessing that dynamic endpoint leads to an error. When you access the static content with debugging turned on, do you see any positive log messages from ziti? I'd expect you to see something like "accepting a connection" or something along those lines. Let's start making sure you see that positive message when accessing static resources. Once you see that, move over to the dynamic resource and see if you see the same sort of thing.

Looking through your ziti-edge-tunnel logs just now I see that it looks like UDP is being tunneled. Can you verify how you made the service? Did you make sure to intercept UDP and TCP (or just UDP) and did you "forward" the protocol or did you maybe forget and are sending 'udp' traffic to a TCP endpoint?

Can you provide the ziti cli commands you used to make the two configs and the service (i'm pretty sure you have the policies setup properly, don't need those)... I'm wondering if you're intercepting TCP/UDP and sending TCP only (which would break udp) or something like that...

Let's start by looking at that service and config setup...

papiris · April 3, 2024, 5:57pm

Yes that's right.
In the browser I go to "http://predalert.ziti", a static website which embeds "http://predalert.ziti/video1" in an < img> tag.
Seems like you've got the crux of it

The service was created in NFconsole, applying some attributes that coincide with both server and laptopclient identities. I didn't use the ziti cli at all, and after installing the ziti binary now, the binary can't seem to "discover" the running edge-tunnel, nor will it let me specify it to use the existing identity in /etc/openziti/identities/laptop_client1.json to investigate. I suppose ziti binary needs to be initialized with the relevant identity, in order to provide any useful information?

$ sudo .local/bin/ziti agent list
╭─────┬────────────┬────────┬─────────────┬──────────┬─────────────┬───────────╮
│ PID │ EXECUTABLE │ APP ID │ UNIX SOCKET │ APP TYPE │ APP VERSION │ APP ALIAS │
├─────┼────────────┼────────┼─────────────┼──────────┼─────────────┼───────────┤
╰─────┴────────────┴────────┴─────────────┴──────────┴─────────────┴───────────╯
$ ziti agent ps
ps is running
p.Path is: /var/home/Jacob/.local/bin/ziti
1927432 1883591 ziti  go1.22.1 /var/home/Jacob/.local/bin/ziti

$ sudo .local/bin/ziti edge list configs -i /etc/openziti/identities/laptop_client1.json
error: no identity '/etc/openziti/identities/laptop_client1.json' found in cli config /root/.config/ziti/ziti-cli.json

$ ziti edge list configs -i laptop_client1.json
error: no identity 'laptop_client1.json' found in cli config /home/Jacob/.config/ziti/ziti-cli.json
                                                                                                                                                                       $ cat /home/Jacob/.config/ziti/ziti-cli.json
cat: /home/Jacob/.config/ziti/ziti-cli.json: No such file or directory

Since NFconsole abstracts ziti cli commands used to make identities, policies, configs and such away (and doesn't seem to provide a convenient way to export the current configuration); I'm not sure what to do to get more verbose information.

This is the tunneler config, if that helps:

$ cat /var/lib/ziti/config.json
{
	"Active":true,
	"Duration":0,
	"StartTime":"2024-03-30T11:30:09.078533Z",
	"Identities":[{
		"Name":"laptopClient1",
		"Identifier":"laptopClient1.json",
		"FingerPrint":"laptopClient1",
		"Active":false,
		"Loaded":false,
		"IdFileStatus":false,
		"MfaEnabled":false,
		"MfaNeeded":false,
		"Metrics":{
			"Up":0,
			"Down":0
			},
		"MfaMinTimeout":0,
		"MfaMaxTimeout":0,
		"MfaMinTimeoutRem":0,
		"MfaMaxTimeoutRem":0,
		"MinTimeoutRemInSvcEvent":0,
		"MaxTimeoutRemInSvcEvent":0,
		"Deleted":false,
		"Notified":false
		}],
	"IpInfo":{
		"Ip":"100.64.0.1",
		"Subnet":"255.192.0.0",
		"MTU":65535,
		"DNS":"100.64.0.2"
		},
	"LogLevel":"info",
	"ServiceVersion":{
		"Version":"v0.22.26",
		"Revision":"local",
		"BuildDate":"Wed-03/27/2024-17:02:35-UTC"
		},
	"TunIpv4":"100.64.0.1",
	"TunIpv4Mask":10,
	"AddDns":false,
	"ApiPageSize":25
	}

I disabled all dynamically loaded/generated content and routes on the site, but this error message in the server process persists:

INFO:      - "GET / HTTP/1.1" 200 OK
INFO:      - "GET /static/css/custom.css HTTP/1.1" 200 OK
INFO:      - "GET /static/js/main.js HTTP/1.1" 200 OK
INFO:      - "GET /static/img/favicon-32x32.png HTTP/1.1" 200 OK
04/03/2024 19:45:53 - ERROR - asyncio -   Exception in callback BaseSelectorEventLoop._accept_connection(<function Ser...x7f3a293cc900>, <openziti.dec...27.0.0.1', 0)>, None, <Server socke...0.0.1', 0)>,)>, 2048, None, None)
handle: <Handle BaseSelectorEventLoop._accept_connection(<function Ser...x7f3a293cc900>, <openziti.dec...27.0.0.1', 0)>, None, <Server socke...0.0.1', 0)>,)>, 2048, None, None)>
Traceback (most recent call last):
  File "/usr/lib64/python3.11/asyncio/events.py", line 84, in _run
    self._context.run(self._callback, *self._args)
  File "/usr/lib64/python3.11/asyncio/selector_events.py", line 165, in _accept_connection
    conn, addr = sock.accept()
                 ^^^^^^^^^^^^^
  File "/var/mnt/data/jacob/git/farm-animal-detection/.env/lib64/python3.11/site-packages/openziti/zitisock.py", line 119, in accept
    fd, peer = zitilib.accept(self.fileno())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/mnt/data/jacob/git/farm-animal-detection/.env/lib64/python3.11/site-packages/openziti/zitilib.py", line 276, in accept
    check_error(clt)
  File "/var/mnt/data/jacob/git/farm-animal-detection/.env/lib64/python3.11/site-packages/openziti/zitilib.py", line 219, in check_error
    raise Exception(err, msg)
Exception: (11, 'unexpected error')
04/03/2024 19:45:53 - ERROR - asyncio -   Exception in callback BaseSelectorEventLoop._accept_connection(<function Ser...x7f3a293cc900>, <openziti.dec...27.0.0.1', 0)>, None, <Server socke...0.0.1', 0)>,)>, 2048, None, None)
handle: <Handle BaseSelectorEventLoop._accept_connection(<function Ser...x7f3a293cc900>, <openziti.dec...27.0.0.1', 0)>, None, <Server socke...0.0.1', 0)>,)>, 2048, None, None)>
Traceback (most recent call last):
  File "/usr/lib64/python3.11/asyncio/events.py", line 84, in _run
    self._context.run(self._callback, *self._args)
  File "/usr/lib64/python3.11/asyncio/selector_events.py", line 165, in _accept_connection
    conn, addr = sock.accept()
                 ^^^^^^^^^^^^^
  File "/var/mnt/data/jacob/git/farm-animal-detection/.env/lib64/python3.11/site-packages/openziti/zitisock.py", line 119, in accept
    fd, peer = zitilib.accept(self.fileno())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/mnt/data/jacob/git/farm-animal-detection/.env/lib64/python3.11/site-packages/openziti/zitilib.py", line 276, in accept
    check_error(clt)
  File "/var/mnt/data/jacob/git/farm-animal-detection/.env/lib64/python3.11/site-packages/openziti/zitilib.py", line 219, in check_error
    raise Exception(err, msg)
Exception: (11, 'unexpected error')
INFO:      - "GET / HTTP/1.1" 200 OK
INFO:      - "GET /static/css/custom.css HTTP/1.1" 200 OK
INFO:      - "GET /static/js/main.js HTTP/1.1" 200 OK
INFO:      - "GET /static/img/favicon-32x32.png HTTP/1.1" 200 OK

This is the journal of ziti-edge-tunnel with INFO-level. Contrary to demo ziti services which have port-ranges and protocols defined, the intercept for predalert service only has port defined.

Apr 03 19:51:21 fedora ziti-edge-tunnel[1935247]: (1935247)[        0.000]    INFO ziti-sdk:utils.c:199 ziti_log_set_level() set log level: root=3/INFO
Apr 03 19:51:21 fedora ziti-edge-tunnel[1935247]: (1935247)[        0.000]    INFO tunnel-sdk:ziti_tunnel.c:60 create_tunneler_ctx() Ziti Tunneler SDK (v0.22.26-local)
Apr 03 19:51:21 fedora ziti-edge-tunnel[1935247]: (1935247)[        0.000]    INFO tunnel-cbs:ziti_dns.c:164 seed_dns() DNS configured with range 100.96.0.0 - 100.127.255.255 (2097150 ips)
Apr 03 19:51:21 fedora ziti-edge-tunnel[1935247]: (1935247)[        0.000]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:1716 run_tunneler_loop() Loading identity files from /etc/openziti/identities
Apr 03 19:51:21 fedora ziti-edge-tunnel[1935247]: (1935247)[        0.000]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:1120 load_identities() loading identity file: laptopClient1.json
Apr 03 19:51:21 fedora ziti-edge-tunnel[1935247]: (1935247)[        0.000]    WARN ziti-edge-tunnel:instance.c:40 find_tunnel_identity() Identity ztx[/etc/openziti/identities/laptopClient1.json] is not loaded yet or already removed.
Apr 03 19:51:21 fedora ziti-edge-tunnel[1935247]: (1935247)[        0.000]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:1120 load_identities() loading identity file: laptop_client.json
Apr 03 19:51:21 fedora ziti-edge-tunnel[1935247]: (1935247)[        0.000]    WARN ziti-edge-tunnel:instance.c:40 find_tunnel_identity() Identity ztx[/etc/openziti/identities/laptop_client.json] is not loaded yet or already removed.
Apr 03 19:51:21 fedora ziti-edge-tunnel[1935247]: (1935247)[        0.000]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:1120 load_identities() loading identity file: laptop_client1.json
Apr 03 19:51:21 fedora ziti-edge-tunnel[1935247]: (1935247)[        0.000]    WARN ziti-edge-tunnel:instance.c:40 find_tunnel_identity() Identity ztx[/etc/openziti/identities/laptop_client1.json] is not loaded yet or already removed.
Apr 03 19:51:21 fedora ziti-edge-tunnel[1935247]: (1935247)[        0.000]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:1642 make_socket_path() effective group set to 'ziti' (gid=965)
Apr 03 19:51:21 fedora ziti-edge-tunnel[1935247]: (1935247)[        0.057]    INFO ziti-edge-tunnel:resolvers.c:68 init_libsystemd() Initializing libsystemd
Apr 03 19:51:21 fedora ziti-edge-tunnel[1935247]: (1935247)[        0.057]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:907 load_ziti_async() attempting to load ziti instance[/etc/openziti/identities/laptop_client1.json]
Apr 03 19:51:21 fedora ziti-edge-tunnel[1935247]: (1935247)[        0.057]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:914 load_ziti_async() loading ziti instance[/etc/openziti/identities/laptop_client1.json]
Apr 03 19:51:21 fedora ziti-edge-tunnel[1935247]: (1935247)[        0.057]    INFO ziti-edge-tunnel:resolvers.c:356 try_libsystemd_resolver() systemd-resolved selected as DNS resolver manager
Apr 03 19:51:21 fedora ziti-edge-tunnel[1935247]: (1935247)[        0.057]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:1135 load_id_cb() identity[/etc/openziti/identities/laptop_client1.json] loaded
Apr 03 19:51:21 fedora ziti-edge-tunnel[1935247]: (1935247)[        0.057]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:907 load_ziti_async() attempting to load ziti instance[/etc/openziti/identities/laptop_client.json]
Apr 03 19:51:21 fedora ziti-edge-tunnel[1935247]: (1935247)[        0.057]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:914 load_ziti_async() loading ziti instance[/etc/openziti/identities/laptop_client.json]
Apr 03 19:51:21 fedora ziti-edge-tunnel[1935247]: (1935247)[        0.057]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:1135 load_id_cb() identity[/etc/openziti/identities/laptop_client.json] loaded
Apr 03 19:51:21 fedora ziti-edge-tunnel[1935247]: (1935247)[        0.057]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:907 load_ziti_async() attempting to load ziti instance[/etc/openziti/identities/laptopClient1.json]
Apr 03 19:51:21 fedora ziti-edge-tunnel[1935247]: (1935247)[        0.057]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:914 load_ziti_async() loading ziti instance[/etc/openziti/identities/laptopClient1.json]
Apr 03 19:51:21 fedora ziti-edge-tunnel[1935247]: (1935247)[        0.057]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:1135 load_id_cb() identity[/etc/openziti/identities/laptopClient1.json] loaded
Apr 03 19:51:21 fedora ziti-edge-tunnel[1935247]: (1935247)[        0.057]   ERROR ziti-edge-tunnel:instance-config.c:136 save_tunnel_status_to_file() Could not copy config file [/var/lib/ziti/config.json] to backup config file, the config might not exists at the moment
Apr 03 19:51:21 fedora ziti-edge-tunnel[1935247]: (1935247)[        0.057]   ERROR ziti-edge-tunnel:instance-config.c:142 save_tunnel_status_to_file() Could not open config file /var/lib/ziti/config.json to store the tunnel status data
Apr 03 19:51:21 fedora ziti-edge-tunnel[1935247]: (1935247)[        0.086]    INFO ziti-sdk:ziti.c:454 ziti_init_async() ztx[0] using tlsuv[v0.28.4], tls[Mbed TLS 2.28.5]
Apr 03 19:51:21 fedora ziti-edge-tunnel[1935247]: (1935247)[        0.086]    INFO ziti-sdk:ziti.c:456 ziti_init_async() ztx[0] Loading ziti context with controller[https://96df698b-b55b-4064-a79b-108624c35043.production.netfoundry.io:443]
Apr 03 19:51:21 fedora ziti-edge-tunnel[1935247]: (1935247)[        0.086]    INFO ziti-sdk:ziti.c:933 ziti_re_auth_with_cb() ztx[0] starting to re-auth with ctrl[https://96df698b-b55b-4064-a79b-108624c35043.production.netfoundry.io:443] api_session_status[0] api_session_expired[TRUE]
Apr 03 19:51:21 fedora ziti-edge-tunnel[1935247]: (1935247)[        0.086]    INFO ziti-sdk:ziti.c:454 ziti_init_async() ztx[1] using tlsuv[v0.28.4], tls[Mbed TLS 2.28.5]
Apr 03 19:51:21 fedora ziti-edge-tunnel[1935247]: (1935247)[        0.086]    INFO ziti-sdk:ziti.c:456 ziti_init_async() ztx[1] Loading ziti context with controller[https://96df698b-b55b-4064-a79b-108624c35043.production.netfoundry.io:443]
Apr 03 19:51:21 fedora ziti-edge-tunnel[1935247]: (1935247)[        0.086]    INFO ziti-sdk:ziti.c:933 ziti_re_auth_with_cb() ztx[1] starting to re-auth with ctrl[https://96df698b-b55b-4064-a79b-108624c35043.production.netfoundry.io:443] api_session_status[0] api_session_expired[TRUE]
Apr 03 19:51:21 fedora ziti-edge-tunnel[1935247]: (1935247)[        0.086]    INFO ziti-sdk:ziti.c:454 ziti_init_async() ztx[2] using tlsuv[v0.28.4], tls[Mbed TLS 2.28.5]
Apr 03 19:51:21 fedora ziti-edge-tunnel[1935247]: (1935247)[        0.086]    INFO ziti-sdk:ziti.c:456 ziti_init_async() ztx[2] Loading ziti context with controller[https://7ce7e424-6a92-4ff2-9459-ebbba32346fa.production.netfoundry.io:443]
Apr 03 19:51:21 fedora ziti-edge-tunnel[1935247]: (1935247)[        0.086]    INFO ziti-sdk:ziti.c:933 ziti_re_auth_with_cb() ztx[2] starting to re-auth with ctrl[https://7ce7e424-6a92-4ff2-9459-ebbba32346fa.production.netfoundry.io:443] api_session_status[0] api_session_expired[TRUE]
Apr 03 19:51:21 fedora ziti-edge-tunnel[1935247]: (1935247)[        0.739]    INFO ziti-sdk:ziti.c:1698 version_cb() ztx[0] connected to controller https://96df698b-b55b-4064-a79b-108624c35043.production.netfoundry.io:443 version v0.32.2(7c53aa006529 2024-02-10T05:53:17Z)
Apr 03 19:51:21 fedora ziti-edge-tunnel[1935247]: (1935247)[        0.773]    INFO ziti-sdk:ziti.c:1698 version_cb() ztx[1] connected to controller https://96df698b-b55b-4064-a79b-108624c35043.production.netfoundry.io:443 version v0.32.2(7c53aa006529 2024-02-10T05:53:17Z)
Apr 03 19:51:21 fedora ziti-edge-tunnel[1935247]: (1935247)[        0.826]    INFO ziti-sdk:ziti.c:1698 version_cb() ztx[2] connected to controller https://7ce7e424-6a92-4ff2-9459-ebbba32346fa.production.netfoundry.io:443 version v0.28.1(f9a62c0baf1c 2023-06-09T20:07:49Z)
Apr 03 19:51:22 fedora ziti-edge-tunnel[1935247]: (1935247)[        0.880]    INFO ziti-sdk:ziti.c:1589 ziti_set_api_session() ztx[0] api session set, setting api_session_timer to 1740s
Apr 03 19:51:22 fedora ziti-edge-tunnel[1935247]: (1935247)[        0.880]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:767 on_ziti_event() ziti_ctx[laptop_client1] connected to controller
Apr 03 19:51:22 fedora ziti-edge-tunnel[1935247]: (1935247)[        0.880]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:1165 on_event() ztx[/etc/openziti/identities/laptop_client1.json] context event : status is OK
Apr 03 19:51:22 fedora ziti-edge-tunnel[1935247]: (1935247)[        0.952]   ERROR ziti-sdk:ziti_ctrl.c:262 ctrl_login_cb() ctrl[96df698b-b55b-4064-a79b-108624c35043.production.netfoundry.io] INVALID_AUTH(The authentication request failed)
Apr 03 19:51:22 fedora ziti-edge-tunnel[1935247]: (1935247)[        0.952]    WARN ziti-sdk:ziti.c:1624 api_session_cb() ztx[1] failed to get api session from ctrl[https://96df698b-b55b-4064-a79b-108624c35043.production.netfoundry.io:443] api_session_state[1] INVALID_AUTH[-14] The authentication request failed
Apr 03 19:51:22 fedora ziti-edge-tunnel[1935247]: (1935247)[        0.952]   ERROR ziti-sdk:ziti.c:1638 api_session_cb() ztx[1] identity[/etc/openziti/identities/laptop_client.json] cannot authenticate with ctrl[https://96df698b-b55b-4064-a79b-108624c35043.production.netfoundry.io:443]
Apr 03 19:51:22 fedora ziti-edge-tunnel[1935247]: (1935247)[        0.952]    INFO ziti-edge-tunnel:tun.c:196 tun_commit_routes() starting 1 route updates
Apr 03 19:51:22 fedora ziti-edge-tunnel[1935247]: (1935247)[        0.952]    WARN tunnel-cbs:ziti_tunnel_ctrl.c:781 on_ziti_event() ziti_ctx controller connections failed: failed to authenticate
Apr 03 19:51:22 fedora ziti-edge-tunnel[1935247]: (1935247)[        0.952]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:1165 on_event() ztx[/etc/openziti/identities/laptop_client.json] context event : status is failed to authenticate
Apr 03 19:51:22 fedora ziti-edge-tunnel[1935247]: (1935247)[        0.952]   ERROR ziti-edge-tunnel:ziti-edge-tunnel.c:1220 on_event() ztx[/etc/openziti/identities/laptop_client.json] failed to connect to controller due to failed to authenticate
Apr 03 19:51:22 fedora ziti-edge-tunnel[1935247]: (1935247)[        0.963]    INFO ziti-edge-tunnel:tun.c:118 route_updates_done() route updates[1]: 0/OK
Apr 03 19:51:22 fedora ziti-edge-tunnel[1935247]: (1935247)[        0.985]    INFO ziti-sdk:ziti.c:1589 ziti_set_api_session() ztx[2] api session set, setting api_session_timer to 1740s
Apr 03 19:51:22 fedora ziti-edge-tunnel[1935247]: (1935247)[        0.985]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:767 on_ziti_event() ziti_ctx[laptopClient1 Predalert YXV0aDB8NjYwN2UyNzUwYTAzMjE0MGI0NGZlNGEx] connected to controller
Apr 03 19:51:22 fedora ziti-edge-tunnel[1935247]: (1935247)[        0.985]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:1165 on_event() ztx[/etc/openziti/identities/laptopClient1.json] context event : status is OK
^[[BApr 03 19:51:22 fedora ziti-edge-tunnel[1935247]: (1935247)[        1.570]    INFO ziti-sdk:channel.c:271 new_ziti_channel() ch[0] (AWS us-east-1 Edge Router 1) new channel for ztx[0] identity[laptop_client1]
Apr 03 19:51:22 fedora ziti-edge-tunnel[1935247]: (1935247)[        1.570]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:839 on_ziti_event() ztx[laptop_client1] added edge router AWS us-east-1 Edge Router 1@389c3da8-4278-44d1-905c-b342c1fd8c84.production.netfoundry.io
Apr 03 19:51:22 fedora ziti-edge-tunnel[1935247]: (1935247)[        1.570]    INFO ziti-sdk:channel.c:777 reconnect_channel() ch[0] reconnecting NOW
Apr 03 19:51:22 fedora ziti-edge-tunnel[1935247]: (1935247)[        1.783]    INFO ziti-sdk:channel.c:271 new_ziti_channel() ch[1] (Zeds Fabric Router Azure) new channel for ztx[2] identity[laptopClient1 Predalert YXV0aDB8NjYwN2UyNzUwYTAzMjE0MGI0NGZlNGEx]
Apr 03 19:51:22 fedora ziti-edge-tunnel[1935247]: (1935247)[        1.783]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:839 on_ziti_event() ztx[laptopClient1 Predalert YXV0aDB8NjYwN2UyNzUwYTAzMjE0MGI0NGZlNGEx] added edge router Zeds Fabric Router Azure@dd9dc8aa-b8c6-4423-b4d2-3d80b15f5f49.production.netfoundry.io
Apr 03 19:51:23 fedora ziti-edge-tunnel[1935247]: (1935247)[        1.783]    INFO ziti-sdk:channel.c:777 reconnect_channel() ch[1] reconnecting NOW
Apr 03 19:51:23 fedora ziti-edge-tunnel[1935247]: (1935247)[        1.783]    INFO ziti-sdk:channel.c:271 new_ziti_channel() ch[2] (Zeds Fabric Router) new channel for ztx[2] identity[laptopClient1 Predalert YXV0aDB8NjYwN2UyNzUwYTAzMjE0MGI0NGZlNGEx]
Apr 03 19:51:23 fedora ziti-edge-tunnel[1935247]: (1935247)[        1.783]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:839 on_ziti_event() ztx[laptopClient1 Predalert YXV0aDB8NjYwN2UyNzUwYTAzMjE0MGI0NGZlNGEx] added edge router Zeds Fabric Router@193.122.155.132
Apr 03 19:51:23 fedora ziti-edge-tunnel[1935247]: (1935247)[        1.783]    INFO ziti-sdk:channel.c:777 reconnect_channel() ch[2] reconnecting NOW
Apr 03 19:51:23 fedora ziti-edge-tunnel[1935247]: (1935247)[        1.935]    INFO ziti-sdk:posture.c:206 ziti_send_posture_data() ztx[0] first run or potential controller restart detected
Apr 03 19:51:23 fedora ziti-edge-tunnel[1935247]: (1935247)[        1.938]    INFO tunnel-cbs:ziti_tunnel_cbs.c:409 new_ziti_intercept() creating intercept for service[predalert_ui] with ziti-tunneler-client.v1 = {"hostname":"predalert.ziti","port":80}
Apr 03 19:51:23 fedora ziti-edge-tunnel[1935247]: (1935247)[        1.938]    INFO tunnel-cbs:ziti_dns.c:292 new_ipv4_entry() registered DNS entry predalert.ziti -> 100.96.0.3
Apr 03 19:51:23 fedora ziti-edge-tunnel[1935247]: (1935247)[        1.938]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:727 on_service() starting intercepting for service[predalert_ui]
Apr 03 19:51:23 fedora ziti-edge-tunnel[1935247]: (1935247)[        1.938]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:1290 on_event() =============== service event (added) - predalert_ui:7RCDLQwl6oe2rqTR8mi4Ty ===============
Apr 03 19:51:23 fedora ziti-edge-tunnel[1935247]: (1935247)[        1.938]    INFO ziti-edge-tunnel:tun.c:196 tun_commit_routes() starting 1 route updates
Apr 03 19:51:23 fedora ziti-edge-tunnel[1935247]: (1935247)[        1.969]    INFO ziti-edge-tunnel:tun.c:118 route_updates_done() route updates[1]: 0/OK
Apr 03 19:51:23 fedora ziti-edge-tunnel[1935247]: (1935247)[        1.985]    INFO ziti-sdk:posture.c:206 ziti_send_posture_data() ztx[2] first run or potential controller restart detected
Apr 03 19:51:23 fedora ziti-edge-tunnel[1935247]: (1935247)[        2.246]    INFO tunnel-cbs:ziti_tunnel_cbs.c:409 new_ziti_intercept() creating intercept for service[https-whatismyip] with intercept.v1 = {"addresses":["https-whatismyip.ziti"],"portRanges":[{"high":443,"low":443}],"protocols":["tcp"]}
Apr 03 19:51:23 fedora ziti-edge-tunnel[1935247]: (1935247)[        2.246]    INFO tunnel-cbs:ziti_dns.c:292 new_ipv4_entry() registered DNS entry https-whatismyip.ziti -> 100.96.0.4
Apr 03 19:51:23 fedora ziti-edge-tunnel[1935247]: (1935247)[        2.246]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:727 on_service() starting intercepting for service[https-whatismyip]
Apr 03 19:51:23 fedora ziti-edge-tunnel[1935247]: (1935247)[        2.246]    INFO tunnel-cbs:ziti_tunnel_cbs.c:409 new_ziti_intercept() creating intercept for service[zedsDemoHttpWhatIsMyIp] with intercept.v1 = {"addresses":["whatismyip.ziti"],"portRanges":[{"high":80,"low":80}],"protocols":["tcp"]}
Apr 03 19:51:23 fedora ziti-edge-tunnel[1935247]: (1935247)[        2.246]    INFO tunnel-cbs:ziti_dns.c:292 new_ipv4_entry() registered DNS entry whatismyip.ziti -> 100.96.0.5
Apr 03 19:51:23 fedora ziti-edge-tunnel[1935247]: (1935247)[        2.246]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:727 on_service() starting intercepting for service[zedsDemoHttpWhatIsMyIp]
Apr 03 19:51:23 fedora ziti-edge-tunnel[1935247]: (1935247)[        2.246]    INFO tunnel-cbs:ziti_tunnel_cbs.c:409 new_ziti_intercept() creating intercept for service[zedsDemoHttpsHttpbin] with intercept.v1 = {"addresses":["httpbin.ziti"],"portRanges":[{"high":443,"low":443}],"protocols":["tcp"]}
Apr 03 19:51:23 fedora ziti-edge-tunnel[1935247]: (1935247)[        2.246]    INFO tunnel-cbs:ziti_dns.c:292 new_ipv4_entry() registered DNS entry httpbin.ziti -> 100.96.0.6
Apr 03 19:51:23 fedora ziti-edge-tunnel[1935247]: (1935247)[        2.246]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:727 on_service() starting intercepting for service[zedsDemoHttpsHttpbin]
Apr 03 19:51:23 fedora ziti-edge-tunnel[1935247]: (1935247)[        2.246]    INFO tunnel-cbs:ziti_tunnel_cbs.c:409 new_ziti_intercept() creating intercept for service[zedsDemoHttpFireworks] with intercept.v1 = {"addresses":["fireworks.ziti"],"portRanges":[{"high":80,"low":80}],"protocols":["tcp"]}
Apr 03 19:51:23 fedora ziti-edge-tunnel[1935247]: (1935247)[        2.246]    INFO tunnel-cbs:ziti_dns.c:292 new_ipv4_entry() registered DNS entry fireworks.ziti -> 100.96.0.7
Apr 03 19:51:23 fedora ziti-edge-tunnel[1935247]: (1935247)[        2.246]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:727 on_service() starting intercepting for service[zedsDemoHttpFireworks]
Apr 03 19:51:23 fedora ziti-edge-tunnel[1935247]: (1935247)[        2.246]    INFO tunnel-cbs:ziti_tunnel_cbs.c:409 new_ziti_intercept() creating intercept for service[zedsDemoHttpHttpbin] with intercept.v1 = {"addresses":["httpbin.ziti"],"portRanges":[{"high":80,"low":80}],"protocols":["tcp"]}
Apr 03 19:51:23 fedora ziti-edge-tunnel[1935247]: (1935247)[        2.246]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:727 on_service() starting intercepting for service[zedsDemoHttpHttpbin]
Apr 03 19:51:23 fedora ziti-edge-tunnel[1935247]: (1935247)[        2.246]    INFO tunnel-cbs:ziti_tunnel_cbs.c:409 new_ziti_intercept() creating intercept for service[zedsDemoHttpHelloWorld] with intercept.v1 = {"addresses":["hello.ziti"],"portRanges":[{"high":80,"low":80}],"protocols":["tcp"]}
Apr 03 19:51:23 fedora ziti-edge-tunnel[1935247]: (1935247)[        2.246]    INFO tunnel-cbs:ziti_dns.c:292 new_ipv4_entry() registered DNS entry hello.ziti -> 100.96.0.8
Apr 03 19:51:23 fedora ziti-edge-tunnel[1935247]: (1935247)[        2.246]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:727 on_service() starting intercepting for service[zedsDemoHttpHelloWorld]
Apr 03 19:51:23 fedora ziti-edge-tunnel[1935247]: (1935247)[        2.246]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:742 on_service() hosting server_address[?:hello.ziggy:?] service[predalertUI Predalert YXV0aDB8NjYwN2UyNzUwYTAzMjE0MGI0NGZlNGEx]
Apr 03 19:51:23 fedora ziti-edge-tunnel[1935247]: (1935247)[        2.246]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:1290 on_event() =============== service event (added) - https-whatismyip:r9lqikJfX ===============
Apr 03 19:51:23 fedora ziti-edge-tunnel[1935247]: (1935247)[        2.246]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:1290 on_event() =============== service event (added) - zedsDemoHttpWhatIsMyIp:owB2Rn.fXR ===============
Apr 03 19:51:23 fedora ziti-edge-tunnel[1935247]: (1935247)[        2.246]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:1290 on_event() =============== service event (added) - zedsDemoHttpsHttpbin:jwMlPbkqXR ===============
Apr 03 19:51:23 fedora ziti-edge-tunnel[1935247]: (1935247)[        2.246]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:1290 on_event() =============== service event (added) - zedsDemoHttpFireworks:RkxoZ-VfX ===============
Apr 03 19:51:23 fedora ziti-edge-tunnel[1935247]: (1935247)[        2.246]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:1290 on_event() =============== service event (added) - zedsDemoHttpHttpbin:PvrVQBJfX ===============
Apr 03 19:51:23 fedora ziti-edge-tunnel[1935247]: (1935247)[        2.246]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:1290 on_event() =============== service event (added) - zedsDemoHttpHelloWorld:B.v.O7JfXR ===============
Apr 03 19:51:23 fedora ziti-edge-tunnel[1935247]: (1935247)[        2.246]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:1290 on_event() =============== service event (added) - predalertUI Predalert YXV0aDB8NjYwN2UyNzUwYTAzMjE0MGI0NGZlNGEx:2xas4BHaoFTDHMp9ixibYa ===============
Apr 03 19:51:23 fedora ziti-edge-tunnel[1935247]: (1935247)[        2.246]    INFO ziti-edge-tunnel:tun.c:196 tun_commit_routes() starting 5 route updates
Apr 03 19:51:23 fedora ziti-edge-tunnel[1935247]: (1935247)[        2.258]    INFO ziti-edge-tunnel:tun.c:118 route_updates_done() route updates[5]: 0/OK
Apr 03 19:51:23 fedora ziti-edge-tunnel[1935247]: (1935247)[        2.325]    INFO ziti-sdk:channel.c:669 hello_reply_cb() ch[0] connected. EdgeRouter version: v0.32.2|7c53aa006529|2024-02-10T05:53:17Z|linux|amd64
Apr 03 19:51:23 fedora ziti-edge-tunnel[1935247]: (1935247)[        2.325]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:843 on_ziti_event() ztx[laptop_client1] router AWS us-east-1 Edge Router 1 connected
Apr 03 19:51:23 fedora ziti-edge-tunnel[1935247]: (1935247)[        2.629]    INFO ziti-sdk:channel.c:669 hello_reply_cb() ch[2] connected. EdgeRouter version: v0.28.1|f9a62c0baf1c|2023-06-09T20:07:49Z|linux|amd64
Apr 03 19:51:23 fedora ziti-edge-tunnel[1935247]: (1935247)[        2.629]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:843 on_ziti_event() ztx[laptopClient1 Predalert YXV0aDB8NjYwN2UyNzUwYTAzMjE0MGI0NGZlNGEx] router Zeds Fabric Router connected
Apr 03 19:51:23 fedora ziti-edge-tunnel[1935247]: (1935247)[        2.699]    INFO ziti-sdk:channel.c:669 hello_reply_cb() ch[1] connected. EdgeRouter version: v0.28.1|f9a62c0baf1c|2023-06-09T20:07:49Z|linux|amd64
Apr 03 19:51:23 fedora ziti-edge-tunnel[1935247]: (1935247)[        2.699]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:843 on_ziti_event() ztx[laptopClient1 Predalert YXV0aDB8NjYwN2UyNzUwYTAzMjE0MGI0NGZlNGEx] router Zeds Fabric Router Azure connected

Edit Managed to export the identities and service configs by clicking the "meatball" . . . menu at the top of the relevant tables in NFconsole:

cat v7-services.yml
---
- id: "xxxxxxxxxxxxxxxxxxxxxx"
  networkId: "xxxxxxxxxxxxxxxxxxxxxxxxxxxx"
  zitiId: "xxxxxxxxxxxxxxxxxxxxxxxxxxx"
  name: "predalert_ui"
  encryptionRequired: true
  attributes:
  - "#admin"
  - "#ui"
  - "#strandmoa"
  modelType: "TunnelerToSdk"
  ownerIdentityId: "xxxxxxxxxxxxxxxxxxxxxxxxxxxx"
  createdBy: "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
  createdAt: "2024-04-01T16:10:47.569897Z"
  updatedAt: "2024-04-01T16:10:47.730143Z"
  deletedBy: null
  deletedAt: null
  configIdByConfigTypeId:
    xxxxxxxxxxxxxxxxxxxxxxxxxxxx:xxxxxxxxxxxxxxxxxxxxxx
  model:
    clientIngress:
      host: "predalert.ziti"
      port: 80
    bindEndpointAttributes:
    - "@predalert_server1"
    edgeRouterAttributes: []

 cat endpoints.yml
---
- id: "xxxxxxxxxxxxx"
  ownerIdentityId: "xxxxxxxxxxxxx"
  createdBy: "xxxxxxxxxxxxx"
  createdAt: "2024-04-01T16:25:46.691512Z"
  updatedAt: "2024-04-03T17:52:18.016486Z"
  deletedBy: null
  deletedAt: null
  networkId: "xxxxxxxxxxxxx"
  zitiId: "tHsg.yb4Ra"
  name: "laptop_client1"
  typeId: "Default"
  appId: "ziti-edge-tunnel"
  appVersion: "v0.22.26-local"
  branch: "HEAD"
  revision: "d336721"
  type: "ziti-sdk-c"
  version: "0.36.9"
  arch: "x86_64"
  os: "Linux"
  osRelease: "6.8.1-300.fc40.x86_64"
  osVersion: "#1 SMP PREEMPT_DYNAMIC Wed Mar 20 04:39:30 UTC 2024"
  externalId: null
  authPolicyId: "xxxxxxxxxxxxx"
  disabled: false
  disabledAt: null
  disabledUntil: null
  hasApiSession: true
  hasEdgeRouterConnection: true
  lastOnlineAt: "2024-04-03T17:52:18Z"
  syncId: null
  syncResourceId: null
  attributes:
  - "#show_ui"
  - "#admin"
  - "#strandmoa"
  interceptConflicts: []
  loopbackConflicts: []
  jwtExpiresAt: null
  online: true
  managedBy: "Unmanaged"
  jwt: null
  mfaEnabled: false
- id: "xxxxxxxxxxxxx"
  ownerIdentityId: "xxxxxxxxxxxxx"
  createdBy: "xxxxxxxxxxx"
  createdAt: "2024-04-01T16:11:33.367875Z"
  updatedAt: "2024-04-03T17:44:38.540645Z"
  deletedBy: null
  deletedAt: null
  networkId: "xxxxxxxxxxx"
  zitiId: "lsx2.yc4R"
  name: "predalert_server1"
  typeId: "Default"
  appId: null
  appVersion: null
  branch: "HEAD"
  revision: "a4fcab2"
  type: "ziti-sdk-c"
  version: "0.35.1"
  arch: "x86_64"
  os: "Linux"
  osRelease: "6.8.1-300.fc40.x86_64"
  osVersion: "#1 SMP PREEMPT_DYNAMIC Wed Mar 20 04:39:30 UTC 2024"
  externalId: null
  authPolicyId: "xxxxxxxxxxx"
  disabled: false
  disabledAt: null
  disabledUntil: null
  hasApiSession: true
  hasEdgeRouterConnection: true
  lastOnlineAt: "2024-04-03T17:44:39Z"
  syncId: null
  syncResourceId: null
  attributes:
  - "#admin"
  - "#server"
  - "#strandmoa"
  interceptConflicts: []
  loopbackConflicts: []
  jwtExpiresAt: null
  online: true
  managedBy: "Unmanaged"
  jwt: null
  mfaEnabled: false
- id: "xxxxxxxx"
  ownerIdentityId: "xxxxxxx"
  createdBy: "xxxxxxx"
  createdAt: "2024-03-11T16:24:07.170520Z"
  updatedAt: "2024-03-11T16:24:07.385978Z"
  deletedBy: null
  deletedAt: null
  networkId: "xxxxxxx"
  zitiId: "XgmlP.lnd"
  name: "testCZ_1"
  typeId: "Default"
  appId: null
  appVersion: null
  branch: null
  revision: null
  type: null
  version: null
  arch: null
  os: null
  osRelease: null
  osVersion: null
  externalId: null
  authPolicyId: "xxxxxxx"
  disabled: false
  disabledAt: null
  disabledUntil: null
  hasApiSession: false
  hasEdgeRouterConnection: false
  lastOnlineAt: null
  syncId: null
  syncResourceId: null
  attributes:
  - "#show_ui"
  - "#notifications"
  interceptConflicts: []
  loopbackConflicts: []
  jwtExpiresAt: "2024-03-13T16:24:07.287Z"
  online: false
  managedBy: "Unmanaged"
  jwt: "xxxxx"
  mfaEnabled: false

TheLumberjack · April 3, 2024, 6:37pm

Ahh, you said that and I forgot. Cool -- Can you show me a screen shot of the service you made? Or if you tell me your network, I can just go into "support" mode and peek myself. Feel free to DM me here on discourse if you don't want to share your network name.

TheLumberjack · April 3, 2024, 6:38pm

Andd nevermind - you're using ZEDS! I got there let me go look at it...

TheLumberjack · April 3, 2024, 7:47pm

Ok. I've been doing some digging on my side. Is there any way you could share a sample project so I could try running it myself and experience the issue and look at the way you're using the SDK? I'm at a point where I think I need to see a bare-bones example of what you're doing to help more.

I just started the "echo server example": ziti-sdk-py/sample/ziti-echo-server at main · openziti/ziti-sdk-py · GitHub and was able to send tcp or udp to it and have it respond. So I'm thinking it's something else going on. Can I get a peek at how your code works? Ideally the smallest possible code that's representative of what you're doing to reproduce the issue?

papiris · April 3, 2024, 7:52pm

I was using ZEDS at the start, but the guided tour for how to make the host.v1 config for the first service didn't provide easily digestible documentation; so I spun up a network in NFConsole instead. All my active identities should be in NFConsole at the moment, so maybe the access to ZEDS demo services are just leftovers from when the ziti-edge-tunnel was using an identity that connected to ZEDS?

Sent you a dm

Sure, absolutely. The full project is AGPL and available at the repo mentioned earlier, but it contains more than a few files with a couple hundred LOC. It'll take a little while since I'm going on a weekend journey starting tomorrow, but I'll get back to you with a minimum viable example next week

TheLumberjack · April 3, 2024, 7:54pm

Oi - also had forgotten at this point that you'd shared it... my bad...

papiris · April 3, 2024, 8:03pm

No worries, this thread is growing rather long. Didn't mean to have such a direct tone in my comment, I accidentally published it before it was finished.
Should we make subthreads, or otherwise divide the topic to make it more manageable?

papiris · April 8, 2024, 11:17am

Made a minimal example, which is basically this with tweaks. Had to use a different app function, because the one shown in the issue gives this error:

error in original code

Exception in callback BaseSelectorEventLoop._accept_connection(<function Ser...x7fafb1c9fba0>, <openziti.dec...27.0.0.1', 0)>, None, <Server socke...0.0.1', 0)>,)>, 2048, None, None)
handle: <Handle BaseSelectorEventLoop._accept_connection(<function Ser...x7fafb1c9fba0>, <openziti.dec...27.0.0.1', 0)>, None, <Server socke...0.0.1', 0)>,)>, 2048, None, None)>
Traceback (most recent call last):
  File "/usr/lib64/python3.11/asyncio/events.py", line 84, in _run
    self._context.run(self._callback, *self._args)
  File "/usr/lib64/python3.11/asyncio/selector_events.py", line 165, in _accept_connection
    conn, addr = sock.accept()
                 ^^^^^^^^^^^^^
  File "/var/mnt/data/jacob/git/ziti_py_MWE/.env/lib64/python3.11/site-packages/openziti/zitisock.py", line 119, in accept
    fd, peer = zitilib.accept(self.fileno())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/mnt/data/jacob/git/ziti_py_MWE/.env/lib64/python3.11/site-packages/openziti/zitilib.py", line 276, in accept
    check_error(clt)
  File "/var/mnt/data/jacob/git/ziti_py_MWE/.env/lib64/python3.11/site-packages/openziti/zitilib.py", line 219, in check_error
    raise Exception(err, msg)
Exception: (11, 'unexpected error')
Exception in callback BaseSelectorEventLoop._accept_connection(<function Ser...x7fafb1c9fba0>, <openziti.dec...27.0.0.1', 0)>, None, <Server socke...0.0.1', 0)>,)>, 2048, None, None)
handle: <Handle BaseSelectorEventLoop._accept_connection(<function Ser...x7fafb1c9fba0>, <openziti.dec...27.0.0.1', 0)>, None, <Server socke...0.0.1', 0)>,)>, 2048, None, None)>
Traceback (most recent call last):
  File "/usr/lib64/python3.11/asyncio/events.py", line 84, in _run
    self._context.run(self._callback, *self._args)
  File "/usr/lib64/python3.11/asyncio/selector_events.py", line 165, in _accept_connection
    conn, addr = sock.accept()
                 ^^^^^^^^^^^^^
  File "/var/mnt/data/jacob/git/ziti_py_MWE/.env/lib64/python3.11/site-packages/openziti/zitisock.py", line 119, in accept
    fd, peer = zitilib.accept(self.fileno())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/mnt/data/jacob/git/ziti_py_MWE/.env/lib64/python3.11/site-packages/openziti/zitilib.py", line 276, in accept
    check_error(clt)
  File "/var/mnt/data/jacob/git/ziti_py_MWE/.env/lib64/python3.11/site-packages/openziti/zitilib.py", line 219, in check_error
    raise Exception(err, msg)
Exception: (11, 'unexpected error')
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/var/mnt/data/jacob/git/ziti_py_MWE/.env/lib64/python3.11/site-packages/uvicorn/protocols/http/h11_impl.py", line 407, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/mnt/data/jacob/git/ziti_py_MWE/.env/lib64/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 69, in __call__
    return await self.app(scope, receive, send)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: app() takes 1 positional argument but 3 were given

So here's my example code:

import openziti
import uvicorn
from fastapi import FastAPI


app = FastAPI()

bind_opts = {}

@app.route("/")
async def app(scope, receive, send):
    assert scope['type'] == 'http'

    await send({
        'type': 'http.response.start',
        'status': 200,
        'headers': [
            [b'content-type', b'text/plain'],
        ],
    })
    await send({
        'type': 'http.response.body',
        'body': b'Hello, world!',
    })


@openziti.zitify(bindings={("127.0.0.1", 8443): {'ztx': "./predalert_server1.json", 'service': "predalert_ui"}})
def run_app():
    uvicorn.run(
        "main:app",
        host='127.0.0.1',
        port=8443
        )


if __name__ == "__main__":
    run_app()

Also deleted all enrolled identities in ziti-edge-tunneler, and re-enrolled only one identity with correct access to the service.

Doing so gives this python log:

INFO:     Started server process [62114]
INFO:     Waiting for application startup.
INFO:     ASGI 'lifespan' protocol appears unsupported.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://127.0.0.1:8443 (Press CTRL+C to quit)
Exception in callback BaseSelectorEventLoop._accept_connection(<function Ser...x7fe1473d6b60>, <openziti.dec...27.0.0.1', 0)>, None, <Server socke...0.0.1', 0)>,)>, 2048, None, None)
handle: <Handle BaseSelectorEventLoop._accept_connection(<function Ser...x7fe1473d6b60>, <openziti.dec...27.0.0.1', 0)>, None, <Server socke...0.0.1', 0)>,)>, 2048, None, None)>
Traceback (most recent call last):
  File "/usr/lib64/python3.11/asyncio/events.py", line 84, in _run
    self._context.run(self._callback, *self._args)
  File "/usr/lib64/python3.11/asyncio/selector_events.py", line 165, in _accept_connection
    conn, addr = sock.accept()
                 ^^^^^^^^^^^^^
  File "/var/mnt/data/jacob/git/ziti_py_MWE/.env/lib64/python3.11/site-packages/openziti/zitisock.py", line 119, in accept
    fd, peer = zitilib.accept(self.fileno())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/mnt/data/jacob/git/ziti_py_MWE/.env/lib64/python3.11/site-packages/openziti/zitilib.py", line 276, in accept
    check_error(clt)
  File "/var/mnt/data/jacob/git/ziti_py_MWE/.env/lib64/python3.11/site-packages/openziti/zitilib.py", line 219, in check_error
    raise Exception(err, msg)
Exception: (11, 'unexpected error')
INFO:      - "GET / HTTP/1.1" 200 OK

Can't upload .txt files to the forum, and the journal logs contain too many characters to be allowed as regular comment, so here are logs on a (temporary) pastebin service.
level=debug
level=Info

Topic		Replies	Views
Architecture advice for ensuring isolation for multiple apps deployed within multiple private corporate networks General Questions	7	39	November 8, 2024
Using OpenZiti and zrok to Support "On-Prem" Services	8	106	August 20, 2024
More OpenZiti Questions General Questions	4	296	February 27, 2023
OpenZiti vs Nebula General Questions	4	1447	July 23, 2022
OpenZiti Controller on Hetzner Cloud General Questions	4	430	September 16, 2023

Using OpenZiti in distributed surveillance system

Related topics