ZAC User Manual or Tutorial

[TheLumberjack]

Here's a nice guide from a community member. I'm not able to test it, but it should still work

How-to Start an OpenZiti Simple Instance with Docker Compose

I compiled my resources and repeated my setup, and wrote a doc for a quick and repeatable docker compose setup. The parts that were confusing to me are in the example doc for the .env file. I wrote this for my testing and team and wanted to share it. This serves as the basis for all the future testing we will do, that is until we outgrow it as a testing environment. How-to Start an OpenZiti Simple Instance with Docker Compose The following how-to will create an OpenZiti instance running on Doc…

[caspat]

Yes i had seen this post already, that is exacly the one i'm base on!!
I know now where port 3022 come from!!

now i ave my linux tunneler that refuse to work correctly
is there a script to uninstall completely the linux tunneler

[TheLumberjack]

I know now where port 3022 come from!!

Ok. You can set that port or just use it, if it's using port 3022, that's fine of course, just make sure the firewall allows that port through.

is there a script to uninstall completely the linux tunneler

you shouldn't need to do that. i don't recall seeing that particular issue before. What you should do is remove the old .jwt/.json files from the identities directory you're using. Usually that will be /opt/openziti/etc/identities/. After that you should be able to enroll an identity and the tunneler should work fine. There's no need to uninstall the ziti-edge-tunnel.

[caspat]

Are you an AI responses a so quick!

Ok i delete the identity, reset enrollement, download the new jwt, here is what i get.

root@vi-conteneur-01:~# ziti-edge-tunnel add --jwt "$(< ./vi-conteneur-01.jwt)" --identity vi-conteneur-01
failed to connect: -111/connection refusedroot@vi-conteneur-01:~# 
root@vi-conteneur-01:~# 
root@vi-conteneur-01:~# systemctl start ziti-edge-tunnel
Job for ziti-edge-tunnel.service failed because of unavailable resources or another system error.
See "systemctl status ziti-edge-tunnel.service" and "journalctl -xeu ziti-edge-tunnel.service" for details.
root@vi-conteneur-01:~# systemctl status ziti-edge-tunnel.service
● ziti-edge-tunnel.service - Ziti Edge Tunnel
     Loaded: loaded (/lib/systemd/system/ziti-edge-tunnel.service; enabled; vendor preset: enabled)
     Active: activating (auto-restart) (Result: resources) since Wed 2024-04-17 06:45:42 EDT; 2s ago
        CPU: 0
root@vi-conteneur-01:~# journalctl -xeu ziti-edge-tunnel.service -n 15
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░ 
░░ The unit ziti-edge-tunnel.service has entered the 'failed' state with result 'resources'.
Apr 17 06:47:20 vi-conteneur-01 systemd[1]: Failed to start Ziti Edge Tunnel.
░░ Subject: A start job for unit ziti-edge-tunnel.service has failed
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░ 
░░ A start job for unit ziti-edge-tunnel.service has finished with a failure.
░░ 
░░ The job identifier is 855683 and the job result is failed.
Apr 17 06:47:23 vi-conteneur-01 systemd[1]: ziti-edge-tunnel.service: Scheduled restart job, restart counter is at 10284.
░░ Subject: Automatic restarting of a unit has been scheduled
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░ 
░░ Automatic restarting of the unit ziti-edge-tunnel.service has been scheduled, as the result for
░░ the configured Restart= setting for the unit.
Apr 17 06:47:23 vi-conteneur-01 systemd[1]: Stopped Ziti Edge Tunnel.
░░ Subject: A stop job for unit ziti-edge-tunnel.service has finished
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░ 
░░ A stop job for unit ziti-edge-tunnel.service has finished.
░░ 
░░ The job identifier is 855766 and the job result is done.
Apr 17 06:47:23 vi-conteneur-01 systemd[1]: ziti-edge-tunnel.service: Failed to load environment files: No such file or directory
Apr 17 06:47:23 vi-conteneur-01 systemd[1]: ziti-edge-tunnel.service: Failed to run 'start-pre' task: No such file or directory
Apr 17 06:47:23 vi-conteneur-01 systemd[1]: ziti-edge-tunnel.service: Failed with result 'resources'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░ 
░░ The unit ziti-edge-tunnel.service has entered the 'failed' state with result 'resources'.
Apr 17 06:47:23 vi-conteneur-01 systemd[1]: Failed to start Ziti Edge Tunnel.
░░ Subject: A start job for unit ziti-edge-tunnel.service has failed
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░ 
░░ A start job for unit ziti-edge-tunnel.service has finished with a failure.
░░ 
░░ The job identifier is 855766 and the job result is failed.
root@vi-conteneur-01:~#

[qrkourier]

Hey there @caspat, It looks like your tunneler service was broken somehow. Does your case require customizing the service?

For troubleshooting, let's check the installed version of the package and reinstall it to ensure it's reset. Then, if any customizations are needed, we can layer those on to work with future versions.

1. check version

   ziti-edge-tunnel version;
   apt show ziti-edge-tunnel
   

2. reset the service, ensuring latest is installed

   sudo apt update;
   sudo apt install ziti-edge-tunnel --reinstall
   

3. check the version again to see if it changed

   ziti-edge-tunnel version;
   apt show ziti-edge-tunnel
   

4. run the service every boot

   sudo systemctl enable --now ziti-edge-tunnel.service
   

Do you get the same result? Did the tunneler version change?

[caspat]

Hi qrkourier, no, nothing special, basic linux tuneller.
Here is the result of those commands.

root@vi-conteneur-01:~# ziti-edge-tunnel version;
v0.22.26-local
root@vi-conteneur-01:~# apt show ziti-edge-tunnel
Package: ziti-edge-tunnel
Version: 0.22.26
Priority: optional
Section: devel
Maintainer: support@netfoundry.io
Installed-Size: 4,668 kB
Depends: debconf, iproute2, sed, systemd, libatomic1, libssl3 | libssl1.1 | libssl1.0.0, login, passwd, policykit-1, zlib1g
Download-Size: 2,029 kB
APT-Manual-Installed: yes
APT-Sources: https://packages.openziti.org/zitipax-openziti-deb-stable jammy/main amd64 Packages
Description: ziti-tunnel-sdk-c built using CMake

N: There are 57 additional records. Please use the '-a' switch to see them.
root@vi-conteneur-01:~# sudo apt update
Hit:1 Index of linux/ubuntu/ jammy InRelease
Hit:2 Index of /ubuntu jammy-security InRelease
Hit:3 Index of /ubuntu jammy InRelease
Hit:4 Index of /ubuntu jammy-updates InRelease
Hit:5 Index of /ubuntu jammy-backports InRelease
Ign:6 https://apt.fury.io/netdevops InRelease
Ign:7 https://apt.fury.io/netdevops Release
Ign:8 https://apt.fury.io/netdevops Packages
Ign:9 https://apt.fury.io/netdevops Translation-en_US
Ign:10 https://apt.fury.io/netdevops Translation-en
Get:8 https://apt.fury.io/netdevops Packages
Ign:9 https://apt.fury.io/netdevops Translation-en_US
Ign:10 https://apt.fury.io/netdevops Translation-en
Ign:9 https://apt.fury.io/netdevops Translation-en_US
Ign:10 https://apt.fury.io/netdevops Translation-en
Ign:9 https://apt.fury.io/netdevops Translation-en_US
Ign:10 https://apt.fury.io/netdevops Translation-en
Ign:9 https://apt.fury.io/netdevops Translation-en_US
Ign:10 https://apt.fury.io/netdevops Translation-en
Ign:9 https://apt.fury.io/netdevops Translation-en_US
Ign:10 https://apt.fury.io/netdevops Translation-en
Ign:9 https://apt.fury.io/netdevops Translation-en_US
Ign:10 https://apt.fury.io/netdevops Translation-en
Hit:11 https://packages.openziti.org/zitipax-openziti-deb-stable jammy InRelease
Fetched 23.7 kB in 1s (42.7 kB/s)
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
7 packages can be upgraded. Run 'apt list --upgradable' to see them.
root@vi-conteneur-01:~# sudo apt install ziti-edge-tunnel --reinstall
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
0 upgraded, 0 newly installed, 1 reinstalled, 0 to remove and 7 not upgraded.
Need to get 0 B/2,029 kB of archives.
After this operation, 0 B of additional disk space will be used.
Preconfiguring packages ...
(Reading database ... 118039 files and directories currently installed.)
Preparing to unpack .../ziti-edge-tunnel_0.22.26_amd64.deb ...
Unpacking ziti-edge-tunnel (0.22.26) over (0.22.26) ...
Setting up ziti-edge-tunnel (0.22.26) ...

--------------------------------------------------------------------------------ziti-edge-tunnel was installed...
First install an OpenZiti identity or enroll token in: /opt/openziti/etc/identities
then start or restart this systemd service unit.

Scanning processes...
Scanning linux images...

Running kernel seems to be up-to-date.

No services need to be restarted.

No containers need to be restarted.

No user sessions are running outdated binaries.

No VM guests are running outdated hypervisor (qemu) binaries on this host.
root@vi-conteneur-01:~# ziti-edge-tunnel version
v0.22.26-local
root@vi-conteneur-01:~# apt show ziti-edge-tunnel
Package: ziti-edge-tunnel
Version: 0.22.26
Priority: optional
Section: devel
Maintainer: support@netfoundry.io
Installed-Size: 4,668 kB
Depends: debconf, iproute2, sed, systemd, libatomic1, libssl3 | libssl1.1 | libssl1.0.0, login, passwd, policykit-1, zlib1g
Download-Size: 2,029 kB
APT-Manual-Installed: yes
APT-Sources: https://packages.openziti.org/zitipax-openziti-deb-stable jammy/main amd64 Packages
Description: ziti-tunnel-sdk-c built using CMake

N: There are 57 additional records. Please use the '-a' switch to see them.
root@vi-conteneur-01:~# sudo systemctl enable --now ziti-edge-tunnel.service
Job for ziti-edge-tunnel.service failed because of unavailable resources or another system error.
See "systemctl status ziti-edge-tunnel.service" and "journalctl -xeu ziti-edge-tunnel.service" for details.
root@vi-conteneur-01:~# systemctl status ziti-edge-tunnel.service
● ziti-edge-tunnel.service - Ziti Edge Tunnel
Loaded: loaded (/lib/systemd/system/ziti-edge-tunnel.service; enabled; vendor preset: enabled)
Active: activating (auto-restart) (Result: resources) since Wed 2024-04-17 17:03:51 EDT; 1s ago
CPU: 0
root@vi-conteneur-01:~# journalctl -xeu ziti-edge-tunnel.service -n15
░░ A stop job for unit ziti-edge-tunnel.service has finished.
░░
░░ The job identifier is 1789641 and the job result is done.
Apr 17 17:04:08 vi-conteneur-01 systemd[1]: ziti-edge-tunnel.service: Failed to load environment files: No such file or directory
Apr 17 17:04:08 vi-conteneur-01 systemd[1]: ziti-edge-tunnel.service: Failed to run 'start-pre' task: No such file or directory
Apr 17 17:04:08 vi-conteneur-01 systemd[1]: ziti-edge-tunnel.service: Failed with result 'resources'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ The unit ziti-edge-tunnel.service has entered the 'failed' state with result 'resources'.
Apr 17 17:04:08 vi-conteneur-01 systemd[1]: Failed to start Ziti Edge Tunnel.
░░ Subject: A start job for unit ziti-edge-tunnel.service has failed
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ A start job for unit ziti-edge-tunnel.service has finished with a failure.
░░
░░ The job identifier is 1789641 and the job result is failed.
Apr 17 17:04:11 vi-conteneur-01 systemd[1]: ziti-edge-tunnel.service: Scheduled restart job, restart counter is at 10.
░░ Subject: Automatic restarting of a unit has been scheduled
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ Automatic restarting of the unit ziti-edge-tunnel.service has been scheduled, as the result for
░░ the configured Restart= setting for the unit.
Apr 17 17:04:11 vi-conteneur-01 systemd[1]: Stopped Ziti Edge Tunnel.
░░ Subject: A stop job for unit ziti-edge-tunnel.service has finished
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ A stop job for unit ziti-edge-tunnel.service has finished.
░░
░░ The job identifier is 1789724 and the job result is done.
Apr 17 17:04:11 vi-conteneur-01 systemd[1]: ziti-edge-tunnel.service: Failed to load environment files: No such file or directory
Apr 17 17:04:11 vi-conteneur-01 systemd[1]: ziti-edge-tunnel.service: Failed to run 'start-pre' task: No such file or directory
Apr 17 17:04:11 vi-conteneur-01 systemd[1]: ziti-edge-tunnel.service: Failed with result 'resources'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ The unit ziti-edge-tunnel.service has entered the 'failed' state with result 'resources'.
Apr 17 17:04:11 vi-conteneur-01 systemd[1]: Failed to start Ziti Edge Tunnel.
░░ Subject: A start job for unit ziti-edge-tunnel.service has failed
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ A start job for unit ziti-edge-tunnel.service has finished with a failure.
░░
░░ The job identifier is 1789724 and the job result is failed.

[qrkourier]

@caspat

root@vi-conteneur-01:~# systemctl status ziti-edge-tunnel.service
● ziti-edge-tunnel.service - Ziti Edge Tunnel
Loaded: loaded (/lib/systemd/system/ziti-edge-tunnel.service; enabled; vendor preset: enabled)
Active: activating (auto-restart) (Result: resources) since Wed 2024-04-17 17:03:51 EDT; 1s ago
CPU: 0

This might mean the device doesn't have enough CPUshares for the service to start. The service unit template doesn't specify a minimum, but there may still be a systemd default minimum.

What's the systemd version and what's the load average and memory free?

[caspat]

Voilà!

@qrkourier

What's the systemd version and what's the load average and memory free?

root@vi-conteneur-01:~# uptime
17:21:30 up 19:51, 3 users, load average: 0.04, 0.01, 0.00
root@vi-conteneur-01:~# free
total used free shared buff/cache available
Mem: 32862676 1336512 29721372 32636 1804792 31018936
Swap: 8388604 0 8388604
root@vi-conteneur-01:~# lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 43 bits physical, 48 bits virtual
Byte Order: Little Endian
CPU(s): 8
On-line CPU(s) list: 0-7
Vendor ID: GenuineIntel
Model name: Intel(R) Xeon(R) Gold 6138 CPU @ 2.00GHz
CPU family: 6
Model: 85
Thread(s) per core: 1
Core(s) per socket: 1
Socket(s): 8
Stepping: 4
BogoMIPS: 3990.62
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon nopl xtopolo
gy tsc_reliable nonstop_tsc cpuid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hyperv
isor lahf_lm abm 3dnowprefetch cpuid_fault invpcid_single pti ssbd ibrs ibpb stibp fsgsbase tsc_adjust bmi1 avx2 smep bmi2 invpcid avx512f avx512dq rdseed adx smap clfl
ushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xsaves arat pku ospke md_clear flush_l1d arch_capabilities
Virtualization features:
Hypervisor vendor: VMware
Virtualization type: full
Caches (sum of all):
L1d: 256 KiB (8 instances)
L1i: 256 KiB (8 instances)
L2: 8 MiB (8 instances)
L3: 220 MiB (8 instances)
NUMA:
NUMA node(s): 1
NUMA node0 CPU(s): 0-7
Vulnerabilities:
Gather data sampling: Unknown: Dependent on hypervisor status
Itlb multihit: KVM: Mitigation: VMX unsupported
L1tf: Mitigation; PTE Inversion
Mds: Mitigation; Clear CPU buffers; SMT Host state unknown
Meltdown: Mitigation; PTI
Mmio stale data: Mitigation; Clear CPU buffers; SMT Host state unknown
Retbleed: Mitigation; IBRS
Spec rstack overflow: Not affected
Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Spectre v2: Mitigation; IBRS, IBPB conditional, STIBP disabled, RSB filling, PBRSB-eIBRS Not affected
Srbds: Not affected
Tsx async abort: Not affected
root@vi-conteneur-01:~# systemctl --version
systemd 249 (249.11-0ubuntu3.12)
+PAM +AUDIT +SELINUX +APPARMOR +IMA +SMACK +SECCOMP +GCRYPT +GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 -PWQUALITY -P11KIT -QRENCODE +BZIP2 +LZ4 +XZ +ZLIB +ZSTD -XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified
root@vi-conteneur-01:~#

[caspat]

This VM run 3 test docker container, not in prod.

[scareything]

@caspat

Apr 17 17:04:08 vi-conteneur-01 systemd[1]: ziti-edge-tunnel.service: Failed to load environment files: No such file or directory
Apr 17 17:04:08 vi-conteneur-01 systemd[1]: ziti-edge-tunnel.service: Failed to run 'start-pre' task: No such file or directory
Apr 17 17:04:08 vi-conteneur-01 systemd[1]: ziti-edge-tunnel.service: Failed with result 'resources'.

This suggests to me that something isn't quite right with the systemd service file(s). Could you please systemctl cat ziti-edge-tunnel.service and share the output here?

You should see EnvironmentFile and ExecStartPre directives in the output. Can you verify that those files exist?

Thanks!

[caspat]

I think you have found the problem...

The EnvironmentFile is missing, ExecStartPre is present.
Is the EnvironmentFile generic or i need to create or generate one?

root@vi-conteneur-01:~$ systemctl cat ziti-edge-tunnel.service

/lib/systemd/system/ziti-edge-tunnel.service

[Unit]
Description=Ziti Edge Tunnel
After=network-online.target

[Service]
Type=simple
EnvironmentFile=/opt/openziti/etc/ziti-edge-tunnel.env
User=ziti
UMask=0007
AmbientCapabilities=CAP_NET_ADMIN
ExecStartPre=/opt/openziti/bin/ziti-edge-tunnel.sh
ExecStart=/opt/openziti/bin/ziti-edge-tunnel run --verbose=${ZITI_VERBOSE} --dns-ip-range=${ZITI_DNS_IP_RANGE} --identity-dir=${ZITI_IDENTITY_DIR}
Restart=always
RestartSec=3

[Install]
WantedBy=multi-user.target

[scareything]

The template for the environment file is here. You can create one yourself based on that template, but it should have been generated when the package was installed. If there were any errors logged during the installation then I'm guessing they are long gone by now. Maybe @qrkourier has some ideas on what might have gone wrong there.

[qrkourier]

It's the first time I've seen that env file fail to appear in the expected location. It's always placed there by the Debian package during install or upgrade. Is /opt writable on this Ubuntu Jammy host?

Here's an example showing the typical identities dir populated in that tempalte.

# all enrollment tokens named *.jwt are consumed and replaced with identity JSON files to be loaded at startup
ZITI_IDENTITY_DIR='/opt/openziti/etc/identities'

# reserved dynamic IP range for proxied services
ZITI_DNS_IP_RANGE='100.64.0.1/10'

# the log level specified in /var/lib/ziti/config.json has higher precedence than this env var; delete or modify that
# file or set via IPC "ziti-edge-tunnel set_log_level --loglevel DEBUG"
ZITI_VERBOSE=2

EDIT: probably tangential, but this reminds me that some immutable OSs require writable paths appear in specific places within the FHS, sometimes excluding /opt, which may force us to stop using /opt in order to avoid switching logic in the scriptlets, not to mention documentation.

[caspat]

What are the permissions that is suppose to have on /opt/openziti ?

Have created the env file and the service start, but having warning:

WARN ziti-edge-tunnel:ziti-edge-tunnel.c:1686 make_socket_path() failed to set ownership of /tmp/.ziti to 998:998: Operation not permitted (errno=1)
Apr 18 12:16:04 vi-conteneur-01 ziti-edge-tunnel[39322]: (39322)[ 0.000] WARN ziti-edge-tunnel:ziti-edge-tunnel.c:1730 run_tunneler_loop() One or more socket servers did not properly start.

[caspat]

Ok i deleted le .ziti folder in tmp directory, the service recreate it and the error is gone.

[caspat]

still get :

ERROR ziti-sdk:channel.c:899 on_channel_connect_internal() ch[0] failed to connect to ER[ziti-edge-router] [-3001/temporary failure]

[TheLumberjack]

Can you confirm what the advertised address is for your router and can you verify you can openssl -connect to that address?

[caspat]

get openssl response but verification error:

subject=C = US, ST = NC, L = Charlotte, O = NetFoundry, OU = Ziti, CN = nYfzDmgOn
issuer=C = US, L = Charlotte, O = NetFoundry, OU = ADV-DEV, CN = ziti-signing-intermediate
---
No client certificate CA names sent
Requested Signature Algorithms: RSA-PSS+SHA256:ECDSA+SHA256:Ed25519:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512:ECDSA+SHA384:ECDSA+SHA512:RSA+SHA1:ECDSA+SHA1
Shared Requested Signature Algorithms: RSA-PSS+SHA256:ECDSA+SHA256:Ed25519:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512:ECDSA+SHA384:ECDSA+SHA512
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 2360 bytes and written 416 bytes
Verification error: unable to verify the first certificate
---
New, TLSv1.3, Cipher is TLS_AES_128_GCM_SHA256
Server public key is 4096 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 21 (unable to verify the first certificate)
---
DONE
root@vi-conteneur-01:~#

[TheLumberjack]

I also asked for you to confirm your router's adveritsed address. Can you post the router's relevant config section?

[caspat]

Sorry here is the info:

#ZITI_ROUTER_NAME=ziti-edge-router
ZITI_ROUTER_ADVERTISED_ADDRESS=router.x.x.ca
ZITI_ROUTER_PORT=8442
#ZITI_ROUTER_IP_OVERRIDE=10.10.10.10
ZITI_ROUTER_LISTENER_BIND_PORT=8444
ZITI_ROUTER_ROLES=public