Tunneler installation on raspberry pi

Tunneler Apps
1

I’m trying out OpenZiti as a replacement for ZeroTier. I was able to successfully install the tunneler on my Ubuntu desktop. On the pi, I got the tunneler installed and enrolled. It won’t maintain a connection to the edge router however. I get the following messages:

pi@raspberrypi:~ $ sudo ./ziti-edge-tunnel run --identity-dir /opt/openziti/etc/identities --verbose 4
(18519)[        0.000]    INFO ziti-sdk:utils.c:173 ziti_log_set_level() set log level: root=4/DEBUG
(18519)[        0.000]    INFO ziti-edge-tunnel:instance-config.c:86 load_tunnel_status_from_file() Loading config file from /var/lib/ziti/config.json
(18519)[        0.000]    INFO ziti-sdk:utils.c:173 ziti_log_set_level() set log level: root=3/INFO
(18519)[        0.000]    INFO tunnel-sdk:ziti_tunnel.c:60 create_tunneler_ctx() Ziti Tunneler SDK (v0.20.22)
(18519)[        0.000]    INFO tunnel-cbs:ziti_dns.c:171 seed_dns() DNS configured with range 100.64.0.0 - 100.127.255.255 (4194302 ips)
(18519)[        0.000]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:1599 run_tunneler_loop() Loading identity files from /opt/openziti/etc/identities
(18519)[        0.000]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:1093 load_identities() loading identity file: raspi1.json
(18519)[        0.060]    INFO ziti-edge-tunnel:resolvers.c:67 init_libsystemd() Initializing libsystemd
(18519)[        0.060]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:864 load_ziti_async() attempting to load ziti instance from file[/opt/openziti/etc/identities/raspi1.json]
(18519)[        0.060]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:871 load_ziti_async() loading ziti instance from /opt/openziti/etc/identities/raspi1.json
(18519)[        0.060]    INFO ziti-sdk:utils.c:173 ziti_log_set_level() set log level: root=3/INFO
(18519)[        0.060]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:1108 load_id_cb() identity[/opt/openziti/etc/identities/raspi1.json] loaded
(18519)[        0.071]    INFO ziti-sdk:ziti.c:426 ziti_init_async() ztx[0] Ziti C SDK version 0.31.2 @c74ab09(HEAD) starting at (2023-03-16T05:21:13.024)
(18519)[        0.071]    INFO ziti-sdk:ziti.c:428 ziti_init_async() ztx[0] using tlsuv[<unknown>], tls[mbed TLS 3.2.1]
(18519)[        0.071]    WARN ziti-edge-tunnel:resolvers.c:351 try_libsystemd_resolver() libsystemd resolver unsuccessful. Falling back to legacy resolvers
(18519)[        0.071]    INFO ziti-sdk:ziti.c:429 ziti_init_async() ztx[0] Loading ziti context with controller[https://1cbcdfd4-c7ac-4be9-8f2e-1075845c774c.production.netfoundry.io:443]
(18519)[        0.071]    INFO ziti-sdk:ziti_ctrl.c:409 ziti_ctrl_init() ctrl[1cbcdfd4-c7ac-4be9-8f2e-1075845c774c.production.netfoundry.io] ziti controller client initialized
(18519)[        0.071]    INFO ziti-sdk:ziti.c:866 ziti_re_auth_with_cb() ztx[0] starting to re-auth with ctlr[https://1cbcdfd4-c7ac-4be9-8f2e-1075845c774c.production.netfoundry.io:443] api_session_status[0] api_session_expired[TRUE]
(18519)[        2.166]    INFO ziti-sdk:ziti.c:1532 version_cb() ztx[0] connected to controller https://1cbcdfd4-c7ac-4be9-8f2e-1075845c774c.production.netfoundry.io:443 version v0.27.5(3d9801e73809 2023-02-13T21:49:17Z)
(18519)[        2.834]    INFO ziti-sdk:ziti.c:1422 ziti_set_api_session() ztx[0] api session set, setting api_session_timer to 1740s
(18519)[        2.834]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:726 on_ziti_event() ziti_ctx[raspi1] connected to controller
(18519)[        2.834]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:1138 on_event() ztx[/opt/openziti/etc/identities/raspi1.json] context event : status is OK
(18519)[        7.920]    INFO ziti-sdk:posture.c:204 ziti_send_posture_data() ztx[0] first run or potential controller restart detected
(18519)[       10.766]    INFO ziti-sdk:channel.c:234 new_ziti_channel() ch[0] (OCI us-ashburn-1 Edge Router 1@tls://91c54d8a-0e63-43aa-9273-140312049c5f.production.netfoundry.io:443) new channel for ztx[0] identity[raspi1]
(18519)[       10.766]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:797 on_ziti_event() ztx[raspi1] added edge router OCI us-ashburn-1 Edge Router 1@tls://91c54d8a-0e63-43aa-9273-140312049c5f.production.netfoundry.io:443@91c54d8a-0e63-43aa-9273-140312049c5f.production.netfoundry.io
(18519)[       10.766]    INFO ziti-sdk:channel.c:733 reconnect_channel() ch[0] reconnecting NOW
(18519)[       18.789]    INFO ziti-sdk:channel.c:822 on_channel_data() ch[0] channel was closed [-4095/end of file]
(18519)[       18.789]   ERROR ziti-sdk:channel.c:623 hello_reply_cb() ch[0] failed to receive Hello response due to -20(Connection to edge router terminated)
(18519)[       18.789]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:810 on_ziti_event() ztx[raspi1] router OCI us-ashburn-1 Edge Router 1@tls://91c54d8a-0e63-43aa-9273-140312049c5f.production.netfoundry.io:443 is unavailable
(18519)[       18.789]    INFO ziti-sdk:channel.c:730 reconnect_channel() ch[0] reconnecting in 58478344 ms (attempt = 2359)
(18519)[       18.789]    INFO ziti-sdk:channel.c:730 reconnect_channel() ch[0] reconnecting in 5841396 ms (attempt = 734)
(18519)[       19.052]    INFO tunnel-cbs:ziti_tunnel_cbs.c:418 new_ziti_intercept() creating intercept for service[Sample_Service] with ziti-tunneler-client.v1 = {"hostname":"sample.tools.netfoundry.io","port":443}
(18519)[       19.052]    INFO tunnel-cbs:ziti_dns.c:296 new_ipv4_entry() registered DNS entry sample.tools.netfoundry.io -> 100.64.0.3
(18519)[       19.052]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:686 on_service() starting intercepting for service[Sample_Service]
(18519)[       19.052]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:1263 on_event() =============== service event (added) - Sample_Service:4jX8C7IMKzoUzdMKFuMGYx ===============
(18519)[       19.052]    INFO ziti-edge-tunnel:tun.c:174 tun_commit_routes() starting 2 route updates
(18519)[       19.062]    INFO ziti-edge-tunnel:tun.c:118 route_updates_done() route updates[2]: 0/OK
(18519)[       19.794]    INFO ziti-sdk:ziti.c:1422 ziti_set_api_session() ztx[0] api session set, setting api_session_timer to 1740

I’ve no idea how to troubleshoot this, so any help would be appreciated.

2

Hi @aepurvis, welcome to OpenZiti and the community! The one error in your log snippet that seems relevant is this:

(18519)[       18.789]   ERROR ziti-sdk:channel.c:623 hello_reply_cb() ch[0] failed to receive Hello response due to -20(Connection to edge router terminated)

That error is strange to me, it makes me think the router terminated your connection forcefully. I’d only expect that to be a problem if there was some kind of problem with your identity. By chance, did you enroll the pi and then recreate the identity? Can you share any additional details about what your process was?

3

I believe I ran the tunneler the first time before creating the identity. I don’t think I recreated the identity (it took me a couple of tries on the desktop, and I’ve been poking at the pi as time permits over the last couple of days).

I followed the instructions for a manual linux installation. Had to tweak that because uname returned a value that wasn’t useful. Downloaded the identity on the desktop and transferred to the pi.

4

Could you post what it returned so we can make the doc a bit better? :slight_smile: that'd be appreciated. I don't have a pi handy. also what sort of pi was it?

5 
pi@raspberrypi:~ $ uname -p
unknown
pi@raspberrypi:~ $ uname -a
Linux raspberrypi 5.15.30-v7+ #1536 SMP Mon Mar 28 13:43:34 BST 2022 armv7l GNU/Linux

It’s a 3B+

6

I created another endpoint and downloaded the registration key to my desktop and copied to the pi. Enrolled there with command below. Same result, with the edge router disconnecting.

sudo ~/ziti-edge-tunnel enroll --jwt ./testpi.jwt --identity ./testpi.json
7

Thanks for doing that. I have a raspberry pi B+, but I can’t get an OS onto it yet. I’ll see if anyone who works on the project has one we can do some testing on to see if it’s somehow related to the pi or to the network or something else.

8

Hi @aepurvis, I tested a scenario running OpenZiti on my Raspberry Pi 4 running the 32-bit Raspbian OS and it was successful.

Linux raspberrypi 5.15.84-v7l+

I’m running Ziti 0.27.5, with the controller and router running on the Pi, hosting a simple HTTP server on the Pi and accessing from an external device. The tunneler version I’m running is 0.19.9.

Can you think of anything we might be missing that might help us better debug the problem you’re seeing?

9

@gberl002 I’m running only the tunneler (v0.20.22) on the pi as a netfoundry endpoint. It occurs to that I didn’t mention that in the initial post. I was equating the two.

10

I’m wondering if maybe the pi OS is too old and needs an update? I noticed that Geoff’s is 5.15.84-v7l+ but @aepurvis is 5.15.30-v7+. I doubt that it’s the problem, but it’s possible.

We also tested Geoff’s PI connecting to my CloudZiti instance similar to what @aepurvis is doing and it worked fine as well. I’ll try to get the PI os installed on mine and on my local network so that I can test it out.

11

I noticed the OS version too. I’ll have to check what the ripple effects of updating it would be.

12

Apparently, my pi is “just too old”. It’s armv6l…

Linux raspberrypi 5.15.84+ #1613 Thu Jan 5 11:58:09 GMT 2023 armv6l GNU/Linux

@aepurvis, how many identities are in the /opt/openziti/etc/identities directory? just one?

@scareything, can you think of anything we could do to figure out why Geoff’s pi works, but aepurvis’ doesn’t?

13

@TheLumberjack, there was initially one identity. I’ve since added a second newly created one, but I’m starting the tunneler with the specific identity instead of the whole folder.

I have a second pi with an older version of the os. It’s close to stock so I’m going to try upgrading it to current to see if that helps. Actually, I’ll try the tunneler first as is.

14

Last thing I can think of is to just make sure the cert/key/ca in your identity are correct and working using openssl… If you’re up for it can you:

  • install jq (sudo apt install jq -y)

  • use jq to rip the cert, key, ca from the identity follwing these steps:

    • cd to /opt/openziti/etc/identities/identity
    • jq -r .id.key id.json > id.key
    • jq -r .id.cert id.json > id.cert
    • jq -r .id.ca id.json > id.ca

  • use openssl to connect to your router (notice I got your router from the log you supplied):

    openssl s_client \
      -connect 91c54d8a-0e63-43aa-9273-140312049c5f.production.netfoundry.io:443 \
      -CAfile ./id.ca \
      -cert ./id.cert \
      -key ./id.key

If you do that you should see the last few lines look like this - the key being the “Verify return code: 0 (ok)”:

    Start Time: 1679055861
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: no
    Max Early Data: 0

Does that process succeed for you?

15

@TheLumberjack, the openssl connection to the router worked

16

That’s both ‘great’ and a ‘bummer’ simultaneously… You don’t have any kind of intrusion detection on the network I expect, nothing ‘sniping’ your connection, no antivirus running that would get in the way… Nothing complicated like that I presume either, right?

I guess the only other thing would be to ask you to run with extra logging and provide the full log to see if there’s anything down in the VERBOSE type of level that might be relevant…

Can you run with -v 6:

sudo ./ziti-edge-tunnel run -i ./id.json -v 6

You can send the logs to clint at openziti.org if you don’t want to publish them here (or if they are just too long, etc)…

17

Nope, nothing fancy running on the nework

18

Looks like our edge routers have a “connectTimeout” feature and it might be set to 1000ms. If that’s the case, it might just be the pi is just taking a tiny bit too long to connect. I’m going to ask for someone to change that timeout on your particular router, restart the router, and have you try again.

If that’s the case, I’ll have you re-test and hopefully that pi will connect!

19

Ok @aepurvis we changed the value to be 10000. Would you try connecting that pi again and let’s see if it stays connected?

Cheers

20

@TheLumberjack , that’s probably it. I didn’t mention that I’m on a satellite connection - the desktop worked. As you say, the pi is likely just a bit slower. I’ll test when I get home. Thanks