Create private router on linux machine

Building/Development
1

I’d like to create a private router so that traffic between endpoints remains local when they’re on the local network. I’ve created the router in the NetFoundry console and not really sure where to go from there. Any instructions I’ve found assume that your creating a full network, or using a VM of some sort. I tried the manual installation instructions, but they glossed over converting the yml template file into a YAML file that can be used to enroll using the ziti-router binary. Played around a bit with the express install as well but ended up with a router that was enrolled to the controller that I don’t need (I think). It wouldn’t start, but that was probably because the controller wasn’t configured or running. Any pointers to get me headed the right direction would be appreciated.

2

Hi @aepurvis,

There’s two basic ways you can go about this. The first is to download the virtual machine from NetFoundry, create the “customer hosted” router and then and run the provisioning script. I believe this is the page that discusses this approach https://support.netfoundry.io/hc/en-us/articles/360016129312-Run-the-Edge-Router-VM-on-Your-Own-Equipment

The other way is to make the router in the console and choose ”advanced downloads" and “download jwt” and “download config”. Once you download the jwt/config, you would download the ziti binary from GitHub and then “enroll” the router using the config file and jwt. You probably want to open that config up and change a few values. I think this is the doc that covers that https://support.netfoundry.io/hc/en-us/articles/360044772432-How-to-Manually-Configure-an-Edge-Router

If you found those two pages and still were not successful, lemme know. Your next step will be to update and craft the edge router policy which hopefully will make sense.

Would it be helpful to have a short video covering this? I will look to see if we have a video covering this process if you’d find that useful or I’ll try to make one tomorrow if I don’t find one (and it’d be helpful for you)

3

The first page is for VMs and I’m using an Antlet on an Antsle server - essentially an LXC instance of Debian.

The second page is what I tried following first. The yml config file is actually a Jinja template, near as I can figure. Step 2 is “Render the Jinja2 template as a YAML file according to your preferences.” I haven’t been able to find anything about the rendering process, or how to select suitable preferences.

If a video is the easiest way to describe the process, that works for me. Otherwise, text is fine.

Thanks for you continued help.

4

Ok here's the text version... I'm going to cover a bit more than just the "how to get the router running" since you need that as well. I'll cover the policy work I think you want too... I can always make a video if you find it helpful. :slight_smile:

Installing the Private Edge Router

I have a raspberry pi-like device called an inovato (check them out if you haven't :slight_smile: ) which I'm going to install the router on. It's just like a pi without the pin-outs. Here's how I installed the router:

  1. used the helper functions to get ziti on my path (do it manually if you prefer)
    source /dev/stdin <<< "$(wget -qO- https://get.openziti.io/quick/ziti-cli-functions.sh)"; getZiti yes
  2. in nfconsole.io, navigate to "Edge Routers"
  3. click the plus in the upper right
  4. give the router a name (I used home-router), give it an attribute of "home-network" (or some attribute you want), choose "Customer Hosted", and click the button to create it.
    image
    image768×729 41.1 KB
  5. You'll be taken to a new screen saying the router has been created.
  6. expand "Advanced Downloads" and download the jwt. We'll make the config below
  7. Transfer the jwt to your private router.
  8. You seem like you'll be familar with bash, so I won't explain these parts too much...
  9. set some environment variables.
    • ROUTER_NAME is whatever you want the files to be called on the machine
    • ZITI_HOME is important if you follow these instructions later on and it represents where you want the files to be put
    • CONTROLLER_ADDRESS should be correct. I pulled it from your other discourse post logs for you
    • ROUTER_LOCAL_ADDRESS is the name that the other devices can address your private edge router at. Notice I'm using "inovato", you MUST change that unless you have an 'inovato' too :slight_smile:
    export ROUTER_NAME="private-router"
export ZITI_HOME="$HOME/ziti-files/${ROUTER_NAME}"
export CONTROLLER_ADDRESS="1cbcdfd4-c7ac-4be9-8f2e-1075845c774c.production.netfoundry.io"
export ROUTER_LOCAL_ADDRESS="inovato"
  10. make the new folder:
    mkdir -p $ZITI_HOME
  11. Emit the config file using a HEREDOC into the folder you just made by literally just copying/pasting this block or do it yourself manually, it should be pretty straightforward to see where the variables are.
    cat > ${ZITI_HOME}/${ROUTER_NAME}.yaml <<HERE
v: 3
identity:
  cert:                 "$ZITI_HOME/client.cert"
  server_cert:          "$ZITI_HOME/server.cert"
  key:                  "$ZITI_HOME/server.key"
  ca:                   "$ZITI_HOME/cas.cert"
ctrl:
  endpoint:             tls:${CONTROLLER_ADDRESS}:80
link:
  dialers:
    - binding: transport
listeners:
  - binding: edge
    address: tls:0.0.0.0:3022
    options:
      advertise: ${ROUTER_LOCAL_ADDRESS}:3022
      connectTimeoutMs: 1000
      getSessionTimeout: 60
  - binding: tunnel
    options:
      mode: host #tproxy|host
edge:
  csr:
    country: US
    province: NC
    locality: Charlotte
    organization: NetFoundry
    organizationalUnit: Ziti
    sans:
      dns:
        - ${ROUTER_LOCAL_ADDRESS}
        - localhost
      ip:
        - "127.0.0.1"
forwarder:
  latencyProbeInterval: 10
  xgressDialQueueLength: 1000
  xgressDialWorkerCount: 128
  linkDialQueueLength: 1000
  linkDialWorkerCount: 32
HERE
  12. enroll your router by running (replace the path to the jwt you transferred):
    ziti router enroll ${ZITI_HOME}/${ROUTER_NAME}.yaml -j /replace/this/with/the/actual.jwt
  13. run the edge router manually to make sure it starts
    ziti router run ${ZITI_HOME}/${ROUTER_NAME}.yaml

Policy Work:

  1. Go to the nfconsole and login
  2. Go to your "phone" or other "roaming" type devices, ones you want to be able to connect to the public router and mark those identities as "roaming" and "home" (or whatever attributes you like, I used roaming). The identity shown here is my laptop and I use it from home as well as from "the office" so I want it to be "roaming" so that it can use the public router when I'm not on the local network:
    image
    image776×335 18.1 KB
  3. Go to "Edge Routers" -> "Edge Router Policies". Make sure you only have one policy. I had two mapping "#auto-edge-routers" to "#all" endpoints, I expect you'll only have one, so I had to remove one, but it's important you only have one at this point. Then I changed the policy. Instead of mapping "#all" endpoints being able to access the routers with the "#auto-edge-routers" attribute, it made it so that only "roaming" endpoints can use those routers since the public edge router should already have this attribute:
    image
    image973×572 38.8 KB
  4. Define a new Edge Router Policy and give all your "home" identities access to your "#home-network" edge routers (there's only one) as shown:
    image
    image969×572 40.8 KB

You can see once I create this policy, my inovato identity (i also have a ziti-edge-tunnel running on the inovato), my windows machine, and my macmini will all be able to communicate to the local edge router now.

Wrap up

I know that looks like a lot of text and a lot of steps, but I wrote it all out to try to make it clear and explain what it's doing along the way. If you have problems connecting, look at the logs. If you see something like this NO_EDGE_ROUTERS_AVAILABLE:

Mar 19 15:04:05 inovato ziti-edge-tunnel[361191]: (361191)[ 8574.089] WARN ziti-sdk:connect.c:454 connect_get_net_session_cb() conn[0.0/Binding] failed to get 'Bind' session for service[inovato.ssh]: NO_EDGE_ROUTERS_AVAILABLE(No edge routers are assigned and online to handle the requested connection)

It probably means you have not associated your identities and edge routers properly, something is probably missing an attribute.

1 Like
5

That was perfect. The only trouble I had was that in all my tinkering after the router was running I ended up with some services not working (websites reporting connection refused). I ended up creating new services and appwans that looked identical but worked.

Thanks for all the help

2 Likes
6

Not sure if it’s all the tinkering, or the wifi on my phone was still turned on when I thought I had remote connection, but the first time I tried with my laptop it didn’t work. It’s also not working on the phone now, if it ever did. Do the routers automatically talk to each other? I also don’t have an #auto-edge-routers attribute, so I’ve just specified the public router by name. I’ve also added the endpoint running on the private router to #roaming just in case that was the magic link.

I’m not sure where to go from here, or what logs would be useful.

Thanks

7

Routers with a "Link Listener" configured will have a port open, waiting for routers to connect to it. If another router comes online, it will be informed of the router with a link listener, and it will connect to that router. So yes, routers automatically connect to each other.

[Edit] .. more to come, I clicked the wrong button...

8

My router doesn’t have a external ip, so I assume it’s the public router that’s listening?

v: 3
identity:
  cert:                 "/home/andrew/.ziti-files/OpenZiti/client.cert"
  server_cert:          "/home/andrew/.ziti-files/OpenZiti/server.cert"
  key:                  "/home/andrew/.ziti-files/OpenZiti/server.key"
  ca:                   "/home/andrew/.ziti-files/OpenZiti/cas.cert"
ctrl:
  endpoint:             tls:1cbcdfd4-c7ac-4be9-8f2e-1075845c774c.production.netfoundry.io:80
link:
  dialers:
    - binding: transport
listeners:
  - binding: edge
    address: tls:0.0.0.0:3022
    options:
      advertise: 10.63.63.95:3022
      connectTimeoutMs: 10000
      getSessionTimeout: 90
  - binding: tunnel
    options:
      mode: host #tproxy|host
edge:
  csr:
    country: US
    province: NC
    locality: Charlotte
    organization: NetFoundry
    organizationalUnit: Ziti
    sans:
      dns:
        - 10.63.63.95
        - localhost
      ip:
        - "127.0.0.1"
forwarder:
  latencyProbeInterval: 10
  xgressDialQueueLength: 1000
  xgressDialWorkerCount: 128
  linkDialQueueLength: 1000
  linkDialWorkerCount: 32
9

Routers with a "Link Listener" configured will have a port open, waiting for routers to connect to it. (since you're using CloudZiti Teams, you'll have a router like this) If another router comes online, it will be informed of the router with a link listener, and the router that just came online will try to connect to the router with a listening port. So yes, routers automatically connect to each other.

That's perfectly fine. IF you ever add another edge router that's public you'll have to update the policy, but you'll likely never have more than one? Easy enough to fix that down the road if you do...

Let's go back to the diagram (updated with attributes):

image
image751×262 49.7 KB

If this were me, here are the steps/policies I would make to get this working:

  • create five (5) identities for the five devices shown and deliver the jwt to each and enroll them onto the overlay. I assume you've done that. For each of the three (3) mobile endpoint/identities I would give the them two attributes: #home and #roaming and for each of the private home computers, I'd give those identities the #home attribute only.
  • I would find my public router and give it the #public identity (again, what you did is fine so you could skip this)
  • I would make/enroll my private "customer hosted" router and give it the attribute #private
  • I would then make two edge router policies:

    • Policy 1 - public/roaming policy. This policy allows the endpoints with the #roaming attribute to connect to the #public router. (represented by the dark blue lines)
      NOTE: the phone that is shown above has two lines. It connects to both the private and public router since it can. It'll use the one that responds first, so it should be using the private router in the diagram.

    • Policy 2 - home/private policy. This policy allows the endpoints with the #home attribute to connect to the home/private router. (represented by the green lines)

At that point, the overlay network is complete and you now need to define services and choose which identities "host" the services (done in CloudZiti when defining the service) and which identities are the "clients" or "dialers" of the service (you do that in CloudZiti with AppWANs)

Listening Routers

Technically, there are two types of listeners. One is for the overlay network equipment (think: routers) and one is for the actual overlay clients (think: phones, tablets, etc).

In your config you have a "link.dialer" and it has a binding type of transport? That is the configuration responsible for telling the private router to look for other routers and reach out and form a link. In CloudZiti, you'll get a public router deployed for you and it will have a port ready to accept links from other routers (blue box on the diagram labeled 'router link'). You do NOT have a "link.binding", so as you guessed, your router is not listening for incoming router connections, it only dialing out to establish them. (exactly one in your overlay, if you had two public routers, you would have three links. private to router1, private to router2 and router1 to router 2)

Also notice that in your config there is a "listener.binding" of type edge specified. That's so the clients that are local will be able to connect to your private router.

10

Oh and if you just need help with the services part - let me know. The logs that are usually the first two things I’ll look for are the ones from the client - the one trying to connect, and the router that is offloading the traffic (for you that’ll be the private router).

Between those two there’s almost always “the cause” listed in there (assuming the clients have access to the proper services)…

11

I believe that describes what I have configured. I’m wondering if the latency from my satellite internet is related to this not working. I increased timeouts in the private router config, just in case. The changes made to to public router to allow the endpoints to connect reliably should also apply to the private router, I would think.

I have noticed that sometimes after the laptop resumes from sleep, the endpoint can access local services, but the console is reporting it offline. Restarting the endpoint service clears that up. In all the experimenting I’ve ended up restarting everything under my control with definite changes to operation - perhaps impatience on my part though and waiting would have had the same result. Is there anyway I can force a restart of the public router?

12

That seems reasonable, yes, but I not certain since there's two different systems at play, the clients (or the 'edge') and the routers (the fabric). They may have different timeouts.

I'm not worried about the online/offline-ness reported in the console, I think that often is a best-effort type of indicator and I don't think it's realtime (though I admit I don't know, some of the CloudZiti bits I'm not 100% familiar with).

You cannot force the public router to restart and by golly I don't expect any of the equipment to need to restart unless there's a bug (which do happen). Networks are inherently unreliable and our code needs to tolerate disconnects.

It sounds like you have things setup properly so maybe we should switch gears and troubleshoot a single connection, shall we? I assume you have some kind of service you're connecting to from a phone or a laptop, right? Is that what isn't working?

Are you comfortable with sharing information here or would you prefer a DM session where we can share perhaps a bit more information?

What I'm looking for is:

  • what identity is acting as the client:
  • what os is the client using
  • what service name and intercept are configured for the client
  • what happens when the client tries to access the service

So for example, I use my mother's RDP as a good example. For that my computer running windows 11 is trying to use Windows Remote Desktop client to attach to my mother's computer using "mom.rdp" as the intercept address.

Then I can look in my logs (for this case, the ziti desktop edge for windows) to see if my client knows about the service, what port it is supposed to be on, what dns name etc.

Can we do that? I will admit, while you're not the first person to have a highly-latent connection (we have seen some "bad internet" in India sometimes) I wouldn't say it's common. So that latency might be causing issues we don't know about.

13

The raspberry pi is running OpenHAB home automation software. My iPhone 13 has the app which supports a local url (10.63.63.151:8080) and a remote url (openhab.ziti). It tries the local first and if that doesn’t work tries the remote. It can also be accessed by web browser at those addresses. The service name is OpenHAB, the intercept is openhab.ziti:80 redirecting to tcp 10.63.63.151:8080, hosted by OpenZiti (the private router endpoint). It was originally hosted by raspi1, but got changed during experimentation.

My latency runs about 700ms. An interesting note, pings to raspi1.ziti from my laptop are about 0.1ms, while pings to 10.63.63.151 are about 5ms. Pings from the phone to raspi1.ziti are 1.2ms while not connected to wifi. I’m guessing the response is actually coming from the device local endpoint and not the raspberry pi itself.

The tunnel log from the phone:

[2023-03-27T03:22:15:104Z]    INFO MobilePacketTunnelProvider:PacketTunnelProvider.swift:38 init() io.netfoundry.ZitiMobilePacketTunnel.MobilePacketTunnelProvider Version: 2.28 (462), OS: Version 16.1.1 (Build 20B101); ziti-sdk-c version 0.31.4-68c3a76(Tue-03/14/2023-20:14:37-UTC)
[2023-03-27T03:22:15:113Z]    INFO MobilePacketTunnelProvider:PacketTunnelProvider.swift:58 startTunnelAsync() 
[2023-03-27T03:22:15:113Z]    INFO MobilePacketTunnelProvider:PacketTunnelProvider.swift:59 startTunnelAsync() options=nil
[2023-03-27T03:22:15:113Z]    INFO MobilePacketTunnelProvider:PacketTunnelProvider.swift:164 loadConfig() ProviderConfig <MobilePacketTunnelProvider.ProviderConfig: 0x102f05540>
ipAddress: 100.64.0.1
subnetMask: 255.192.0.0
mtu: 4000
dns: 100.64.0.2
fallbackDnsEnabled: false
fallbackDns: 1.1.1.1
interceptMatchedDns: true
enableMfa: true
lowPowerMode: false
logLevel: 3
logRotateDaily: true
logRotateCount: 2
logRotateSizeMB: 5
[2023-03-27T03:22:15:113Z]    INFO MobilePacketTunnelProvider:PacketTunnelProvider.swift:87 startTunnelAsync() Setting log level to INFO
[2023-03-27T03:22:15:113Z]    INFO MobilePacketTunnelProvider:Logger.swift:242 updateRotateSettings() Updating log rotate config to daily:true, count:2, sizeMB:5
[2023-03-27T03:22:15:121Z]    INFO MobilePacketTunnelProvider:UserNotifications.swift:94 requestAuth() Auth request authorized? true
[2023-03-27T03:22:15:133Z]    INFO CZiti:ZitiTunnel.swift:205 loadAndRunZiti() Starting SnvMY.Q9r:"Optional("iPhoneAndrew")" at https://1cbcdfd4-c7ac-4be9-8f2e-1075845c774c.production.netfoundry.io:443
(3834)[2023-03-27T03:22:15.159Z]    INFO ziti-sdk:utils.c:173 ziti_log_set_level() set log level: root=3/INFO
(3834)[2023-03-27T03:22:15.160Z]    INFO ziti-sdk:ziti.c:426 ziti_init_async() ztx[0] Ziti C SDK version 0.31.4 @68c3a76(HEAD) starting at (2023-03-27T03:22:15.160)
(3834)[2023-03-27T03:22:15.160Z]    INFO ziti-sdk:ziti.c:428 ziti_init_async() ztx[0] using tlsuv[<unknown>], tls[mbed TLS 3.2.1]
(3834)[2023-03-27T03:22:15.160Z]    INFO ziti-sdk:ziti.c:429 ziti_init_async() ztx[0] Loading ziti context with controller[https://1cbcdfd4-c7ac-4be9-8f2e-1075845c774c.production.netfoundry.io:443]
(3834)[2023-03-27T03:22:15.160Z]    INFO ziti-sdk:ziti_ctrl.c:409 ziti_ctrl_init() ctrl[1cbcdfd4-c7ac-4be9-8f2e-1075845c774c.production.netfoundry.io] ziti controller client initialized
(3834)[2023-03-27T03:22:15.160Z]    INFO ziti-sdk:ziti.c:866 ziti_re_auth_with_cb() ztx[0] starting to re-auth with ctlr[https://1cbcdfd4-c7ac-4be9-8f2e-1075845c774c.production.netfoundry.io:443] api_session_status[0] api_session_expired[TRUE]
(3834)[2023-03-27T03:22:16.278Z]    INFO ziti-sdk:ziti.c:1532 version_cb() ztx[0] connected to controller https://1cbcdfd4-c7ac-4be9-8f2e-1075845c774c.production.netfoundry.io:443 version v0.27.5(3d9801e73809 2023-02-13T21:49:17Z)
(3834)[2023-03-27T03:22:16.483Z]    INFO ziti-sdk:ziti.c:1422 ziti_set_api_session() ztx[0] api session set, setting api_session_timer to 1740s
(3834)[2023-03-27T03:22:16.483Z]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:726 on_ziti_event() ziti_ctx[iPhoneAndrew] connected to controller
[2023-03-27T03:22:16:487Z]    INFO MobilePacketTunnelProvider:ZitiTunnelDelegate.swift:195 excludeRoute() excludeRoute 0.0.0.0 => 0.0.0.0, 255.255.255.255
[2023-03-27T03:22:16:488Z]    INFO MobilePacketTunnelProvider:ZitiTunnelDelegate.swift:195 excludeRoute() excludeRoute 0.0.0.0 => 0.0.0.0, 255.255.255.255
[2023-03-27T03:22:16:488Z]    INFO MobilePacketTunnelProvider:ZitiTunnelDelegate.swift:195 excludeRoute() excludeRoute 141.148.6.137 => 141.148.6.137, 255.255.255.255
[2023-03-27T03:22:16:489Z]    INFO MobilePacketTunnelProvider:ZitiTunnelDelegate.swift:195 excludeRoute() excludeRoute 141.148.6.137 => 141.148.6.137, 255.255.255.255
[2023-03-27T03:22:16:502Z]    INFO MobilePacketTunnelProvider:ZitiTunnelDelegate.swift:222 tunnelEventCallback() ZitiTunnelEvent: <CZiti.ZitiTunnelContextEvent: 0x102b0f960>
   identity: iPhoneAndrew:"SnvMY.Q9r"
   status: OK
   name: iPhoneAndrew
   version: v0.27.5
   controller: 
   code: 0
(3834)[2023-03-27T03:22:16.965Z]    INFO ziti-sdk:channel.c:234 new_ziti_channel() ch[0] (OpenZiti@tls://10.63.63.95:3022) new channel for ztx[0] identity[iPhoneAndrew]
(3834)[2023-03-27T03:22:16.965Z]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:797 on_ziti_event() ztx[iPhoneAndrew] added edge router OpenZiti@tls://10.63.63.95:3022@10.63.63.95
[2023-03-27T03:22:16:967Z]    INFO MobilePacketTunnelProvider:ZitiTunnelDelegate.swift:195 excludeRoute() excludeRoute 10.63.63.95 => 10.63.63.95, 255.255.255.255
[2023-03-27T03:22:16:967Z]    INFO MobilePacketTunnelProvider:ZitiTunnelDelegate.swift:195 excludeRoute() excludeRoute 10.63.63.95 => 10.63.63.95, 255.255.255.255
(3834)[2023-03-27T03:22:16.965Z]    INFO ziti-sdk:channel.c:733 reconnect_channel() ch[0] reconnecting NOW
(3834)[2023-03-27T03:22:16.965Z]    INFO ziti-sdk:channel.c:234 new_ziti_channel() ch[1] (OCI us-ashburn-1 Edge Router 1@tls://91c54d8a-0e63-43aa-9273-140312049c5f.production.netfoundry.io:443) new channel for ztx[0] identity[iPhoneAndrew]
(3834)[2023-03-27T03:22:16.965Z]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:797 on_ziti_event() ztx[iPhoneAndrew] added edge router OCI us-ashburn-1 Edge Router 1@tls://91c54d8a-0e63-43aa-9273-140312049c5f.production.netfoundry.io:443@91c54d8a-0e63-43aa-9273-140312049c5f.production.netfoundry.io
[2023-03-27T03:22:17:057Z]    INFO MobilePacketTunnelProvider:ZitiTunnelDelegate.swift:195 excludeRoute() excludeRoute 0.0.0.0 => 0.0.0.0, 255.255.255.255
[2023-03-27T03:22:17:058Z]    INFO MobilePacketTunnelProvider:ZitiTunnelDelegate.swift:195 excludeRoute() excludeRoute 0.0.0.0 => 0.0.0.0, 255.255.255.255
[2023-03-27T03:22:17:058Z]    INFO MobilePacketTunnelProvider:ZitiTunnelDelegate.swift:195 excludeRoute() excludeRoute 129.80.69.210 => 129.80.69.210, 255.255.255.255
[2023-03-27T03:22:17:058Z]    INFO MobilePacketTunnelProvider:ZitiTunnelDelegate.swift:195 excludeRoute() excludeRoute 129.80.69.210 => 129.80.69.210, 255.255.255.255
(3834)[2023-03-27T03:22:16.965Z]    INFO ziti-sdk:channel.c:733 reconnect_channel() ch[1] reconnecting NOW
(3834)[2023-03-27T03:22:17.381Z]    INFO tunnel-cbs:ziti_tunnel_cbs.c:417 new_ziti_intercept() creating intercept for service[OpenHAB] with ziti-tunneler-client.v1 = {"hostname":"openhab.ziti","port":80}
(3834)[2023-03-27T03:22:17.381Z]    INFO tunnel-cbs:ziti_dns.c:296 new_ipv4_entry() registered DNS entry openhab.ziti -> 100.64.0.3
(3834)[2023-03-27T03:22:17.381Z]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:686 on_service() starting intercepting for service[OpenHAB]
(3834)[2023-03-27T03:22:17.381Z]    INFO tunnel-cbs:ziti_tunnel_cbs.c:417 new_ziti_intercept() creating intercept for service[Weewx] with ziti-tunneler-client.v1 = {"hostname":"weewx.ziti","port":80}
(3834)[2023-03-27T03:22:17.381Z]    INFO tunnel-cbs:ziti_dns.c:296 new_ipv4_entry() registered DNS entry weewx.ziti -> 100.64.0.4
(3834)[2023-03-27T03:22:17.381Z]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:686 on_service() starting intercepting for service[Weewx]
(3834)[2023-03-27T03:22:17.381Z]    INFO tunnel-cbs:ziti_tunnel_cbs.c:417 new_ziti_intercept() creating intercept for service[SSH-raspi1] with ziti-tunneler-client.v1 = {"hostname":"raspi1.ziti","port":22}
(3834)[2023-03-27T03:22:17.381Z]    INFO tunnel-cbs:ziti_dns.c:296 new_ipv4_entry() registered DNS entry raspi1.ziti -> 100.64.0.5
(3834)[2023-03-27T03:22:17.381Z]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:686 on_service() starting intercepting for service[SSH-raspi1]
(3834)[2023-03-27T03:22:17.381Z]    INFO tunnel-cbs:ziti_tunnel_cbs.c:417 new_ziti_intercept() creating intercept for service[SSH-raspi2] with ziti-tunneler-client.v1 = {"hostname":"raspi2.ziti","port":22}
(3834)[2023-03-27T03:22:17.381Z]    INFO tunnel-cbs:ziti_dns.c:296 new_ipv4_entry() registered DNS entry raspi2.ziti -> 100.64.0.6
(3834)[2023-03-27T03:22:17.381Z]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:686 on_service() starting intercepting for service[SSH-raspi2]
(3834)[2023-03-27T03:22:17.381Z]    INFO tunnel-cbs:ziti_tunnel_cbs.c:417 new_ziti_intercept() creating intercept for service[Home-router] with ziti-tunneler-client.v1 = {"hostname":"router.ziti","port":80}
(3834)[2023-03-27T03:22:17.381Z]    INFO tunnel-cbs:ziti_dns.c:296 new_ipv4_entry() registered DNS entry router.ziti -> 100.64.0.7
(3834)[2023-03-27T03:22:17.381Z]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:686 on_service() starting intercepting for service[Home-router]
[2023-03-27T03:22:17:383Z]    INFO MobilePacketTunnelProvider:ZitiTunnelDelegate.swift:222 tunnelEventCallback() ZitiTunnelEvent: <CZiti.ZitiTunnelServiceEvent: 0x102d14130>
   identity: iPhoneAndrew:"SnvMY.Q9r"
   status: 
   removed: (0)
   added: (5)
      0:{"ziti-tunneler-server.v1":{"protocol":"tcp","hostname":"10.63.63.151","port":8080},"id":"sHtk6vp7b3U3HQ4Zi0BwV","postureQuerySets":[{"policyId":"P0YVyj3uZdarHAR6m32wf","policyType":"Dial","isPassing":true}],"encrypted":true,"permFlags":1,"name":"OpenHAB","ziti-tunneler-client.v1":{"hostname":"openhab.ziti","port":80}}
      1:{"ziti-tunneler-server.v1":{"protocol":"tcp","hostname":"10.63.63.151","port":80},"id":"6aINPjT5RQNxwVkB6cQRk4","postureQuerySets":[{"policyId":"34Ug6MPnhJdZgAvOoKwLs8","policyType":"Dial","isPassing":true}],"encrypted":true,"permFlags":1,"name":"Weewx","ziti-tunneler-client.v1":{"hostname":"weewx.ziti","port":80}}
      2:{"ziti-tunneler-server.v1":{"protocol":"tcp","hostname":"localhost","port":22},"id":"5aaikmP4FUgmfIPGSWKV6f","postureQuerySets":[{"policyId":"4ZUO8P8IBdC0JbzKjQN0Zl","policyType":"Dial","isPassing":true}],"encrypted":true,"permFlags":1,"name":"SSH-raspi1","ziti-tunneler-client.v1":{"hostname":"raspi1.ziti","port":22}}
      3:{"ziti-tunneler-server.v1":{"protocol":"tcp","hostname":"localhost","port":22},"id":"1cvY9fD1L8hXxCE9ffzYHI","postureQuerySets":[{"policyId":"4ZUO8P8IBdC0JbzKjQN0Zl","policyType":"Dial","isPassing":true}],"encrypted":true,"permFlags":1,"name":"SSH-raspi2","ziti-tunneler-client.v1":{"hostname":"raspi2.ziti","port":22}}
      4:{"ziti-tunneler-server.v1":{"protocol":"tcp","hostname":"10.63.63.1","port":80},"id":"1GBnec0CZB7nbqk1i5J2W2","postureQuerySets":[{"policyId":"5QoKIQPwvSqgpxfQxByZGw","policyType":"Dial","isPassing":true}],"encrypted":true,"permFlags":1,"name":"Home-router","ziti-tunneler-client.v1":{"hostname":"router.ziti","port":80}}

[2023-03-27T03:22:17:400Z]    INFO MobilePacketTunnelProvider:PacketTunnelProvider.swift:207 updateTunnelNetworkSettings() route: 100.64.0.1 / 255.192.0.0
[2023-03-27T03:22:17:400Z]    INFO MobilePacketTunnelProvider:PacketTunnelProvider.swift:210 updateTunnelNetworkSettings() excluding route: 0.0.0.0 / 255.255.255.255
[2023-03-27T03:22:17:400Z]    INFO MobilePacketTunnelProvider:PacketTunnelProvider.swift:210 updateTunnelNetworkSettings() excluding route: 141.148.6.137 / 255.255.255.255
[2023-03-27T03:22:17:401Z]    INFO MobilePacketTunnelProvider:PacketTunnelProvider.swift:210 updateTunnelNetworkSettings() excluding route: 10.63.63.95 / 255.255.255.255
[2023-03-27T03:22:17:401Z]    INFO MobilePacketTunnelProvider:PacketTunnelProvider.swift:210 updateTunnelNetworkSettings() excluding route: 129.80.69.210 / 255.255.255.255
[2023-03-27T03:22:17:401Z]    INFO MobilePacketTunnelProvider:PacketTunnelProvider.swift:207 updateTunnelNetworkSettings() route: 100.64.0.1 / 255.192.0.0
[2023-03-27T03:22:17:401Z]    INFO MobilePacketTunnelProvider:PacketTunnelProvider.swift:210 updateTunnelNetworkSettings() excluding route: 0.0.0.0 / 255.255.255.255
[2023-03-27T03:22:17:401Z]    INFO MobilePacketTunnelProvider:PacketTunnelProvider.swift:210 updateTunnelNetworkSettings() excluding route: 141.148.6.137 / 255.255.255.255
[2023-03-27T03:22:17:402Z]    INFO MobilePacketTunnelProvider:PacketTunnelProvider.swift:210 updateTunnelNetworkSettings() excluding route: 10.63.63.95 / 255.255.255.255
[2023-03-27T03:22:17:402Z]    INFO MobilePacketTunnelProvider:PacketTunnelProvider.swift:210 updateTunnelNetworkSettings() excluding route: 129.80.69.210 / 255.255.255.255
[2023-03-27T03:22:17:402Z]    INFO MobilePacketTunnelProvider:PacketTunnelProvider.swift:379 logNetworkPath() Network Path Update:
Status:satisfied, Expensive:true, Cellular:true, DNS:true
   Interfaces: 
     3: name:pdp_ip0, type:cellular
(3834)[2023-03-27T03:22:17.484Z]    INFO ziti-sdk:posture.c:204 ziti_send_posture_data() ztx[0] first run or potential controller restart detected
[2023-03-27T03:22:17:695Z]    INFO MobilePacketTunnelProvider:PacketTunnelProvider.swift:379 logNetworkPath() Network Path Update:
Status:satisfied, Expensive:true, Cellular:true, DNS:true
   Interfaces: 
     3: name:pdp_ip0, type:cellular 
     25: name:utun5, type:other
(3834)[2023-03-27T03:22:17.743Z]    INFO ziti-sdk:channel.c:631 hello_reply_cb() ch[1] connected. EdgeRouter version: v0.27.5|3d9801e73809|2023-02-13T21:49:17Z|linux|amd64
(3834)[2023-03-27T03:22:17.743Z]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:801 on_ziti_event() ztx[iPhoneAndrew] router OCI us-ashburn-1 Edge Router 1@tls://91c54d8a-0e63-43aa-9273-140312049c5f.production.netfoundry.io:443 connected
[2023-03-27T03:22:19:842Z]    INFO MobilePacketTunnelProvider:PacketTunnelProvider.swift:379 logNetworkPath() Network Path Update:
Status:satisfied, Expensive:true, Cellular:true, DNS:true
   Interfaces: 
     3: name:pdp_ip0, type:cellular 
     25: name:utun5, type:other
[2023-03-27T03:22:28:541Z]    INFO MobilePacketTunnelProvider:PacketTunnelProvider.swift:379 logNetworkPath() Network Path Update:
Status:satisfied, Expensive:true, Cellular:true, DNS:true
   Interfaces: 
     3: name:pdp_ip0, type:cellular 
     25: name:utun5, type:other
[2023-03-27T03:22:31:645Z]    INFO MobilePacketTunnelProvider:PacketTunnelProvider.swift:379 logNetworkPath() Network Path Update:
Status:satisfied, Expensive:true, Cellular:true, DNS:true
   Interfaces: 
     3: name:pdp_ip0, type:cellular 
     25: name:utun5, type:other
(3834)[2023-03-27T03:22:37.059Z]   ERROR ziti-sdk:channel.c:672 ch_connect_timeout() ch[0] connect timeout
(3834)[2023-03-27T03:22:37.059Z]    INFO ziti-sdk:channel.c:730 reconnect_channel() ch[0] reconnecting in 2073 ms (attempt = 1)
(3834)[2023-03-27T03:22:37.059Z]   ERROR ziti-sdk:channel.c:858 on_channel_connect_internal() ch[0] failed to connect [-89/operation canceled]
(3834)[2023-03-27T03:22:37.059Z]    INFO ziti-sdk:channel.c:730 reconnect_channel() ch[0] reconnecting in 870 ms (attempt = 2)

Attempts to connect to openhab.ziti in both the app and Safari timeout

These are the most recent messages from systemctl status on the private router VM:

Mar 26 20:26:27 antlet19 ziti[18529]: {"_channels":["establishPath"],"apiSessionId":"clfpzubmftpbnk0mjjfdiwqmz","attemptNumber":"1","binding":"edge","circuitId":"TM-fLoeCu","destination":"b3b31184-38d5-467b-9c7d-a02e09819743","file":"github.com/openziti/edge@v0.24.125/router/xgress_edge_tunnel/dialer.go:60","func":"github.com/openziti/edge/router/xgress_edge_tunnel.(*tunneler).Dial","level":"info","msg":"successful connection 10.63.63.95:44448-\u003e10.63.63.151:8080 for destination b3b31184-38d5-467b-9c7d-a02e09819743","serviceId":"sHtk6vp7b3U3HQ4Zi0BwV","sessionId":"clfq25cw2tshhk0mjuh9mlz4d","time":"2023-03-26T20:26:27.559Z"}
Mar 26 20:26:59 antlet19 ziti[18529]: {"_channels":["link","linkDialer"],"address":"tls:1cbcdfd4-c7ac-4be9-8f2e-1075845c774c.production.netfoundry.io:6262","file":"github.com/openziti/fabric@v0.22.24/router/handler_ctrl/dial.go:98","func":"github.com/openziti/fabric/router/handler_ctrl.(*dialHandler).handle","level":"info","linkId":"168LSTDBLVAe0LvDmdz5Yg","linkProtocol":"tls","msg":"dialing link","routerId":"gBxvTOQ9cd","routerVersion":"v0.27.5","time":"2023-03-26T20:26:59.450Z"}
Mar 26 20:27:00 antlet19 ziti[18529]: {"_channels":["link","linkDialer"],"address":"tls:1cbcdfd4-c7ac-4be9-8f2e-1075845c774c.production.netfoundry.io:6262","error":"error dialing outgoing link [l/168LSTDBLVAe0LvDmdz5Yg]: error dialing payload channel for [l/168LSTDBLVAe0LvDmdz5Yg]: context deadline exceeded","file":"github.com/openziti/fabric@v0.22.24/router/handler_ctrl/dial.go:109","func":"github.com/openziti/fabric/router/handler_ctrl.(*dialHandler).handle","level":"error","linkId":"168LSTDBLVAe0LvDmdz5Yg","linkProtocol":"tls","msg":"link dialing failed","routerId":"gBxvTOQ9cd","routerVersion":"v0.27.5","time":"2023-03-26T20:27:00.451Z"}
Mar 26 20:28:01 antlet19 ziti[18529]: {"_channels":["link","linkDialer"],"address":"tls:1cbcdfd4-c7ac-4be9-8f2e-1075845c774c.production.netfoundry.io:6262","file":"github.com/openziti/fabric@v0.22.24/router/handler_ctrl/dial.go:98","func":"github.com/openziti/fabric/router/handler_ctrl.(*dialHandler).handle","level":"info","linkId":"2GauWe1jDBcL3UyL6M7Z0d","linkProtocol":"tls","msg":"dialing link","routerId":"gBxvTOQ9cd","routerVersion":"v0.27.5","time":"2023-03-26T20:28:01.085Z"}
Mar 26 20:28:02 antlet19 ziti[18529]: {"_channels":["link","linkDialer"],"address":"tls:1cbcdfd4-c7ac-4be9-8f2e-1075845c774c.production.netfoundry.io:6262","error":"error dialing outgoing link [l/2GauWe1jDBcL3UyL6M7Z0d]: error dialing payload channel for [l/2GauWe1jDBcL3UyL6M7Z0d]: context deadline exceeded","file":"github.com/openziti/fabric@v0.22.24/router/handler_ctrl/dial.go:109","func":"github.com/openziti/fabric/router/handler_ctrl.(*dialHandler).handle","level":"error","linkId":"2GauWe1jDBcL3UyL6M7Z0d","linkProtocol":"tls","msg":"link dialing failed","routerId":"gBxvTOQ9cd","routerVersion":"v0.27.5","time":"2023-03-26T20:28:02.086Z"}
14

I have confirmation that the link dialer also has a setting that you can change (also defaults to 1 second currently.

On the private router update this section and restart the router:

link:
  dialers:
    - binding: transport
      options:
        connectTimeoutMs: 10000

I don't know if it'll matter, but maybe that's somehow getting in the way.

Yes, ping is ICMP, OpenZiti doesn't really support that in the way most people are used to so pings appear unnaturally fast...

Lastly, just to confirm, this is a problem when your local -- or -- roaming, right? It's not just a problem when roaming?

15

As Clint says, ICMP and ping is not supported. Therefore we created a ping utility - zping command - github.com/openziti/sdk-golang/example/zping - Go Packages. You can also extract latencies from the fabric itself.

16

Adding the timeout didn’t seem to change anything. Everything local is working. It’s when I try to access service remotely that it fails.

I’m still seeing errors in the router log.

Mar 27 12:05:19 antlet19 ziti[7292]: {"_channels":["establishPath"],"apiSessionId":"clfquw8wmutvpk0mjsl0tkqnt","attemptNumber":"1","binding":"edge","circuitId":"y8HF5eeSu","destination":"4d21fba8-9361-4da9-a0fa-8b59927d2a7a","file":"github.com/openziti/edge@v0.24.125/router/xgress_edge_tunnel/dialer.go:60","func":"github.com/openziti/edge/router/xgress_edge_tunnel.(*tunneler).Dial","level":"info","msg":"successful connection 10.63.63.95:38346-\u003e10.63.63.151:80 for destination 4d21fba8-9361-4da9-a0fa-8b59927d2a7a","serviceId":"6aINPjT5RQNxwVkB6cQRk4","sessionId":"clfquwjxqutwak0mj4ly9q2fd","time":"2023-03-27T12:05:19.419Z"}
Mar 27 12:05:21 antlet19 ziti[7292]: {"_channels":["establishPath"],"apiSessionId":"clfquw8wmutvpk0mjsl0tkqnt","attemptNumber":"1","binding":"edge","circuitId":"xZNFUeeSu","destination":"4d21fba8-9361-4da9-a0fa-8b59927d2a7a","file":"github.com/openziti/edge@v0.24.125/router/xgress_edge_tunnel/dialer.go:60","func":"github.com/openziti/edge/router/xgress_edge_tunnel.(*tunneler).Dial","level":"info","msg":"successful connection 10.63.63.95:38348-\u003e10.63.63.151:80 for destination 4d21fba8-9361-4da9-a0fa-8b59927d2a7a","serviceId":"6aINPjT5RQNxwVkB6cQRk4","sessionId":"clfquwjxqutwak0mj4ly9q2fd","time":"2023-03-27T12:05:21.002Z"}
Mar 27 12:05:21 antlet19 ziti[7292]: {"_channels":["establishPath"],"apiSessionId":"clfquw8wmutvpk0mjsl0tkqnt","attemptNumber":"1","binding":"edge","circuitId":"BCveUeFCy","destination":"4d21fba8-9361-4da9-a0fa-8b59927d2a7a","file":"github.com/openziti/edge@v0.24.125/router/xgress_edge_tunnel/dialer.go:60","func":"github.com/openziti/edge/router/xgress_edge_tunnel.(*tunneler).Dial","level":"info","msg":"successful connection 10.63.63.95:38352-\u003e10.63.63.151:80 for destination 4d21fba8-9361-4da9-a0fa-8b59927d2a7a","serviceId":"6aINPjT5RQNxwVkB6cQRk4","sessionId":"clfquwjxqutwak0mj4ly9q2fd","time":"2023-03-27T12:05:21.164Z"}
Mar 27 12:05:21 antlet19 ziti[7292]: {"_channels":["establishPath"],"apiSessionId":"clfquw8wmutvpk0mjsl0tkqnt","attemptNumber":"1","binding":"edge","circuitId":"fSNeUeeCy","destination":"4d21fba8-9361-4da9-a0fa-8b59927d2a7a","file":"github.com/openziti/edge@v0.24.125/router/xgress_edge_tunnel/dialer.go:60","func":"github.com/openziti/edge/router/xgress_edge_tunnel.(*tunneler).Dial","level":"info","msg":"successful connection 10.63.63.95:38350-\u003e10.63.63.151:80 for destination 4d21fba8-9361-4da9-a0fa-8b59927d2a7a","serviceId":"6aINPjT5RQNxwVkB6cQRk4","sessionId":"clfquwjxqutwak0mj4ly9q2fd","time":"2023-03-27T12:05:21.164Z"}
Mar 27 12:06:01 antlet19 ziti[7292]: {"_channels":["establishPath"],"apiSessionId":"clfquw8wmutvpk0mjsl0tkqnt","attemptNumber":"1","binding":"edge","circuitId":"qy0j5FFSuQ","destination":"02f3effe-c711-451a-bb0f-15e35d44a92a","file":"github.com/openziti/edge@v0.24.125/router/xgress_edge_tunnel/dialer.go:60","func":"github.com/openziti/edge/router/xgress_edge_tunnel.(*tunneler).Dial","level":"info","msg":"successful connection 10.63.63.95:52572-\u003e10.63.63.151:8080 for destination 02f3effe-c711-451a-bb0f-15e35d44a92a","serviceId":"sHtk6vp7b3U3HQ4Zi0BwV","sessionId":"clfquwdrcutvxk0mjn996pks8","time":"2023-03-27T12:06:01.943Z"}
Mar 27 12:06:02 antlet19 ziti[7292]: {"_channels":["link","linkDialer"],"address":"tls:1cbcdfd4-c7ac-4be9-8f2e-1075845c774c.production.netfoundry.io:6262","file":"github.com/openziti/fabric@v0.22.24/router/handler_ctrl/dial.go:98","func":"github.com/openziti/fabric/router/handler_ctrl.(*dialHandler).handle","level":"info","linkId":"TBlBhMPr6SKspzKHV8NqR","linkProtocol":"tls","msg":"dialing link","routerId":"gBxvTOQ9cd","routerVersion":"v0.27.5","time":"2023-03-27T12:06:02.121Z"}
Mar 27 12:06:04 antlet19 ziti[7292]: {"_context":"tls:1cbcdfd4-c7ac-4be9-8f2e-1075845c774c.production.netfoundry.io:6262","error":"EOF","file":"github.com/openziti/channel/v2@v2.0.27/classic_dialer.go:69","func":"github.com/openziti/channel/v2.(*classicDialer).Create","level":"warning","msg":"error initiating channel with hello","time":"2023-03-27T12:06:04.420Z"}
Mar 27 12:06:04 antlet19 ziti[7292]: {"file":"github.com/openziti/channel/v2@v2.0.27/message.go:649","func":"github.com/openziti/channel/v2.getRetryVersionFor","level":"info","msg":"defaulting to version 2","time":"2023-03-27T12:06:04.420Z"}
Mar 27 12:06:04 antlet19 ziti[7292]: {"_context":"tls:1cbcdfd4-c7ac-4be9-8f2e-1075845c774c.production.netfoundry.io:6262","file":"github.com/openziti/channel/v2@v2.0.27/classic_dialer.go:73","func":"github.com/openziti/channel/v2.(*classicDialer).Create","level":"warning","msg":"Retrying dial with protocol version 2","time":"2023-03-27T12:06:04.420Z"}
Mar 27 12:06:06 antlet19 ziti[7292]: {"_channels":["link","linkDialer"],"address":"tls:1cbcdfd4-c7ac-4be9-8f2e-1075845c774c.production.netfoundry.io:6262","error":"error dialing outgoing link [l/TBlBhMPr6SKspzKHV8NqR]: error dialing payload channel for [l/TBlBhMPr6SKspzKHV8NqR]: EOF","file":"github.com/openziti/fabric@v0.22.24/router/handler_ctrl/dial.go:109","func":"github.com/openziti/fabric/router/handler_ctrl.(*dialHandler).handle","level":"error","linkId":"TBlBhMPr6SKspzKHV8NqR","linkProtocol":"tls","msg":"link dialing failed","routerId":"gBxvTOQ9cd","routerVersion":"v0.27.5","time":"2023-03-27T12:06:06.146Z"}
17

The EOF makes me think the router you’re dialing is closing the incoming connection. Can you check the router logs on the other side and see if you can see a corresponding error, please?

18

The router on the other side is the NetFoundry one. I don’t how, or if I can, check the logs on it.

19

I’ve been unable to get remote access working. The NetFoundry console says that the private router and the associated endpoint are online. Still seeing the same errors in the router log though. I’m unable to check the logs for the public router so don’t know if there’s any further info there.

The logs on my phone show all the advertised services that I’m expecting to see. The simplest is a webpage hosted on the raspberry pi. I get “Safari cannot open the page because it could not connect to the server” almost immediately. I mistakenly tried accessing one that the phone wasn’t authorized for, and instead of an immediate failure, Safari just seemed to hang - not sure if significant or relevant. If wifi is turned on before the tunneller is started, the pages load.

20

Update. I’ve abandoned the private router and reverted to having all traffic go through the NetFoundry hosted router - seems like abuse of a free service and given my satellite based internet, provides less than stellar network performance. However, the exceptional initial support totally disappeared after the private router was created and sort of working. Disappointing, because by that point I was grasping just how flexible Openziti is and how it could be used.