# Ziti desktop edge + controller/network version compatibility?

**URL:** <https://openziti.discourse.group/t/ziti-desktop-edge-controller-network-version-compatibility/3274>\
**Category:** Ziti Desktop Edge for MacOS\
**Created:** [October 18, 2024, 9:18pm UTC](https://openziti.discourse.group/t/ziti-desktop-edge-controller-network-version-compatibility/3274 "2024-10-18T21:18:12Z")\
**Posts on this page:** 19\
**Page:** 1

<div class="post-metadata">

**Author:** ![woodwardjd](https://avatars.discourse-cdn.com/v4/letter/w/ac91a4/32.png) [@woodwardjd](https://openziti.discourse.group/u/woodwardjd)\
**Post date:** [October 18, 2024, 9:18pm UTC](https://openziti.discourse.group/t/ziti-desktop-edge-controller-network-version-compatibility/3274/1 "2024-10-18T21:18:12Z")

</div>

Hi! Today I went to use a resource I have on a small self-managed ziti network from my macOS laptop like I have a couple times a week for a couple of years (thanks for helping me set it up, you know who you are!). But, Desktop Edge doesn't show me my services like it usually does. It shows me as "Status: Connected", but with a red icon next to my client name in the left panel, and no services showing, instead of the 4 that usually show.

I think I remember taking the app store update for Ziti Desktop Edge the other day (running 2.44 (522) now), and pretty sure I haven't used it since. But I also updated to macOS 14.7 at roughly the same time.

I know I'm a terrible person for doing this, but... here's the kicker: I haven't updated the controller / router since I installed it using the quickstart non-docker method. A while ago. So it / they're at v0.27.2. Yup :hangs-head-in-shame:.

Any chance there's an incompatibility between that (old) router and desktop edge? This has been rock solid for the single user (me) plus 2 occasional users over the past couple of years.

Why haven't I upgraded? A combination of laziness and lack of confidence in the upgrade process. I took a look around this morning and didn't find "upgrade" instructions for the "quickstart no docker" method. And I don't see instructions on how to make a docker install use the data from a quickstart. And I'm not sure if I have to upgrade to intermediate versions as stepping stones. Got pointers to docs that provide guidance? I'm definitely capable of muddling my way through this, but clear guidance \> educated guesswork. So I figured I'd ask around first before upgrading RIGHT NOW to get work done, or finding another temporary fix.

Thanks!

Jason

---

<div class="post-metadata">

**Author:** ![TheLumberjack](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/thelumberjack/32/113_2.png) [@TheLumberjack](https://openziti.discourse.group/u/TheLumberjack)\
**Post date:** [October 19, 2024, 1:18am UTC](https://openziti.discourse.group/t/ziti-desktop-edge-controller-network-version-compatibility/3274/2 "2024-10-19T01:18:48Z")

</div>

> [@woodwardjd](#):
>
> Any chance there's an incompatibility between that (old) router and desktop edge?

Hi @woodwardjd, nice to see you back in the forums! 🙂 Yes there's a chance. It's something that was just discovered, and I'm not actively working on it myself so I don't know the details. I did see a message indicating an update of the overlay network is likely to fix the issue. I'm not well-versed enough with the ZDEM versions to know if you're affected but I definitely think it's worthwhile to update the overlay.

The upgrade process is pretty straightforward. For the quickstart, it's really easy. You can just stop the controller/router and copy the directory. This has the benfit of capturing the database as well as your config files and your PKI... So I'd recommend you do that "here and there" (or you know, _routinely_... 🙂 )

You can also backup the database itself by running `ziti edge db snapshot` at any time the controller is running if you don't want to stop the overlay network.

Once backed up, just download a new binary on top of the existing quickstart path. The version will be mismatched, it'll be in the "0.27.2" folder, but it'll be verison 1.1.15 (or whatever) but then you won't need to change the systemd files. OR you can pull a new binary and update the systemd unit files and restart. The overlay should migrate itself and you'll be back running.

Hope that helps,  
-Clint

---

<div class="post-metadata">

**Author:** ![woodwardjd](https://avatars.discourse-cdn.com/v4/letter/w/ac91a4/32.png) [@woodwardjd](https://openziti.discourse.group/u/woodwardjd)\
**Post date:** [October 19, 2024, 1:54am UTC](https://openziti.discourse.group/t/ziti-desktop-edge-controller-network-version-compatibility/3274/3 "2024-10-19T01:54:33Z")

</div>

Thanks! Will give that a shot first thing tomorrow, and report back.

---

<div class="post-metadata">

**Author:** ![scareything](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/scareything/32/106_2.png) [@scareything](https://openziti.discourse.group/u/scareything)\
**Post date:** [October 19, 2024, 1:56am UTC](https://openziti.discourse.group/t/ziti-desktop-edge-controller-network-version-compatibility/3274/4 "2024-10-19T01:56:36Z")

</div>

FYI the latest build in the App Store (2.45) as of minutes ago resolves an issue that would prevent you from hosting services when you’re connecting to an older controller. Your issue sounds a little different thought so I’m not 100% sure that it will get you back in the game, but it’s easy enough to try so I thought I’d mention it.

---

<div class="post-metadata">

**Author:** ![woodwardjd](https://avatars.discourse-cdn.com/v4/letter/w/ac91a4/32.png) [@woodwardjd](https://openziti.discourse.group/u/woodwardjd)\
**Post date:** [October 19, 2024, 12:30pm UTC](https://openziti.discourse.group/t/ziti-desktop-edge-controller-network-version-compatibility/3274/5 "2024-10-19T12:30:22Z")

</div>

> latest build in the App Store (2.45) as of minutes ago

Good call, thanks, but no dice. On to the router/controller (and then probably console) upgrade!

---

<div class="post-metadata">

**Author:** ![woodwardjd](https://avatars.discourse-cdn.com/v4/letter/w/ac91a4/32.png) [@woodwardjd](https://openziti.discourse.group/u/woodwardjd)\
**Post date:** [October 19, 2024, 1:17pm UTC](https://openziti.discourse.group/t/ziti-desktop-edge-controller-network-version-compatibility/3274/6 "2024-10-19T13:17:23Z")

</div>

@TheLumberjack had to do a little more work because 0.27 was before the unification into a single executable in 0.29. While I was in the service editing for that I also changed them to reference `.../ziti-bin/current/ziti`, where `current` is a symlink to a directory, say, `ziti-v1.1.15`. So upgrading will be easy in the future. I will say that was pretty easy ("just replace an executable"), but I didn't have good confidence that was the only thing that was needed.

All that being said... still the same issue ☹ I even upgraded the web console to poke around (looks very nice!).

Gonna dig into logging, server side, to try to figure out what's going on.

The web console Identity Management -\> my identity -\> Visualizer shows connection between my identity (macOS ZDE) to the router as "red" "errored" and the popup shows apiSession No routerConnection No.

ZDE shows "connected" with no services. It also shows Controller Version of v0.27.2 still. I've confirmed via `ps` and inspection of the command line that it's using the v1.1.15 executable (and the new single-command style).

More when I have it!

---

<div class="post-metadata">

**Author:** ![woodwardjd](https://avatars.discourse-cdn.com/v4/letter/w/ac91a4/32.png) [@woodwardjd](https://openziti.discourse.group/u/woodwardjd)\
**Post date:** [October 19, 2024, 1:23pm UTC](https://openziti.discourse.group/t/ziti-desktop-edge-controller-network-version-compatibility/3274/7 "2024-10-19T13:23:51Z")

</div>

I tried creating a new Identity. When I clicked "Enroll" in ZDE (after selecting the JWT) it tells me "Unable to enroll woodwardjd-walnut-2.jwt CONTROLLER\_UNAVAILABLE".

Controller process is running. No logs indicate it's crashed and been restarted by systemd.

Continuing to dig.

---

<div class="post-metadata">

**Author:** ![woodwardjd](https://avatars.discourse-cdn.com/v4/letter/w/ac91a4/32.png) [@woodwardjd](https://openziti.discourse.group/u/woodwardjd)\
**Post date:** [October 19, 2024, 1:37pm UTC](https://openziti.discourse.group/t/ziti-desktop-edge-controller-network-version-compatibility/3274/8 "2024-10-19T13:37:57Z")

</div>

Found some logs (hey, they're in syslog like they should be, unlike most of the applications I deal with on a daily basis which put their logs in some random place).

When ZDE is open and is in "Connecting..." /OR/ "Connected" state, lines like this come up every few seconds, with the client port changing (that remote IP is my ISP's carrier-grade NAT outlet IPv4 address)

```auto
Oct 19 13:23:15 my-server ziti[457428]: {"_context":"tls:0.0.0.0:8441","error":"local error: tls: bad record MAC","file":"github.com/openziti/transport/v2@v2.0.146/tls/listener.go:257","func":"github.com/openziti/transport/v2/tls.(*sharedListener).processConn","level":"error","msg":"handshake failed","remote":"8.47.100.15:25566","time":"2024-10-19T13:23:15.194Z"}

```

Going back in the syslog I see these started when I upgraded the router/controller executable.

... BUT ... there were similar ones prior to the router/controller executable upgrade, but AFTER the macOS 14.7 upgrade / ZDE upgrade

```auto
Oct 18 12:17:06 my-server ziti-controller[3637757]: {"level":"info","msg":"http: TLS handshake error from 8.47.100.15:25398: local error: tls: bad record MAC","time":"2024-10-18T12:17:06.821Z"}

```

Prior to that there were periodic "read connection timed out" errors, going back "forever" (or at least a couple of weeks, during which "everything worked fine"), but none of these MAC errors

```auto
Oct 17 11:15:19 my-server ziti-router[3637772]: {"_context":"ch{edge}-\u003eu{classic}-\u003ei{gj2B}","file":"github.com/openziti/channel/v2@v2.0.25/impl.go:320","func":"github.com/openziti/channel/v2.(*channelImpl).rxer","level":"error","msg":"rx error (read tcp my.server.ip.address:8442-\u003e8.47.100.15:25598: read: connection timed out)","time":"2024-10-17T11:15:19.191Z"}

```

---

<div class="post-metadata">

**Author:** ![TheLumberjack](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/thelumberjack/32/113_2.png) [@TheLumberjack](https://openziti.discourse.group/u/TheLumberjack)\
**Post date:** [October 19, 2024, 2:33pm UTC](https://openziti.discourse.group/t/ziti-desktop-edge-controller-network-version-compatibility/3274/9 "2024-10-19T14:33:41Z")

</div>

> [@woodwardjd](#):
>
> before the unification into a single executable in 0.29

wow. seems like forever ago that we did that! 🙂

bad record MAC/tls errors always make me think the PKI has changed in some way. Are you comfortable sharing the external url of the controller? I'll poke at it - or if you'd rather learn I can show you how to diagnose these issues.

First thing I'd do is run `ziti ops verify-network`.

```auto
ziti ops verify-traffic --help
A tool to verify traffic can flow over the overlay properly. You must be authenticated to use this tool.

Usage:
  ziti ops verify-traffic [flags]

Flags:
      --allow-multiple-servers Whether to allows the same server multiple times.
      --cleanup Whether to perform cleanup.
  -h, --help help for verify-traffic
      --host string the controller host
  -m, --mode string [optional, default 'both'] The mode to perform: server, client, both.
  -p, --password string password to use for authenticating to the Ziti Edge Controller, if -u is supplied and -p is not, a value will be prompted for
      --port string the controller port
  -x, --prefix string [optional] The prefix to apply to generated objects, necessary when not using the 'both' role.
  -u, --username string username to use for authenticating to the Ziti Edge Controller
      --verbose Show additional output.

```

That will ensure things are working properly and the problem is somewhere outside the overlay. Can you try that, now that you're on a newer ziti build?

You'll see output like this on success:

 ![image](https://global.discourse-cdn.com/free1/uploads/netfoundry/original/2X/7/790798c5f59756d6582567851aa50937c8cf9177.png)

---

<div class="post-metadata">

**Author:** ![woodwardjd](https://avatars.discourse-cdn.com/v4/letter/w/ac91a4/32.png) [@woodwardjd](https://openziti.discourse.group/u/woodwardjd)\
**Post date:** [October 19, 2024, 3:30pm UTC](https://openziti.discourse.group/t/ziti-desktop-edge-controller-network-version-compatibility/3274/10 "2024-10-19T15:30:52Z")

</div>

> [@TheLumberjack](#):
>
> `ziti ops verify-network`

```auto
$ ziti ops verify-network
INFO All requested checks passed.

```

```auto
$ ziti ops verify-traffic --host localhost --port 8441 --username admin
WARNING no prefix and mode [] is not 'both'. default prefix of 2024-10-19-1529 will be used
INFO connecting with user admin to https://localhost:8441
INFO generating P-384 EC key
INFO generating P-384 EC key
INFO waiting 10s for terminator for service: 2024-10-19-1529.verify-traffic
INFO successfully bound service: 2024-10-19-1529.verify-traffic.

INFO Server is listening for a connection and will exit when one is received.
INFO new service session session token=bf053611-935e-4056-9e9f-8f8a3298ebca
INFO found terminator for service: 2024-10-19-1529.verify-traffic
INFO found service named: 2024-10-19-1529.verify-traffic
INFO Server has accepted a connection and will exit soon.
INFO successfully dialed service: 2024-10-19-1529.verify-traffic.
INFO verify-traffic test successfully detected
INFO Server complete. exiting
INFO client complete

```

---

<div class="post-metadata">

**Author:** ![TheLumberjack](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/thelumberjack/32/113_2.png) [@TheLumberjack](https://openziti.discourse.group/u/TheLumberjack)\
**Post date:** [October 19, 2024, 3:36pm UTC](https://openziti.discourse.group/t/ziti-desktop-edge-controller-network-version-compatibility/3274/11 "2024-10-19T15:36:08Z")

</div>

Great, that ensures the network works properly. You made a new identity and it didn't succeed with the Mac client. That's surprised me. Are you open to trying with the ziti-edge-tunnel binary instead, just to rule that out? Are you open to sending me a jet with access to no services which I could try on my side?

---

<div class="post-metadata">

**Author:** ![woodwardjd](https://avatars.discourse-cdn.com/v4/letter/w/ac91a4/32.png) [@woodwardjd](https://openziti.discourse.group/u/woodwardjd)\
**Post date:** [October 19, 2024, 3:36pm UTC](https://openziti.discourse.group/t/ziti-desktop-edge-controller-network-version-compatibility/3274/12 "2024-10-19T15:36:19Z")

</div>

Partner tested w/ macOS 14.6.1 and ZDE 2.44 (522) against current config (1.1.15) yields same behavior (no services) and server-side logs. (using an old identity that hasn't been used in 6 months)

I haven't made any changes to anything ziti-related prior to yesterday (zde) and this morning (server process versions). Certainly nothing related to PKI. Checking timestamps... Yup, just turning over the 1-year-limit certs back in January.

---

<div class="post-metadata">

**Author:** ![woodwardjd](https://avatars.discourse-cdn.com/v4/letter/w/ac91a4/32.png) [@woodwardjd](https://openziti.discourse.group/u/woodwardjd)\
**Post date:** [October 19, 2024, 3:38pm UTC](https://openziti.discourse.group/t/ziti-desktop-edge-controller-network-version-compatibility/3274/13 "2024-10-19T15:38:04Z")

</div>

Yes and yes 🙂 Gonna need the specific guidance tho. (also check the local community slack for private-ish info 🙂

---

<div class="post-metadata">

**Author:** ![TheLumberjack](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/thelumberjack/32/113_2.png) [@TheLumberjack](https://openziti.discourse.group/u/TheLumberjack)\
**Post date:** [October 19, 2024, 3:39pm UTC](https://openziti.discourse.group/t/ziti-desktop-edge-controller-network-version-compatibility/3274/14 "2024-10-19T15:39:19Z")

</div>

Download from [https://github.com/openziti/ziti-tunnel-sdk-c/releases/download/v1.2.3/ziti-edge-tunnel-Darwin\_arm64.zip](https://github.com/openziti/ziti-tunnel-sdk-c/releases/download/v1.2.3/ziti-edge-tunnel-Darwin_arm64.zip)

Make an identity/jwt and try to issue ziti-exte-tunnel enroll -j -o accordingly

---

<div class="post-metadata">

**Author:** ![woodwardjd](https://avatars.discourse-cdn.com/v4/letter/w/ac91a4/32.png) [@woodwardjd](https://openziti.discourse.group/u/woodwardjd)\
**Post date:** [October 19, 2024, 3:51pm UTC](https://openziti.discourse.group/t/ziti-desktop-edge-controller-network-version-compatibility/3274/15 "2024-10-19T15:51:31Z")

</div>

```auto
 sudo ./ziti-edge-tunnel enroll -j the-jwt.jwt -i the-identity.json
(44556)[0.000] INFO ziti-sdk:utils.c:198 ziti_log_set_level() set log level: root=3/INFO
(44556)[0.000] INFO ziti-sdk:utils.c:169 ziti_log_init() Ziti C SDK version 1.1.3 @gf713dc6(HEAD) starting at (2024-10-19T15:49:18.946)
(44556)[0.000] INFO ziti-sdk:ziti_enroll.c:91 ziti_enroll() Ziti C SDK version 1.1.3 @gf713dc6(HEAD) starting enrollment at (2024-10-19T15:49:18.947)
(44556)[0.000] INFO ziti-sdk:ziti_ctrl.c:593 ziti_ctrl_init() ctrl[(null):] using https://the-server.com:8441
(44556)[0.000] INFO ziti-sdk:ziti_ctrl.c:593 ziti_ctrl_init() ctrl[(null):] using https://the-server.com:8441
(44556)[0.000] WARN ziti-sdk:ziti_ctrl.c:180 ctrl_resp_cb() ctrl[the-server.com:8441] request failed: -53(software caused connection abort)
(44556)[0.000] WARN ziti-sdk:ziti_ctrl.c:319 internal_version_cb() ctrl[the-server.com:8441] CONTROLLER_UNAVAILABLE(software caused connection abort)
(44556)[0.000] WARN ziti-sdk:ziti_ctrl.c:180 ctrl_resp_cb() ctrl[the-server.com:8441] request failed: -53(software caused connection abort)
(44556)[0.000] INFO ziti-sdk:ziti_ctrl.c:183 ctrl_resp_cb() ctrl[the-server.com:8441] attempting to switch endpoint
(44556)[0.000] WARN ziti-sdk:ziti_ctrl.c:566 ctrl_next_ep() ctrl[the-server.com:8441] no controllers are online
(44556)[0.000] ERROR ziti-sdk:ziti_enroll.c:263 enroll_cb() failed to enroll with controller: https://the-server.com:8441 CONTROLLER_UNAVAILABLE (software caused connection abort)
(44556)[0.000] ERROR ziti-edge-tunnel:ziti-edge-tunnel.c:2251 enroll_cb() enrollment failed: CONTROLLER_UNAVAILABLE(-3)

```

---

<div class="post-metadata">

**Author:** ![TheLumberjack](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/thelumberjack/32/113_2.png) [@TheLumberjack](https://openziti.discourse.group/u/TheLumberjack)\
**Post date:** [October 19, 2024, 3:57pm UTC](https://openziti.discourse.group/t/ziti-desktop-edge-controller-network-version-compatibility/3274/16 "2024-10-19T15:57:36Z")

</div>

Thanks for trying that and confirming it. Oh it just occurs to me that you probably will need to modify one more thing about your controller. We x moved to openssl and it requires the controller config to change

> **[Upgrade a Quickstart Network | OpenZiti](https://openziti.io/docs/learn/quickstarts/network/help/upgrade-quickstart-network/#upgrading-versions-v0282)**
>
> This document explains the process for backing up, and upgrading your controller and router(s).

See if that resolves the enrollment issue. The ca bundle needs to contain only root cas in these clients. I think that's what's happening.

I'll try to dig up the command that recreates the bundle in a few

---

<div class="post-metadata">

**Author:** ![woodwardjd](https://avatars.discourse-cdn.com/v4/letter/w/ac91a4/32.png) [@woodwardjd](https://openziti.discourse.group/u/woodwardjd)\
**Post date:** [October 19, 2024, 4:06pm UTC](https://openziti.discourse.group/t/ziti-desktop-edge-controller-network-version-compatibility/3274/17 "2024-10-19T16:06:55Z")

</div>

I did the referenced `ziti-controller run` =\> `zit controller run` update to systemd config earlier today, controller is running 1.1.15. I don't see anything referenced in there about config of the controller (say, in the yaml), only how to start it.

---

<div class="post-metadata">

**Author:** ![woodwardjd](https://avatars.discourse-cdn.com/v4/letter/w/ac91a4/32.png) [@woodwardjd](https://openziti.discourse.group/u/woodwardjd)\
**Post date:** [October 19, 2024, 4:07pm UTC](https://openziti.discourse.group/t/ziti-desktop-edge-controller-network-version-compatibility/3274/18 "2024-10-19T16:07:47Z")

</div>

(also, thanks for the pointer to the upgrade docs. I couldn't for the life of me find them through google or an in-page search (command-f). Turns out it is very findable through the search box in the upper right, and I didn't notice that).

---

<div class="post-metadata">

**Author:** ![woodwardjd](https://avatars.discourse-cdn.com/v4/letter/w/ac91a4/32.png) [@woodwardjd](https://openziti.discourse.group/u/woodwardjd)\
**Post date:** [October 26, 2024, 10:55am UTC](https://openziti.discourse.group/t/ziti-desktop-edge-controller-network-version-compatibility/3274/19 "2024-10-26T10:55:39Z")

</div>

When he gets time I'm sure @TheLumberjack will chime in here with more details, but this ended up being an incompatibility with the certificate bundles, certificate chaining, and contents of the identity files on clients created by enrollment. Backing out to a version `ziti-edge-tunnel` prior to these changes in the C SDK (v1.1.3 in the macOS case) got me back on my feet until we could re-bundle certificates server-side and re-enroll new identities (to get the right certs into the identities stored on the client). Since Ziti Desktop Edge is only available in the Mac App Store, I couldn't revert to a previous version of that. Much thanks to Clint's help!
