Ziti Desktop App v1.9.2 incompatible with Controller v1.1.15 – Windows clients crashing after auto-upgrade

We are currently running an OpenZiti controller version 1.1.15 in production.

A few external clients (hospitals) and internal users use the Ziti Desktop App on both Windows and macOS.

Starting this morning, all Windows Ziti Desktop Apps began crashing. Upon investigation, we found that:

  • All Windows clients were automatically upgraded to Ziti Desktop App latest version. Older version 2.7.4.0 was working fine.

  • macOS clients remain on v1.7.2 and are working fine

  • Our controller is still on v1.1.15

We suspect that the Windows Desktop App v1.9.2 (SDK / API) is incompatible with controller v1.1.15, which is causing the crashes.

This issue is blocking our customers in production, while macOS users continue to work without issues.

We do plan to upgrade our controller to a newer version, but this unexpected client auto-upgrade is currently a blocker.

Questions / Request:

  1. Is ziti-tunneler v1.9.0 compatible not with controller v1.1.15?

  2. If not, is there a known workaround (e.g., disabling auto-upgrade, forcing client downgrade, or compatibility fix)?

  3. Is it possible to make v1.9.0 backward-compatible with older controllers, or recommend a minimum controller version?

  4. For now i m disabling auto upgrade and uninstall and reinstall older version 2.7.4.0

Please let us know if logs, crash dumps, or additional details are required.

Thank you for your support.

Hello, I’m sorry the automatic update has caused your Windows users so much trouble. I can say for certain that this is not an intentional incompatibility.

We do have some fixes in the Windows Desktop Edge beta release stream (currently v2.8.4.0) that might fix your crash. If it’s easy/safe for you to do, could you try the beta stream? Specifically earlier versions of 1.9.x of ziti-sdk-c would crash when it tried to authenticate with some controllers. This issue was fixed in ziti-sdk-c 1.9.19, which is included in ziti-edge-tunnel v1.9.8 / Ziti Desktop Edge v2.8.2.0 and later.

If the current beta does not help, please send logs (TRACE-level if possible) from one of the crashed tunnelers to help us get a handle on exactly what you’re running into.

just to add on +1 here, i am on controller 1.2.2 and automated upgrade also caused all >160 identities now crashed .

After spending hours , I fall back to client 2.5.5.0 to ensure this work. this upgrade had caused major outage in my environment.

addon: this also applied to android SDK.

@scareything I think it is not possible ask our customers to install beta version, i ll ask them to roll back to working version and disable auto upgrade. but i could help on testing with beta version and see if that works

btw how can we do rollback on mac pc’s? if we have installed via app store?

I’m hoping that you or one of your users could try the beta to see if it addresses this problem. I think there’s a decent chance that it will, and if so I’d like to promote that beta to the stable release so your users will get.

In the absence of trying the beta, I’d need to see logs from a tunneler that crashed so I can have a chance at diagnosing the issue.

I’m not aware of the Apple app store letting you select previous versions of an app. I think the general solution on macOS is to have recent backups ready. If there’s an issue you can restore from the backup and report the issue to us, hopefully with logs so we can figure out what happened.

I have tested deploying 2.8.2.0 and 2.8.4.0 both has started , but services are not listing it is failing to authenticate, it looks similar issue ive raised for chromebook. I recent tunneler version has blocked all installer, like windows, mac app store, chromebook androids apps. I guess its all same solution i guess.

ziti-logs-windows.txt (3.0 KB)

Similar issue on chromebook. i think all needs same solution to support 1.1.15 controller on enrollment and authentication.

+1. 2.7.4.0 works. Latest stable release and pre-release 2.8.4.0 & 2.8.5.0 - auth error. I am not sending the logs because they seem to have already been sent here earlier. If you need anything from my side, please let me know.

Thanks for the log segment. I really can’t make out much from it though because it’s only a few lines. I really need to see the entire log from when the process starts if possible, and it would really help if they were at an increased level to show more information.

The log level can be changed from the UI main menu via “Advanced Settings” → “Set Logging Level” → “Trace”

After setting the log level, please restart OpenZiti with the “Stop” and “Start” button in the UI.

Once when the authentication failures have occurred, please go back to the UI main menu top level and select “Feedback”. This will produce a ZIP file with the logs and some other information and open an Explorer window showing the location of the ZIP. You can email this ZIP file to me at “scarey@netfoundry.io”.

Thanks

Hello,

If you have TRACE logs showing this issue they would certainly be helpful. Which version of the controller are you using?

Thanks

Hello.
Controller version: v1.5.4
Em.... I did send you logs by email (too big of a file, and I'm too lazy to remove all tokens and URLs for public access (almost 300k log lines for a few seconds of app running)).

Hi,

Thanks @wusikijeronii I got your email. Something isn’t adding up though, because:

  1. The controller you’re connecting to is version 1.2.0:
INFO ziti-sdk:ziti.c:1698 version_cb() ztx[0] connected to controller https://*****:/edge/client/v1 version v1.2.0(e9d6d837a579 2024-11-07T20:29:14Z)
  1. The logs that you sent are not from a tunneler. Apparently it’s your own app (which is awesome!), but the ziti-sdk it’s using is very old:
INFO ziti-sdk:utils.c:170 ziti_log_init() Ziti C SDK version 1.0.11 @g92c7d63(HEAD) starting at (2024-12-10T23:02:53.027) 

Also the log shows that your connections are getting initiated:

[        1.978] VERBOSE ziti-sdk:connect.c:105 conn_set_state() conn[0.2/gnn_xr_j/Connecting] transitioning Connecting => Connected

And most seem to be working without issue, but some connections are failing after stating successfully:

[        1.986]    WARN ziti-sdk:conn_bridge.c:284 on_ziti_data() br[0.2] write failed: -32(broken pipe)

If this is the issue that you’re here for, it’s different from the issue that this thread was started for. Please feel free to start a new thread if that’s the case.

I may have been mistaken about the controller version. It's been a while since I configured Ziti, and I may have forgotten some commands. What I wrote is what the ziti controller version command returned.

No, this is the official ZDEW app v2.8.5.0 obtained from GitHub Releases.

Are you sure? From my point of view, it's the same problem. After updating the app, everything stopped working. And rolling back to the previous version of the app fixes the problem.

Ok, I’m pretty sure that the auth failures with older controllers are now fixed. Can you please try ZDEW 2.8.6.0?

My apologies, @wusikijeronii. I had a an existing “log.txt” file in my downloads folder, and when I downloaded yours I didn’t notice that it had been renamed to “log-1.txt”. If you could try 2.8.6.0 and let me know, that would be great.

Thanks!

Unfortunately, the problem remains. I have sent you the new logs by email.

@wusikijeronii thanks for the update. I looked at your most recent tunneler log and I do think the issue you’re seeing is technically different than the one that was fixed with ZDEW 2.8.6.0, although admittedly the primary symptom is similar.

The log shows that you’re connecting to controller version 1.6.1, which is marked as pre-release and there have been several authentication/oidc fixes in subsequent versions. Could you please update your controller to latest (currently 1.6.12)? If you still see the issue we can pick it up here in a new thread.

Thanks!

I can update the controller without any problems, and I think it will help solve the problem, but are you sure about that? Rolling back the controller is quite difficult due to migration versions. What I mean is that if compatibility with other controllers is important to you, then I am the best platform for testing. Are you sure I should update the controller?

Thanks for the note. If 1.6.1 were marked as stable then yes, we would be more concerned with client compatibility and issues that we may not have seen anywhere else. In this case though I think there’s a good chance that we’ve already seen and fixed this issue, but if it does still exist in the stable release then we’ll want to fix it relative to that codebase revision and not 1.6.1.

If you are concerned about rolling back, you can take a snapshot of your database with ziti edge db snapshot that you can put back in place after restoring the previous controller binary. A few more details are here.

Thanks

Recent beta vesrion 2.8.6.0 works with old controller for me on windows. Please Release an official version and i need this same fix for android and ,mac version too.