Reliable way to use open ziti clients

Ziti Overlay
1

Hi everyone,

We started deploying OpenZiti agents for both macOS and Linux users a week ago, after deploying OpenZiti across most of our infrastructure.

Unfortunately, there have been many inconsistencies and issues with the tunnelers over the past week. Users are experiencing problems, and as a result, we are receiving negative feedback. Your swift assistance would be appreciated.

The two most frequent scenarios are:

  1. The client gets disconnected but shows as connected in the status.

  2. Users unknowingly open multiple sessions and receive an "invalid session" error.

It’s difficult to pinpoint exactly when and why this occurs. However, it seems more frequent when the network is switched off/on or when the laptop is put to sleep while the client is still connected.

Steps to reproduce (not always consistent):

  • Enable/run the client.

  • Log in the user (we're using OIDC).

  • Turn off the internet or put the laptop to sleep for an extended period (e.g., lunch break).

  • Try to use Ziti → fails.

  • Re-login → does not help.

First issue diagnosis on Linux machines:

  • Running ziti-edge-tunnel tunnel_status shows:

    • Loaded: True

    • Active: True

    • NeedsExtAuth: False

    • The connected identity

    • All attached services

  • Running ifconfig shows the Ziti interface.

  • DNS resolution works.

  • Logs typically end with a warning that the client disconnected from all edge routers.

  • Running any ziti-edge-tunnel commands after this does not produce any log entries.

Current workaround: Restart the service and re-authenticate.

First issue diagnosis on macOS machines:

  • Edge is connected and the identity is enabled, but the status icon shows a yellow circle.

  • Re-authentication opens the browser, succeeds, the circle turns green, and services begin loading.

  • appex.log shows a warning: "connection failed: failed to authenticate".

Current workaround: Uncheck "Enable", disconnect the tunnel, quit the program, and restart everything.

Second issue diagnosis (Linux and macOS):

  • Everything appears connected and operational.

  • The client attempts to connect using ziti network but fails.

  • Logs show: "failed to connect, reason=invalid session".

  • Re-authenticating opens a new session, but the same error appears (even after killing all services and restarting).

(In the scrrenshot, the same identitiy opened 4 different sessions)

image
image1506×725 86 KB

Current workaround: An admin manually kills all sessions associated with the identity on the controller, then re-authenticate.

Questions:

  • Are we using the clients incorrectly?

  • Could this be related to OIDC?

  • Since most users are technical but not deeply experienced, is there a better way to reflect the current status or force them to handle these issues more effectively?

Thanks!

2

Hi @NullZiti, the first thing that sticks out to me is this: "Turn off the internet or put the laptop to sleep for an extended period (e.g., lunch break).".

The default session that OpenZiti uses is 30 minutes. If you are offline for more than 30 minutes then you'll need to 'refresh' your session with OpenZiti. When you use external jwt signers, here's what should happen based on my understanding (starting from initial login)

  • 0h:00m - user logs in using ext-jwt-signer. (This process will use the token from the IdP and exchange it for tokens an OpenZiti controller, these are the tokens the clients use going forward when authenticating to OpenZiti)
  • 0h:15m - user has been using OpenZiti fine, decides to go to lunch, puts laptop to sleep (stopping all processing/network activity)
  • 1h:00m - 45user comes back from lunch

Here, everything below should be 'pretty instant':

  • OpenZiti client notices it has been disconnected from the overlay due to session timeout and tries to reauthenticate
  • OpenZiti client notices the duration of disconnect is greater than the OZ refresh token assigned by the OpenZiti controller and tries to auth with the ext-jwt-signer
  • OpenZiti client looks at ext-jwt-signer tokens, if there a refresh token, OpenZiti client uses ext-jwt-signer refresh token to obtain a new ext-jwt-signer security token
  • OpenZiti client submits new ext-jwt-signer security token from ext-jwt-signer IdP and attempts to exchange it for OZ tokens from OpenZiti controller
  • User is now authenticated and can continue about their day

So my expectation here is that somehow that refresh is not succeeding properly. You need to start with a set of logs from a client that had 'the problem'. See if there are any helpful errors in there (hopefully there are) and share them here. If not, look at the contoller and IdP and look for any interesting logs that would help us help you.

Hopefully this helps