Running the latest April 15th Ziti binaries (1.6) HA with 2 controllers at the moment. Client is ZDEW latest 2.6.5 and I am using ext-jwt-auth to an external provider (https://oidc-test.example.internal)
I am observing 404s in my logs during token refresh. Seemingly not usually causing traffic loss during every 404, however I am indeed troubleshooting some intermittent connectivity issues and I am not sure if these 404s are an issue, or if they are a red herring and this is expected. Any insight is appreciated..
Example snippet:
[2025-04-17T17:29:23.800Z] DEBUG ziti-sdk:oidc.c:927 oidc_client_set_tokens() using access_token={"aud":["3ecb9d18-d4a7-4697-a1f8-4e308cb6a1ed"],"exp":1744914562,"iat":1744910962,"iss":"https://oidc-test.example.internal","sub":"5e27f747-73e8-4bde-a0a8-58f954c027cf","type":"oauth-access-token"}
[2025-04-17T17:29:23.800Z] DEBUG ziti-sdk:external_auth.c:94 ext_token_cb() received access token: eyJhbGciOiJSUzI1NiIs...
[2025-04-17T17:29:23.800Z] DEBUG ziti-sdk:oidc.c:938 oidc_client_set_tokens() scheduling token refresh in 3600 seconds
[2025-04-17T17:29:23.853Z] ERROR ziti-sdk:oidc.c:198 parse_cb() unexpected content-type[.well-known/openid-configuration]: text/plain; charset=utf-8
[2025-04-17T17:29:23.853Z] ERROR ziti-sdk:ha_auth.c:145 config_cb() failed to configure OIDC[https://ctrl1.testing.internal:443/oidc] client: -4071/(null)
[2025-04-17T17:29:23.853Z] WARN ziti-sdk:oidc.c:171 handle_unexpected_resp() unexpected OIDC response
[2025-04-17T17:29:23.853Z] WARN ziti-sdk:oidc.c:172 handle_unexpected_resp() 1.1 404 Not Found
[2025-04-17T17:29:23.853Z] WARN ziti-sdk:oidc.c:175 handle_unexpected_resp() transfer-encoding: chunked
[2025-04-17T17:29:23.853Z] WARN ziti-sdk:oidc.c:175 handle_unexpected_resp() Date: Thu, 17 Apr 2025 17:29:23 GMT
[2025-04-17T17:29:23.853Z] WARN ziti-sdk:oidc.c:175 handle_unexpected_resp() X-Content-Type-Options: nosniff
[2025-04-17T17:29:23.853Z] WARN ziti-sdk:oidc.c:175 handle_unexpected_resp() Content-Type: text/plain; charset=utf-8
[2025-04-17T17:29:23.853Z] WARN ziti-sdk:oidc.c:175 handle_unexpected_resp() Content-Encoding: gzip
[2025-04-17T17:29:23.853Z] WARN ziti-sdk:oidc.c:164 unhandled_body_cb() 404 page not found
[2025-04-17T17:29:23.853Z] WARN ziti-sdk:oidc.c:166 unhandled_body_cb() status = -4095
Complete logs:
[2025-04-17T17:29:19.995Z] DEBUG ziti-sdk:oidc.c:973 refresh_time_cb() refreshing OIDC token
[2025-04-17T17:29:20.070Z] DEBUG ziti-sdk:oidc.c:946 refresh_cb() token refresh success
[2025-04-17T17:29:20.070Z] DEBUG ziti-sdk:oidc.c:927 oidc_client_set_tokens() using access_token={"aud":["openziti"],"exp":1744912759,"iat":1744910959,"iss":"https://ctrl1.testing.internal:443/oidc","jti":"50d2c5a8-c316-440e-b0c2-54c7bde76a91","nbf":1744910959,"sub":"PJ27lxXJGe","z_aid":"openziti","z_asid":"47df4413-fb08-42ac-9b91-08703f813587","z_authid":"extJwtId:1ukzvj1YizqOzgTZfYEHNw","z_cfs":null,"z_eid":"5e27f747-73e8-4bde-a0a8-58f954c027cf","z_env":{},"z_ice":false,"z_ra":"208.118.129.23:23529","z_sdk":{},"z_t":"a"}
[2025-04-17T17:29:20.070Z] DEBUG ziti-sdk:ziti.c:379 ziti_set_fully_authenticated() ztx[0] setting auth_state[3] to 3
[2025-04-17T17:29:20.070Z] DEBUG ziti-sdk:ziti_ctrl.c:1090 ctrl_paging_req() ctrl[https://ctrl1.testing.internal:443/edge/client/v1] starting paging request GET[/controllers]
[2025-04-17T17:29:20.070Z] DEBUG ziti-sdk:ziti_ctrl.c:1090 ctrl_paging_req() ctrl[https://ctrl1.testing.internal:443/edge/client/v1] starting paging request GET[/controllers]
[2025-04-17T17:29:20.070Z] DEBUG ziti-sdk:channel.c:328 ziti_channel_update_token() ch[3] sending token update
[2025-04-17T17:29:20.070Z] DEBUG ziti-sdk:channel.c:328 ziti_channel_update_token() ch[2] sending token update
[2025-04-17T17:29:20.070Z] DEBUG ziti-sdk:channel.c:328 ziti_channel_update_token() ch[1] sending token update
[2025-04-17T17:29:20.070Z] DEBUG ziti-sdk:channel.c:328 ziti_channel_update_token() ch[0] sending token update
[2025-04-17T17:29:20.070Z] DEBUG ziti-sdk:oidc.c:938 oidc_client_set_tokens() scheduling token refresh in 1799 seconds
[2025-04-17T17:29:20.070Z] DEBUG ziti-sdk:ziti_ctrl.c:1090 ctrl_paging_req() ctrl[https://ctrl1.testing.internal:443/edge/client/v1] starting paging request GET[/current-identity/edge-routers]
[2025-04-17T17:29:20.070Z] DEBUG ziti-sdk:ziti_ctrl.c:1090 ctrl_paging_req() ctrl[https://ctrl1.testing.internal:443/edge/client/v1] starting paging request GET[/controllers]
[2025-04-17T17:29:20.071Z] DEBUG ziti-sdk:posture.c:210 ziti_send_posture_data() ztx[0] posture checks must_send set to TRUE, new_session_id[TRUE], must_send_every_time[TRUE], new_controller_instance[FALSE]
[2025-04-17T17:29:20.081Z] DEBUG ziti-sdk:channel.c:307 token_update_cb() ch[2] token update success
[2025-04-17T17:29:20.085Z] DEBUG ziti-sdk:channel.c:307 token_update_cb() ch[3] token update success
[2025-04-17T17:29:20.085Z] DEBUG ziti-sdk:channel.c:307 token_update_cb() ch[1] token update success
[2025-04-17T17:29:20.128Z] DEBUG ziti-sdk:ziti_ctrl.c:485 ctrl_body_cb() ctrl[https://ctrl1.testing.internal:443/edge/client/v1] received 2/2 for paging request GET[/controllers]
[2025-04-17T17:29:20.128Z] DEBUG ziti-sdk:ziti_ctrl.c:495 ctrl_body_cb() ctrl[https://ctrl1.testing.internal:443/edge/client/v1] completed paging request GET[/controllers] in 0.058 s
[2025-04-17T17:29:20.145Z] DEBUG ziti-sdk:ziti_ctrl.c:485 ctrl_body_cb() ctrl[https://ctrl1.testing.internal:443/edge/client/v1] received 2/2 for paging request GET[/controllers]
[2025-04-17T17:29:20.145Z] DEBUG ziti-sdk:ziti_ctrl.c:495 ctrl_body_cb() ctrl[https://ctrl1.testing.internal:443/edge/client/v1] completed paging request GET[/controllers] in 0.074 s
[2025-04-17T17:29:20.145Z] INFO ziti-sdk:ziti.c:349 ctrl_list_cb() ztx[0] controller[ctrl1/ctrl1] url[https://ctrl1.testing.internal:443/edge/client/v1]
[2025-04-17T17:29:20.145Z] INFO ziti-sdk:ziti.c:349 ctrl_list_cb() ztx[0] controller[ctrl3/ctrl3] url[https://ctrl3.testing.internal:443/edge/client/v1]
[2025-04-17T17:29:20.149Z] DEBUG ziti-sdk:channel.c:307 token_update_cb() ch[0] token update success
[2025-04-17T17:29:20.177Z] DEBUG ziti-sdk:ziti_ctrl.c:500 ctrl_body_cb() ctrl[https://ctrl1.testing.internal:443/edge/client/v1] completed GET[/current-identity] in 0.106 s
[2025-04-17T17:29:20.193Z] DEBUG ziti-sdk:ziti_ctrl.c:500 ctrl_body_cb() ctrl[https://ctrl1.testing.internal:443/edge/client/v1] completed GET[/current-identity] in 0.122 s
[2025-04-17T17:29:20.209Z] DEBUG ziti-sdk:ziti_ctrl.c:485 ctrl_body_cb() ctrl[https://ctrl1.testing.internal:443/edge/client/v1] received 4/4 for paging request GET[/current-identity/edge-routers]
[2025-04-17T17:29:20.209Z] DEBUG ziti-sdk:ziti_ctrl.c:495 ctrl_body_cb() ctrl[https://ctrl1.testing.internal:443/edge/client/v1] completed paging request GET[/current-identity/edge-routers] in 0.139 s
[2025-04-17T17:29:20.226Z] DEBUG ziti-sdk:ziti_ctrl.c:485 ctrl_body_cb() ctrl[https://ctrl1.testing.internal:443/edge/client/v1] received 2/2 for paging request GET[/controllers]
[2025-04-17T17:29:20.226Z] DEBUG ziti-sdk:ziti_ctrl.c:495 ctrl_body_cb() ctrl[https://ctrl1.testing.internal:443/edge/client/v1] completed paging request GET[/controllers] in 0.156 s
[2025-04-17T17:29:20.242Z] DEBUG ziti-sdk:ziti_ctrl.c:500 ctrl_body_cb() ctrl[https://ctrl1.testing.internal:443/edge/client/v1] completed GET[/current-api-session/service-updates] in 0.172 s
[2025-04-17T17:29:20.242Z] DEBUG ziti-sdk:ziti_ctrl.c:1090 ctrl_paging_req() ctrl[https://ctrl1.testing.internal:443/edge/client/v1] starting paging request GET[/services?configTypes=all]
[2025-04-17T17:29:20.257Z] DEBUG ziti-sdk:ziti_ctrl.c:485 ctrl_body_cb() ctrl[https://ctrl1.testing.internal:443/edge/client/v1] received 1/1 for paging request GET[/services?configTypes=all]
[2025-04-17T17:29:20.257Z] DEBUG ziti-sdk:ziti_ctrl.c:495 ctrl_body_cb() ctrl[https://ctrl1.testing.internal:443/edge/client/v1] completed paging request GET[/services?configTypes=all] in 0.015 s
[2025-04-17T17:29:23.417Z] DEBUG ziti-edge-tunnel:ipc_cmd.c:264 on_cmd() received cmd <{"Command":"Status"}
>
[2025-04-17T17:29:23.418Z] DEBUG ziti-edge-tunnel:ipc_cmd.c:114 on_cmd_write() IPC write complete
[2025-04-17T17:29:23.694Z] DEBUG ziti-sdk:oidc.c:973 refresh_time_cb() refreshing OIDC token
[2025-04-17T17:29:23.800Z] DEBUG ziti-sdk:oidc.c:946 refresh_cb() token refresh success
[2025-04-17T17:29:23.800Z] DEBUG ziti-sdk:oidc.c:927 oidc_client_set_tokens() using access_token={"aud":["3ecb9d18-d4a7-4697-a1f8-4e308cb6a1ed"],"exp":1744914562,"iat":1744910962,"iss":"https://oidc-test.example.internal","sub":"5e27f747-73e8-4bde-a0a8-58f954c027cf","type":"oauth-access-token"}
[2025-04-17T17:29:23.800Z] DEBUG ziti-sdk:external_auth.c:94 ext_token_cb() received access token: eyJhbGciOiJSUzI1NiIs...
[2025-04-17T17:29:23.800Z] DEBUG ziti-sdk:oidc.c:938 oidc_client_set_tokens() scheduling token refresh in 3600 seconds
[2025-04-17T17:29:23.853Z] ERROR ziti-sdk:oidc.c:198 parse_cb() unexpected content-type[.well-known/openid-configuration]: text/plain; charset=utf-8
[2025-04-17T17:29:23.853Z] ERROR ziti-sdk:ha_auth.c:145 config_cb() failed to configure OIDC[https://ctrl1.testing.internal:443/oidc] client: -4071/(null)
[2025-04-17T17:29:23.853Z] WARN ziti-sdk:oidc.c:171 handle_unexpected_resp() unexpected OIDC response
[2025-04-17T17:29:23.853Z] WARN ziti-sdk:oidc.c:172 handle_unexpected_resp() 1.1 404 Not Found
[2025-04-17T17:29:23.853Z] WARN ziti-sdk:oidc.c:175 handle_unexpected_resp() transfer-encoding: chunked
[2025-04-17T17:29:23.853Z] WARN ziti-sdk:oidc.c:175 handle_unexpected_resp() Date: Thu, 17 Apr 2025 17:29:23 GMT
[2025-04-17T17:29:23.853Z] WARN ziti-sdk:oidc.c:175 handle_unexpected_resp() X-Content-Type-Options: nosniff
[2025-04-17T17:29:23.853Z] WARN ziti-sdk:oidc.c:175 handle_unexpected_resp() Content-Type: text/plain; charset=utf-8
[2025-04-17T17:29:23.853Z] WARN ziti-sdk:oidc.c:175 handle_unexpected_resp() Content-Encoding: gzip
[2025-04-17T17:29:23.853Z] WARN ziti-sdk:oidc.c:164 unhandled_body_cb() 404 page not found
[2025-04-17T17:29:23.853Z] WARN ziti-sdk:oidc.c:166 unhandled_body_cb() status = -4095
[2025-04-17T17:29:28.422Z] DEBUG ziti-edge-tunnel:ipc_cmd.c:264 on_cmd() received cmd <{"Command":"Status"}
>
[2025-04-17T17:29:28.422Z] DEBUG ziti-edge-tunnel:ipc_cmd.c:114 on_cmd_write() IPC write complete