List controller errors, refresh services errors after updating ziti sdk versions

ZITI_IMAGE=openziti/quickstart
ZITI_VERSION=1.6.6

my go mod:

	github.com/openziti/channel/v4 v4.2.21 // indirect
	github.com/openziti/edge-api v0.26.47 // indirect
	github.com/openziti/foundation/v2 v2.0.70 // indirect
	github.com/openziti/identity v1.0.109 // indirect
	github.com/openziti/metrics v1.4.2 // indirect
	github.com/openziti/sdk-golang v1.2.3 // indirect
	github.com/openziti/secretstream v0.1.38 // indirect
	github.com/openziti/transport/v2 v2.0.183 // indirect
	github.com/openziti/ziti v1.6.7 // indirect

My code:

package openziti

import (
	"context"
	"log"
	"net"
	"net/http"
	"strings"
	"time"

	"github.com/gorilla/websocket"
	"github.com/openziti/sdk-golang/ziti"
)

var ZitiContext ziti.Context
var ZitiContexts *ziti.CtxCollection
var ZitiTransport *http.Transport
var ZitiClient *http.Client
var ZitiCustomDialer CustomDialer

// CustomDialer wraps a ziti.Context
// Needed when can't use ziti transport(ex. nats)
type CustomDialer struct {
	ZitiContext ziti.Context
}

func (cd CustomDialer) Dial(network, address string) (net.Conn, error) {
	addr := strings.Split(address, ":")
	return cd.ZitiContext.Dial(addr[0])
}

type FallbackDialer struct {
	UnderlayDialer *net.Dialer
}

func (z FallbackDialer) Dial(network, address string) (net.Conn, error) {
	return z.UnderlayDialer.Dial(network, address)
}

// Enrolls api identity if needed
// Creates openziti api session
// Creates a keepalive goroutine which pings openziti api every X minutes(default 10)
func SetupOpenziti(ctrlUrl string, zitiIDPath string) error {
	err := EnrollIfNeeded(zitiIDPath)
	if err != nil {
		log.Print(err)
		return err
	}
	log.Print(1111111)
	err = CreateApiSession(ctrlUrl, zitiIDPath)
	if err != nil {
		log.Print(err)
		return err
	}
	log.Print(22222)
	err = InitCon(zitiIDPath)
	if err != nil {
		log.Print(err)
		return err
	}
	log.Print(33333)

	go KeepAlive(ctrlUrl)
	return nil
}

func InitCon(zitiIDPath string) error {
	log.Print(00000000000000)
	err := SetupZitiContext(zitiIDPath)
	if err != nil {
		log.Print(err)
		return err
	}
	log.Print(010101010)

	err = SetupZitiTransport()
	if err != nil {
		log.Print(err)
		return err
	}
	log.Print(90909090)

	ZitiClient = &http.Client{
		Transport: ZitiTransport,
		Timeout:   30 * time.Second,
	}
	ZitiCustomDialer = CustomDialer{ZitiContext: ZitiContext}

	// Set ziti transport for websocket
	websocket.ZitiTransport = ZitiTransport
	return nil
}

func SetupZitiContext(path string) (err error) {
	identityFile := path + ".json"
	log.Print(identityFile)

	cfg, err := ziti.NewConfigFromFile(identityFile)
	if err != nil {
		log.Print(err)
		return err
	}
	cfg.ConfigTypes = append(cfg.ConfigTypes, "all")

	ZitiContext, err = ziti.NewContext(cfg)
	if err != nil {
		log.Print(err)
		return err
	}
	log.Print("refresh services gliches?")
	return ZitiContext.RefreshServices()
}

func SetupZitiTransport() error {
	ZitiContexts = ziti.NewSdkCollection()
	ZitiContexts.Add(ZitiContext)
	fallback := &FallbackDialer{
		UnderlayDialer: &net.Dialer{},
	}

	ZitiTransport = http.DefaultTransport.(*http.Transport).Clone() // copy default transport
	ZitiTransport.DialContext = func(ctx context.Context, network, addr string) (net.Conn, error) {
		dialer := ZitiContexts.NewDialerWithFallback(ctx, fallback)
		return dialer.Dial(network, addr)
	}
	ZitiTransport.Dial = func(network, addr string) (net.Conn, error) {
		dialer := ZitiContexts.NewDialerWithFallback(context.Background(), fallback)
		return dialer.Dial(network, addr)
	}
	return nil
}

After updating to newest ziti sdk, my code fails at return ZitiContext.RefreshServices()
also there is a new error of listing controllers. The identity used has admin rights.

Heres the error log:

api-server-1  | screen-control-lib/openziti/setup.go has changed
api-server-1  | building...
api-server-1  | running...
api-server-1  | 2025/08/14 08:22:31 setup.go:49: 1111111
api-server-1  | 2025/08/14 08:22:31 auth.go:100: Ziti session token: 320e65de-0ea4-43a3-be0c-22aaac4bd672
api-server-1  | 2025/08/14 08:22:31 setup.go:55: 22222
api-server-1  | 2025/08/14 08:22:31 setup.go:68: 0
api-server-1  | 2025/08/14 08:22:31 setup.go:96: /api-ziti/api.json
api-server-1  | 2025/08/14 08:22:32 setup.go:110: refresh services gliches?
api-server-1  | ERRO[0000] error listing controllers, continuing with 1 default configured controller  error="[GET /controllers][401] listControllersUnauthorized  &{Error:0xc0007d60c0 Meta:0xc000620100}"
api-server-1  | 2025/08/14 08:22:32 setup.go:71: failed to refresh services: no apiSession, authentication attempt failed: error for request Gg4jdbUKGL: UNAUTHORIZED: The request could not be completed. The session is not authorized or the credentials are invalid, caused by: error for request : UNHANDLED: UNAUTHORIZED: The request could not be completed. The session is not authorized or the credentials are invalid
api-server-1  | 2025/08/14 08:22:32 setup.go:58: failed to refresh services: no apiSession, authentication attempt failed: error for request Gg4jdbUKGL: UNAUTHORIZED: The request could not be completed. The session is not authorized or the credentials are invalid, caused by: error for request : UNHANDLED: UNAUTHORIZED: The request could not be completed. The session is not authorized or the credentials are invalid
api-server-1  | 2025/08/14 08:22:32 main.go:50: failed to refresh services: no apiSession, authentication attempt failed: error for request Gg4jdbUKGL: UNAUTHORIZED: The request could not be completed. The session is not authorized or the credentials are invalid, caused by: error for request : UNHANDLED: UNAUTHORIZED: The request could not be completed. The session is not authorized or the credentials are invalid

Heres policy-advisor for api identity:

ziti@71fb88f17ca8:/persistent$ ziti edge policy-advisor identities api

Policy General Guidelines
  In order for an identity to dial or bind a service, the following must be true:
    - The identity must have access to the service via a service policy of the correct type (dial or bind)
    - The identity must have access to at least one on-line edge router via an edge router policy
    - The service must have access to at least one on-line edge router via a service edge router policy
    - There must be at least one on-line edge router that both the identity and service have access to.

Policy Advisor Output Guide:
  STATUS = The status of the identity -> service reachability. Will be OKAY or ERROR. 
  ID = identity name
  ID ROUTERS = number of routers accessible to the identity via edge router policies.
    - See edge router polices for an identity: ziti edge controller list identity edge-router-policies <identity>
  SVC = service name
  SVC ROUTERS = number of routers accessible to the service via service edge router policies.
    - See service edge router policies for a service with: ziti edge controller list service service-edge-router-policies <service>
  ONLINE COMMON ROUTERS = number of routers the identity and service have in common which are online.
  COMMON ROUTERS = number of routers (online or offline) the identity and service have in common.
  DIAL_OK = indicates if the identity has permission to dial the service.
    - See service polices for a service  : ziti edge controller list service service-policies <service>
    - See service polices for an identity: ziti edge controller list identity service-policies <identity>
  BIND_OK = indicates if the identity has permission to bind the service.
  ERROR_LIST = if the status is ERROR, error details will be listed on the following lines

Output format: STATUS: ID (ID ROUTERS) -> SVC (SVC ROUTERS) Common Routers: (ONLINE COMMON ROUTERS/COMMON ROUTERS) Dial: DIAL_OK Bind: BIND_OK. ERROR_LIST
-------------------------------------------------------------------------------
OKAY : api (1) -> nats.my.domain (1) Common Routers: (1/1) Dial: Y Bind: N 

OKAY : api (1) -> livekit.my.domain (1) Common Routers: (1/1) Dial: Y Bind: N 

ziti@71fb88f17ca8:/persistent$ 

and heres logs if i get rid of refresh services:

api-server-1  | ERRO[0092] error listing controllers, continuing with 1 default configured controller  error="[GET /controllers][401] listControllersUnauthorized  &{Error:0xc0007908a0 Meta:0xc0008719c0}"
api-server-1  | 2025/08/14 08:38:54 setup.go:110: refresh services gliches?2025/08/14 08:38:54 conn.go:40: Failed to connect to NATS: failed to dial: no apiSession, authentication attempt failed: error for request q-hy8b0KlL: UNAUTHORIZED: The request could not be completed. The session is not authorized or the credentials are invalid, caused by: error for request : UNHANDLED: UNAUTHORIZED: The request could not be completed. The session is not authorized or the credentials are invalid
api-server-1  | 2025/08/14 08:38:54 main.go:59: failed to dial: no apiSession, authentication attempt failed: error for request q-hy8b0KlL: UNAUTHORIZED: The request could not be completed. The session is not authorized or the credentials are invalid, caused by: error for request : UNHANDLED: UNAUTHORIZED: The request could not be completed. The session is not authorized or the credentials are invalid

I created a “hello world“ test app and i still get the same errors:

package main

import (
	"log"

	"github.com/openziti/sdk-golang/ziti"
)

func main() {
	cfg, err := ziti.NewConfigFromFile("api.json")
	if err != nil {
		log.Fatal(err)
	}

	context, err := ziti.NewContext(cfg)
	if err != nil {
		log.Fatal(err)
	}

	conn, err := context.Dial("nats.my.domain")
	if err != nil {
		log.Fatal(err)
	}

	if _, err := conn.Write([]byte("hello I am myTestClient")); err != nil {
		panic(err)
	}

}

logs:

go run main.go 
ERRO[0000] error listing controllers, continuing with 1 default configured controller  error="[GET /controllers][401] listControllersUnauthorized  &{Error:0xc0004a0300 Meta:0xc0000af980}"
2025/08/14 10:44:25 failed to dial: no apiSession, authentication attempt failed: error for request vIxfrbUKlL: UNAUTHORIZED: The request could not be completed. The session is not authorized or the credentials are invalid, caused by: error for request : UNHANDLED: UNAUTHORIZED: The request could not be completed. The session is not authorized or the credentials are invalid
exit status 1

Note: When i import api.json into my ziti tunnel, i can access services.

I not sure, but i think you're hitting a race condition, but I'm surprised you're just hitting it now. There are events you can subscribe to that are emitted that allow you to know when services are available. If you put a delay between new context and dialing, i would bet it works properly. Could you give a try and see if that's what's happening? What version of go were you on before?

Code:

package main

import (
	"log"
	"time"

	"github.com/openziti/sdk-golang/ziti"
)

func main() {
	cfg, err := ziti.NewConfigFromFile("api.json")
	if err != nil {
		log.Fatal(err)
	}

	context, err := ziti.NewContext(cfg)
	if err != nil {
		log.Fatal(err)
	}

	time.Sleep(3 * time.Second)
	conn, err := context.Dial("nats.my.domain")
	if err != nil {
		log.Fatal(err)
	}

	if _, err := conn.Write([]byte("hello I am myTestClient")); err != nil {
		panic(err)
	}

}

logs:

go run main.go 
ERRO[0003] error listing controllers, continuing with 1 default configured controller  error="[GET /controllers][401] listControllersUnauthorized  &{Error:0xc00048c600 Meta:0xc0004a6180}"
2025/08/14 13:34:57 failed to dial: no apiSession, authentication attempt failed: error for request MM1EycBjW: UNAUTHORIZED: The request could not be completed. The session is not authorized or the credentials are invalid, caused by: error for request : UNHANDLED: UNAUTHORIZED: The request could not be completed. The session is not authorized or the credentials are invalid
exit status 1

Previous go version: go 1.24.1

go version now: go 1.24.6

Instead of the sleep, add a check to make sure the service exists before trying to dial.

	foundSvc, ok := ctx.GetService(serviceName)
	if !ok {
		panic("error when retrieving the service")
	}

see if that resolves your problem, I believe it will block (which will be quick) until the service is available (or return that it's not)

I just ran the reflect example and it works fine for me, I believe I've hit this particular nuance before and the getService will work through it. It's also probably a good thing to check for anyway imo.

Also you're sure that identity has access, right? "authentication attempt failed: error for request MM1EycBjW: UNAUTHORIZED" seems clear, but you're saying just moving between go versions 'fixes' it?

I added get service, getting errors:

ERRO[0000] error listing controllers, continuing with 1 default configured controller  error="[GET /controllers][401] listControllersUnauthorized  &{Error:0xc000060600 Meta:0xc0000afa00}" WARN[0000] failed to get service: no apiSession, authentication attempt failed: error for request Gd3sWGBjW: UNAUTHORIZED: The request could not be completed. The session is not authorized or the credentials are invalid, caused by: error for request : UNHANDLED: UNAUTHORIZED: The request could not be completed. The session is not authorized or the credentials are invalid  panic: error when retrieving the service


No, this started to happen when i updated ziti golang sdk(go get -U ./…).

Inside of zac it has dial access to the service.

I'm not able to reproduce the problem you're seeing using the examples you've provided so far. It seems like the identity isn't authorized to me.

Can you give me steps to reproduce using a ziti edge quickstart instance? For example, I did the following:

# terminal instance 1:
* clone/get the latest from main of sdk-golang
* cd sdk-golang/example/reflect
* ziti edge quickstart

# in new terminal instance 2:
* ziti edge create identity test -o test.jwt
* ziti edge enroll test.jwt
* ziti edge create service testsvc
* ziti edge create service-policy bind-testservice Bind --identity-roles '@test' --service-roles '@testsvc'
* ziti edge create service-policy dial-testservice Dial --identity-roles '@test' --service-roles '@testsvc'
* go run main.go server --serviceName testsvc --identity ./test.json

# in new terminal instance 3:
* go run main.go client --serviceName testsvc --identity ./test.json

quickstart-example-go

You could try this with your controller instead of the quickstart as well

I tried to recreate your example, but with my ziti ctrl/router, still no luck.

go run main.go server --serviceName testsvc --identity ./api.json
ERROR   error listing controllers, continuing with 1 default configured controller  error="[GET /controllers][401] listControllersUnauthorized  &{Error:0xc00019a780 Meta:0xc000116500}"
PANIC   failed to listen: no apiSession, authentication attempt failed: error for request f0bvdouV1S: UNAUTHORIZED: The request could not be completed. The session is not authorized or the credentials are invalid, caused by: error for request : UNHANDLED: UNAUTHORIZED: The request could not be completed. The session is not authorized or the credentials are invalid

policy advisor:

OKAY : api (1) -> testsvc (1) Common Routers: (1/1) Dial: Y Bind: Y 

OKAY : api (1) -> nats.r1.bct.tech (1) Common Routers: (1/1) Dial: Y Bind: N 

OKAY : api (1) -> zac.r1.bct.tech (1) Common Routers: (1/1) Dial: Y Bind: N 

OKAY : api (1) -> livekit.r1.bct.tech (1) Common Routers: (1/1) Dial: Y Bind: N 

Another weird thing(maybe this is new), after creating policies for identities there are no tags associated, but right side columns still shows testsvc.

I will try to use clean ziti ctrl/router later to check if it works that way.

Ok let us know how that goes. You should also try with a new, clean identity instead of the api.json like I showed using the ziti CLI against your controller too. I'm wondering if the identity was reset or if something odd like that is possibly happening.

Hello,

I did try with a fresh controller/router from ziti quickstart docker-compose.yaml. I get the same error. I created a github repo: GitHub - CarlosHleb/ziti-test

go run main.go server --serviceName testsvc --identity ./test.json
ERROR   error listing controllers, continuing with 1 default configured controller  error="[GET /controllers][401] listControllersUnauthorized  &{Error:0xc0002dc240 Meta:0xc00012fac0}"
PANIC   failed to listen: no apiSession, authentication attempt failed: error for request 0p4l8B21GD: UNAUTHORIZED: The request could not be completed. The session is not authorized or the credentials are invalid, caused by: error for request : UNHANDLED: UNAUTHORIZED: The request could not be completed. The session is not authorized or the credentials are invalid 
panic: (*logrus.Entry) 0xc000214a80

goroutine 1 [running]:
github.com/sirupsen/logrus.(*Entry).log(0xc0003fe770, 0x0, {0xc00062a480, 0x166})
        /home/carlos/.gvm/pkgsets/go1.24.6/global/pkg/mod/github.com/sirupsen/logrus@v1.9.3/entry.go:260 +0x485
github.com/sirupsen/logrus.(*Entry).Log(0xc0003fe770, 0x0, {0xc000599c00?, 0xc0001c8401?, 0xa38274?})
        /home/carlos/.gvm/pkgsets/go1.24.6/global/pkg/mod/github.com/sirupsen/logrus@v1.9.3/entry.go:304 +0x48
github.com/sirupsen/logrus.(*Entry).Panic(...)
        /home/carlos/.gvm/pkgsets/go1.24.6/global/pkg/mod/github.com/sirupsen/logrus@v1.9.3/entry.go:342
github.com/openziti/sdk-golang/example/reflect/cmd.Server(0x1371580?, {0x7ffdad89621c, 0x7})
        /home/carlos/.gvm/pkgsets/go1.24.6/global/pkg/mod/github.com/openziti/sdk-golang/example@v0.0.0-20250814072646-4b3d24004b8c/reflect/cmd/server.go:21 +0x96
main.main.func2(0xc0001a6f00?, {0xceb83c?, 0x4?, 0xceb840?})
        /home/carlos/Projects/screen-control/ziti/main.go:41 +0x4e
github.com/spf13/cobra.(*Command).execute(0xc0001d0f08, {0xc0003fa780, 0x4, 0x4})
        /home/carlos/.gvm/pkgsets/go1.24.6/global/pkg/mod/github.com/spf13/cobra@v1.9.1/command.go:1019 +0xa91
github.com/spf13/cobra.(*Command).ExecuteC(0x1371580)
        /home/carlos/.gvm/pkgsets/go1.24.6/global/pkg/mod/github.com/spf13/cobra@v1.9.1/command.go:1148 +0x46f
github.com/spf13/cobra.(*Command).Execute(...)
        /home/carlos/.gvm/pkgsets/go1.24.6/global/pkg/mod/github.com/spf13/cobra@v1.9.1/command.go:1071
main.main()
        /home/carlos/Projects/screen-control/ziti/main.go:54 +0x2bc
exit status 2

and policy advisor:

OKAY : test (2) -> testsvc (2) Common Routers: (2/2) Dial: Y Bind: Y 

I was able to reproduce the issue with your repo, thank you for putting that together.

I tried your example using 1.2.2, and the problem did not happen. The only change from 1.2.2 to 1.2.3 is that 1.2.3 will check if OIDC is available and use it, if possible.

I tried running my local dev environment with and without the OIDC endpoint configured, and it worked fine both ways. There’s definitely something odd going on there, and I’ll see if I can track it down today.

I’ll let you know what I find.

Thank you,

Paul

2 Likes

Here's the issue: OIDC authentication fails if the client api has a separate cert chain · Issue #3231 · openziti/ziti · GitHub

There a PR with a fix, so hopefully the fix will be in the next release.

Cheers,
Paul

1 Like