Helm Port Mappings

81

Was that this step?:

ziti edge create edge-router-policy "all-routers" \
    --edge-router-roles '#all' --identity-roles '#all'

ziti edge create service-edge-router-policy "all-routers" \
    --edge-router-roles '#all' --service-roles '#all'

There is only one router currently.

The output of command is this:

Output format: STATUS: ID (ID ROUTERS) -> SVC (SVC ROUTERS) Common Routers: (ONLINE COMMON ROUTERS/COMMON ROUTERS) Dial: DIAL_OK Bind: BIND_OK. ERROR_LIST
-------------------------------------------------------------------------------
OKAY : device1 (1) -> Nginx (1) Common Routers: (1/1) Dial: Y Bind: N 

OKAY : edge-router (1) -> Nginx (1) Common Routers: (1/1) Dial: N Bind: Y 

OKAY : device2 (1) -> Nginx (1) Common Routers: (1/1) Dial: Y Bind: N

The only mention of "bind" in the controller logs is this:

[30680.106]    INFO ziti/controller/handler_ctrl.(*CtrlAccepter).Bind: {routerId=[OPHIgSXPFi]} accepted new router connection [r/OPHIgSXPFi]

And the router:

[   0.005] WARNING ziti/router/internal/edgerouter.parseEdgeListenerOptions: port in [listeners[0].options.advertise] must equal port in [listeners[0].address] for edge binding but did not. Got [8443] [3022]
[   0.006]    INFO ziti/router.(*Router).showOptions: ctrl = {"OutQueueSize":4,"MaxQueuedConnects":1,"MaxOutstandingConnects":16,"ConnectTimeout":5000000000,"DelayRxStart":false,"WriteTimeout":0}
[   0.006]    INFO ziti/router.(*Router).showOptions: metrics = {"ReportInterval":60000000000,"IntervalAgeThreshold":0,"MessageQueueSize":10}
[   0.006]    INFO ziti/common/metrics.GoroutinesPoolMetricsConfigF.func1.1: {idleTime=[30s] maxQueueSize=[5000] poolType=[pool.link.dialer] minWorkers=[0] maxWorkers=[32]} starting goroutine pool
[   0.006]    INFO ziti/router.(*Router).initializeHealthChecks: starting health check with ctrl ping initially after 15s, then every 30s, timing out after 15s
[   0.006]    INFO ziti/router.(*Router).startXlinkDialers: started Xlink dialer with binding [transport]
[   0.006] WARNING ziti/router/xlink_transport.loadListenerConfig: {addr=[tls:0.0.0.0:10080] error=[no network interface found for 0.0.0.0]} unable to get interface for address
[   0.006]    INFO ziti/common/metrics.GoroutinesPoolMetricsConfigF.func1.1: {maxWorkers=[16] poolType=[pool.listener.link] maxQueueSize=[1] idleTime=[10s] minWorkers=[1]} starting goroutine pool
[   0.006]    INFO ziti/router.(*Router).startXlinkListeners: started Xlink listener with binding [transport] advertising [tls:ziti-router.domain.com:443]
[   0.006]    INFO ziti/router/xgress_edge.(*listener).Listen: {address=[tls:0.0.0.0:3022]} starting channel listener
[   0.006]    INFO ziti/common/metrics.GoroutinesPoolMetricsConfigF.func1.1: {idleTime=[10s] maxQueueSize=[1] poolType=[pool.listener.xgress_edge] minWorkers=[1] maxWorkers=[16]} starting goroutine pool
[   0.006]    INFO ziti/router.(*Router).startXgressListeners: created xgress listener [edge] at [tls:0.0.0.0:3022]
[   0.006]    INFO ziti/router.(*Router).startXgressListeners: created xgress listener [tunnel] at []
[   0.006]    INFO ziti/router/xgress_edge.(*Acceptor).Run: starting
[   0.007]    INFO ziti/router.(*Router).getInitialCtrlEndpoints: controller endpoints file [/etc/ziti/config/endpoints] doesn't exist. Using initial endpoints from config
82

Does the full openssl pipeline give you an x509 subject matching the Ziti ID of the Ziti Router named "edge-router?"

openssl s_client -connect ziti-router.domain.com:8443  -alpn ziti-edge <>/dev/null \
|& openssl x509 -noout -subject

No need to re-do those two create commands. Those were a one-time setup to provide blanket router policies for your Ziti Network.

[   0.005] WARNING ziti/router/internal/edgerouter.parseEdgeListenerOptions: port in [listeners[0].options.advertise] must equal port in [listeners[0].address] for edge binding but did not. Got [8443] [3022]

I believe we can ignore this WARNING message because it's saying the listening and advertised ports don't match, which is expected when Ziti Router is behind a load balancer. I'll try to get clarification from @andrew.martinez who last modified that line of edgerouter/config.go.

Here's what my Router's config.yaml looks like for comparison. I'm also asking others to peek at what might be preventing the Ziti Router named "edge-router" from creating a terminator for the Service it's authorized to bind.

kubectl get configmap ziti-router-config -o go-template='{{ index .data "ziti-router.yaml"}}'

v: 3

# this dir is a var because the value is a writeable tmp dir during
#  enrollment and a read-only mountpoint when running
identity:
  cert:        ${ZITI_ROUTER_IDENTITY_DIR}/client.crt
  server_cert: ${ZITI_ROUTER_IDENTITY_DIR}/tls.crt
  key:         ${ZITI_ROUTER_IDENTITY_DIR}/tls.key
  ca:          ${ZITI_ROUTER_IDENTITY_DIR}/ca.crt

ctrl:
  # router control plane API (:6262)
  endpoint:    tls:ziti-controller-ctrl.miniziti.svc:443

link:
  dialers:
    - binding: transport
  # When 'transport' is disabled this means we are a 'private' router, i.e.,
  # not providing incoming links to other routers. Private routers still
  # join the mesh, but only form outgoing links.
  listeners:
    - binding:          transport
      bind:             tls:0.0.0.0:10080
      advertise:        tls:miniziti-router-transport.192.168.49.2.sslip.io:443
      options:
        outQueueSize:   4
listeners:
# bindings of edge and tunnel requires an "edge" section below
  - binding: edge
    address: tls:0.0.0.0:3022
    options:
        advertise: miniziti-router.192.168.49.2.sslip.io:443
        connectTimeoutMs: 1000
        getSessionTimeout: 60
  - binding: tunnel
    options:
        mode: host

edge:
  csr:
    sans:
      dns:
        - localhost
        - miniziti-router.192.168.49.2.sslip.io
        - miniziti-router-transport.192.168.49.2.sslip.io
      ip:
        - 127.0.0.1

#transport:
#  ws:
#    writeTimeout: 10
#    readTimeout: 5
#    idleTimeout: 5
#    pongTimeout: 60
#    pingInterval: 54
#    handshakeTimeout: 10
#    readBufferSize: 4096
#    writeBufferSize: 4096
#    enableCompression: true
#    server_cert: ~/.ziti/config/certs/192.168.10.11.server.chain.cert
#    key: ~/.ziti/config/certs/192.168.10.11.key

forwarder:
    latencyProbeInterval: 10
    xgressDialQueueLength: 1000
    xgressDialWorkerCount: 128
    linkDialQueueLength: 1000
    linkDialWorkerCount: 32%
83

@thedarkula Just in case there's a bug with the Ziti Router, will you please delete the pod and re-test after K8s re-schedules the replacement pod and it is ready?

84

I deleted the pod and retried everything.

Here are the router configmap contents:

v: 3

# this dir is a var because the value is a writeable tmp dir during
#  enrollment and a read-only mountpoint when running
identity:
  cert:        ${ZITI_ROUTER_IDENTITY_DIR}/client.crt
  server_cert: ${ZITI_ROUTER_IDENTITY_DIR}/tls.crt
  key:         ${ZITI_ROUTER_IDENTITY_DIR}/tls.key
  ca:          ${ZITI_ROUTER_IDENTITY_DIR}/ca.crt

ctrl:
  # router control plane API (:6262)
  endpoint:    tls:ziti-controller.domain.com:8440

link:
  dialers:
    - binding: transport
  # When 'transport' is disabled this means we are a 'private' router, i.e.,
  # not providing incoming links to other routers. Private routers still
  # join the mesh, but only form outgoing links.
  listeners:
    - binding:          transport
      bind:             tls:0.0.0.0:10080
      advertise:        tls:ziti-router.domain.com:443
      options:
        outQueueSize:   4
listeners:
# bindings of edge and tunnel requires an "edge" section below
  - binding: edge
    address: tls:0.0.0.0:3022
    options:
        advertise: ziti-router.domain.com:8443
        connectTimeoutMs: 1000
        getSessionTimeout: 60
  - binding: tunnel
    options:
        mode: host

edge:
  csr:
    sans:
      dns:
        - localhost
        - ziti-router.domain.com
        - ziti-router.domain.com
      ip:
        - 127.0.0.1

#transport:
#  ws:
#    writeTimeout: 10
#    readTimeout: 5
#    idleTimeout: 5
#    pongTimeout: 60
#    pingInterval: 54
#    handshakeTimeout: 10
#    readBufferSize: 4096
#    writeBufferSize: 4096
#    enableCompression: true
#    server_cert: ~/.ziti/config/certs/192.168.10.11.server.chain.cert
#    key: ~/.ziti/config/certs/192.168.10.11.key

forwarder:
    latencyProbeInterval: 10
    xgressDialQueueLength: 1000
    xgressDialWorkerCount: 128
    linkDialQueueLength: 1000
    linkDialWorkerCount: 32

Here are the router logs when I try to pull up http://nginx.ziti:

[167041.248]   ERROR channel/v2.AcceptNextChannel.func1: {error=[no api session found for token [935a22c2-d932-40fa-b7e7-6c40b725d840], fingerprints: [map[604b17ff54f562511d7a53c37c9599632b078f81:0xc000c52000]], subjects [[CN=crJtg0XPf]]]} failure accepting channel edge with underlay u{classic}->i{j1Nv}
[167060.608] WARNING ziti/router/xgress_edge.(*edgeClientConn).processConnect [ch{edge}->u{classic}->i{m7Dj}]: {error=[service XZpvBjOpcakT6jrlEoZsE has no terminators] token=[41376fcc-2b6d-47f2-b708-b3c122c34e2d] connId=[12] type=[EdgeConnectType] chSeq=[39] edgeSeq=[0]} failed to dial fabric
[167061.707] WARNING ziti/router/xgress_edge.(*edgeClientConn).processConnect [ch{edge}->u{classic}->i{m7Dj}]: {chSeq=[41] edgeSeq=[0] error=[service XZpvBjOpcakT6jrlEoZsE has no terminators] connId=[13] type=[EdgeConnectType] token=[41376fcc-2b6d-47f2-b708-b3c122c34e2d]} failed to dial fabric
[167066.756] WARNING ziti/router/xgress_edge.(*edgeClientConn).processConnect [ch{edge}->u{classic}->i{m7Dj}]: {connId=[14] type=[EdgeConnectType] chSeq=[43] error=[service XZpvBjOpcakT6jrlEoZsE has no terminators] edgeSeq=[0] token=[41376fcc-2b6d-47f2-b708-b3c122c34e2d]} failed to dial fabric
85

I don't see any glaring issues there. Were you able to fetch the Edge listener's TLS server certificate with that openssl command, and parse out the subject to see if it matches the Router's Ziti ID?

86

I went to the console, at /router-policies. There are two policies, the All-Routers one you had me create, and one that I did not create, called edge-router-OPHIgSXPFi-system.

The output of the openssl command is this:

subject=C = , ST = , L = , O = , OU = , CN = OPHIgSXPFi
1 Like
87

Let's try something. Change the service edge router policy to allow @edge-router instead of #all. I'm wondering if we're encountering this issue.

You can do this with the CLI or Ziti Console.

ziti edge update serp "all-routers" \
    --edge-router-roles "@edge-router" \
    --service-roles "#all"
88

A thread summary to assist helpgivers and onlookers: The Ziti Router's built-in tunnel has mode "host," has permission to bind a Ziti Service, and can reach the address in the host.v1 config, but no terminators are created.

The ERP and SERP are #all/#all.

89

I updated the service edge router policy, deleted both the controller and router pod (just to make sure), and it still says no terminators:

[  29.065] WARNING ziti/router/xgress_edge.(*edgeClientConn).processConnect [ch{edge}->u{classic}->i{1ol0}]: {token=[f1b24994-3e3a-488c-b299-1e249056437d] connId=[6] type=[EdgeConnectType] chSeq=[16] edgeSeq=[0] error=[service XZpvBjOpcakT6jrlEoZsE has no terminators]} failed to dial fabric
90

Have you collected a full, clean set of logs from the controller and the router? This is so consistent, it must be entirely reproducible for you (which would be great) or it's something unexpected and silly happening... I've had this problem in the past but it's only ever a single time and it's never been as consistently reproducible as this.

You can send them to clint at openziti.org if you want, or try to DM, or send through a zrok.io share or any other mechanism you are comfortable with?

1 Like
91

I just collected the logs from fresh pods and repeated the whole connection process.
You should have them in your inbox :slight_smile:

1 Like
92

I've received the logs and am analyzing them now. Paul doesn't think this is the same issue I linked earlier in which a service is not authorized to connect to the edge router when it is bound or dialed via the router's built-in tunneler. He said this looks more like a race condition.

93

@thedarkula Are there no other mentions of service id XZpvBjOpcakT6jrlEoZsE in the router's log? The first mention I see is the no terminators error, but that router has bind permission for the service according to policy-advisor, so we should see lines like these. My test service is named "kentest."

[ 465.338]    INFO ziti/tunnel/intercept.(*ServiceListener).HandleServicesChange: {service=[kentest]} adding service
[ 465.338]    INFO ziti/tunnel/intercept.(*ServiceListener).addService: {serviceId=[4GldntOsH5GxPGqQ2CIbZ5] serviceName=[kentest]} Hosting newly available service
[ 465.338]    INFO ziti/router/xgress_edge_tunnel.(*fabricProvider).establishTerminatorWithRetry.func1: {service=[kentest]} attempting to establish terminator
[ 465.348]    INFO ziti/router/xgress_edge_tunnel.(*fabricProvider).HandleTunnelResponse: {routerId=[9Q9ukYTfI2] terminatorId=[a0912e47-c791-43df-9dc0-d2ab272e8644] sessionId=[clph0ao3000f90w4v1nd8c1s7]} received new session
[ 465.348]    INFO ziti/router/xgress_edge_tunnel.(*fabricProvider).HandleTunnelResponse: {terminatorId=[a0912e47-c791-43df-9dc0-d2ab272e8644] createDuration=[9.484674ms] routerId=[9Q9ukYTfI2]} received terminator created notification

Let's also run this command to look for policy misalignment between the Ziti Router's tunneler named "edge-router" and the Ziti Service named "Nginx."

ziti edge policy-advisor identities --quiet "edge-router"
94

Hi @thedarkula ,
I don't see it in the log snippet above, can you check that something like the following line is being emitted?

[50861.379]    INFO ziti/router/xgress_edge_tunnel.(*tunneler).Start: {mode=[host]} creating interceptor

That's a good indicator that the edge tunneling function is enabled and starting up. The other thing to check is that the router was created with tunneling enabled (via the --tunneler-enabled flag). If tunneling is configured in the config file, but not in the model, the router should fail to start with an error like this:

[50769.562]   ERROR ziti/router/xgress_edge_tunnel.(*fabricProvider).authenticate.func1: {error=[tunneling not enabled] ctrlId=[ctrl_client]} failed to authenticate
[50769.562]   FATAL ziti/router/xgress_edge_tunnel.(*servicePoller).pollServices: {error=[tunneling not enabled]} xgress_edge_tunnel unable to authenticate to controller

The isTunnelerEnabled flag can be check in the json output of ziti edge list edge-routers.

ziti edge list ers -j  | jq '.data | .[] | { id: .id, isTunnelerEnabled: .isTunnelerEnabled }'

Thank you,
Paul

1 Like
95

@qrkourier Here are some logs looking for that service ID:

kubectl -n ziti logs --selector app.kubernetes.io/component=ziti-router --tail=-1 -f | rg XZpvBjOpcakT6jrlEoZsE

[  73.409] WARNING ziti/router/xgress_edge.(*edgeClientConn).processConnect [ch{edge}->u{classic}->i{2704}]: {type=[EdgeConnectType] chSeq=[3] edgeSeq=[0] token=[1f0c1b46-142e-4876-91bb-d72c141d63f6] error=[service XZpvBjOpcakT6jrlEoZsE has no terminators] connId=[1]} failed to dial fabric
[  73.418] WARNING ziti/router/xgress_edge.(*edgeClientConn).processConnect [ch{edge}->u{classic}->i{2704}]: {chSeq=[4] edgeSeq=[0] token=[1f0c1b46-142e-4876-91bb-d72c141d63f6] error=[service XZpvBjOpcakT6jrlEoZsE has no terminators] connId=[2] type=[EdgeConnectType]} failed to dial fabric
[  74.535] WARNING ziti/router/xgress_edge.(*edgeClientConn).processConnect [ch{edge}->u{classic}->i{2704}]: {connId=[3] type=[EdgeConnectType] chSeq=[7] edgeSeq=[0] token=[1f0c1b46-142e-4876-91bb-d72c141d63f6] error=[service XZpvBjOpcakT6jrlEoZsE has no terminators]} failed to dial fabric

Here is the output of the identity:

ziti edge policy-advisor identities --quiet "edge-router"

OKAY : edge-router (1) -> Nginx (1) Common Routers: (1/1) Dial: N Bind: Y

@plorenz Here are the router logs, looking for interceptor:

kubectl -n ziti logs --selector app.kubernetes.io/component=ziti-router --tail=-1 -f | rg interceptor

[   0.007]    INFO ziti/router/xgress_edge_tunnel.(*tunneler).Start: {mode=[host]} creating interceptor

And here, looking for unable:

kubectl -n ziti logs --selector app.kubernetes.io/component=ziti-router --tail=-1 -f | rg unable

[   0.005] WARNING ziti/router/xlink_transport.loadListenerConfig: {error=[no network interface found for 0.0.0.0] addr=[tls:0.0.0.0:10080]} unable to get interface for address
[   0.008] WARNING ziti/tunnel/dns.flushDnsCaches: {error=[exec: "resolvectl": executable file not found in $PATH]} unable to find systemd-resolve or resolvectl in path, consider adding a dns flush to your restart process

And the output of edge routers:

ziti edge list ers -j  | jq '.data | .[] | { id: .id, isTunnelerEnabled: .isTunnelerEnabled }'

{
  "id": "OPHIgSXPFi",
  "isTunnelerEnabled": true
}

Also, at the end of the router logs, there is a bunch of this on loop, now:

[371312.412]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:3022]: {error=[tls: client offered only unsupported versions: []]} handshake failed
[371547.554]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:3022]: {error=[tls: client didn't provide a certificate]} handshake failed
[376818.941]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:3022]: {error=[tls: first record does not look like a TLS handshake]} handshake failed
[381816.800]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:3022]: {error=[not handler for requested protocols [hq h2c h2 spdy/3 spdy/2 spdy/1 http/1.1 http/1.0 http/0.9]]} handshake failed
[381817.282]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:3022]: {error=[not handler for requested protocols [http/0.9 http/1.0 http/1.1 spdy/1 spdy/2 spdy/3 h2 h2c hq]]} handshake failed
[381817.845]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:3022]: {error=[not handler for requested protocols [http/0.9 http/1.0 http/1.1 spdy/1 spdy/2 spdy/3 h2 h2c hq]]} handshake failed
[381818.318]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:3022]: {error=[not handler for requested protocols [http/0.9 http/1.0 spdy/1 spdy/2 spdy/3 h2c hq]]} handshake failed
[381818.729]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:3022]: {error=[not handler for requested protocols [hq h2c spdy/3 spdy/2 spdy/1 http/1.0 http/0.9]]} handshake failed
[381819.134]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:3022]: {error=[not handler for requested protocols [http/0.9 http/1.0 http/1.1 spdy/1 spdy/2 spdy/3 h2 h2c hq]]} handshake failed
[381819.582]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:3022]: {error=[not handler for requested protocols [hq h2c h2 spdy/3 spdy/2 spdy/1 http/1.1 http/1.0 http/0.9]]} handshake failed
[381819.965]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:3022]: {error=[not handler for requested protocols [http/0.9 http/1.0 http/1.1 spdy/1 spdy/2 spdy/3 h2 h2c hq]]} handshake failed
[381820.374]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:3022]: {error=[not handler for requested protocols [http/0.9 http/1.0 http/1.1 spdy/1 spdy/2 spdy/3 h2 h2c hq]]} handshake failed
[381820.870]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:3022]: {error=[not handler for requested protocols [hq h2c h2 spdy/3 spdy/2 spdy/1 http/1.1 http/1.0 http/0.9]]} handshake failed
[387868.792]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:3022]: {error=[tls: client didn't provide a certificate]} handshake failed
[388118.764]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:3022]: {error=[tls: client didn't provide a certificate]} handshake failed
[388119.600]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:3022]: {error=[tls: client didn't provide a certificate]} handshake failed
[388119.798]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:3022]: {error=[tls: first record does not look like a TLS handshake]} handshake failed
[388125.182]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:3022]: {error=[context deadline exceeded]} handshake failed
[388125.603]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:3022]: {error=[tls: first record does not look like a TLS handshake]} handshake failed
[388500.136]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:3022]: {error=[tls: client didn't provide a certificate]} handshake failed
96

I'm unsure why the Router's tunneler isn't creating a terminator. Let's try extending the hosting permission to an instance of ziti-edge-tunnel run-host.

  1. You'll need another Ziti Identity with a role like "nginx-hosts"
  2. Install the ziti-host chart with the enrollment token of the new Identity, which provides a reverse proxy pod in the cluster: How to Install a Ziti Hosting Tunneler in Kubernetes | OpenZiti
  3. In the Bind Service Policy that grants permissions for the Ziti Service, allow Identity role #nginx-hosts.
  4. Re-test the Nginx Service.

You should see the Nginx Service name and ID mentioned in the application log from the ziti-host reverse proxy pod, and the Service should be reachable from Android.

If this works then it points toward a configuration problem with the Router, or possibly a bug.

97

@qrkourier Reviving this thread with all new versions :slight_smile:

Using the latest controller helm chart:

helm upgrade --install ziti-controller openziti/ziti-controller --namespace ziti --create-namespace --version 1.0.16 --values values-controller.yaml

Ziti CLI version:

ziti --version
v1.1.15

Logging in:

ziti edge login ziti-controller.domain.com:8441 --yes --username admin --password $(kubectl -n ziti get secrets ziti-controller-admin-secret -o go-template='{{index .data "admin-password" | base64decode }}')

Login Error:

error: unable to authenticate to https://ziti-controller.domain.com:8441/edge/management/v1. Status code: 401 Unauthorized, Server returned: {
    "error": {
        "code": "INVALID_AUTH",
        "message": "The authentication request failed",
        "requestId": "IkVCwTPHr"
    },
    "meta": {
        "apiEnrollmentVersion": "0.0.1",
        "apiVersion": "0.0.1"
    }
}

The controller logs show this:

{"_context":"tls:0.0.0.0:1280","error":"remote error: tls: bad certificate","file":"github.com/openziti/transport/v2@v2.0.143/tls/listener.go:257","func":"github.com/openziti/transport/v2/tls.(*sharedListener).processConn","level":"error","msg":"handshake failed","remote":"192.168.1.1:58044","time":"2024-10-20T18:07:02.419Z"}

I have also removed the ~/.config/ziti folder to be certain that there were no artifacts.

98

Hi again :wave: The controller is rejecting the password. I recommend troubleshooting this by breaking it up into more steps.

  1. ensure the get secret command is actually yielding a password
  2. try the login password with --verbose --output-request-json to see more detail
❯ ziti edge login --verbose --output-request-json https://miniziti-controller.192.168.58.2.sslip.io -u admin -p "$ZITI_PWD"
RESTY 2024/10/21 13:13:53
---------------------- REQUEST LOG -----------------------
POST  /edge/management/v1/authenticate?method=password  HTTP/1.1
HOST   : miniziti-controller.192.168.58.2.sslip.io
HEADERS:
                   Accept: application/json
             Content-Type: application/json
               User-Agent: go-resty/1.12.0 (https://github.com/go-resty/resty)
BODY   :

{
   "password": "CqcRVtmX3vyn5fIkgEU8nIfAHZsNtNbX",
   "username": "admin"
}

----------------------------------------------------------
RESTY 2024/10/21 13:13:53
---------------------- RESPONSE LOG -----------------------
STATUS          : 200 OK
RECEIVED AT     : 2024-10-21T13:13:53.091329149-04:00
RESPONSE TIME   : 20.694292ms
HEADERS:
             Content-Type: application/json
                     Date: Mon, 21 Oct 2024 17:13:53 GMT
       Expiration-Seconds: 1800
               Expires-At: 2024-10-21 17:43:53.078079891 +0000 UTC
                   Server: ziti-controller/v1.1.15
         Ziti-Instance-Id: cm2j8oozp00000d6hle8c0qc2
               Zt-Session: 375177db-e478-4db2-b85d-90d014a29ed1
BODY   :

{
   "data": {
      "_links": {
         "self": {
            "href": "./api-sessions/cm2ja03p201tb0d6hp71h7vog"
         },
         "sessions": {
            "href": "./api-sessions/cm2ja03p201tb0d6hp71h7vog/sessions"
         }
      },
      "createdAt": "2024-10-21T17:13:53.078Z",
      "id": "cm2ja03p201tb0d6hp71h7vog",
      "tags": {},
      "updatedAt": "2024-10-21T17:13:53.078Z",
      "authQueries": [],
      "authenticatorId": "0kFiG.HA5Q",
      "cachedLastActivityAt": "2024-10-21T17:13:53.078Z",
      "configTypes": [],
      "identity": {
         "_links": {
            "auth-policies": {
               "href": "./auth-policies/default"
            },
            "authenticators": {
               "href": "./identities/0k1i2.HAV/authenticators"
            },
            "edge-router-policies": {
               "href": "./identities/0k1i2.HAV/edge-router-policies"
            },
            "edge-routers": {
               "href": "./identities/0k1i2.HAV/edge-routers"
            },
            "enrollments": {
               "href": "./identities/0k1i2.HAV/enrollments"
            },
            "failed-service-requests": {
               "href": "./identities/0k1i2.HAV/failed-service-requests"
            },
            "posture-data": {
               "href": "./identities/0k1i2.HAV/posture-data"
            },
            "self": {
               "href": "./identities/0k1i2.HAV"
            },
            "service-configs": {
               "href": "./identities/0k1i2.HAV/service-configs"
            },
            "service-policies": {
               "href": "./identities/0k1i2.HAV/service-policies"
            },
            "services": {
               "href": "./identities/0k1i2.HAV/services"
            }
         },
         "entity": "identities",
         "id": "0k1i2.HAV",
         "name": "Default Admin"
      },
      "identityId": "0k1i2.HAV",
      "ipAddress": "10.244.0.6",
      "isCertExtendable": false,
      "isMfaComplete": false,
      "isMfaRequired": false,
      "lastActivityAt": "2024-10-21T17:13:53.077Z",
      "token": "375177db-e478-4db2-b85d-90d014a29ed1",
      "expirationSeconds": 1800,
      "expiresAt": "2024-10-21T17:43:53.077Z"
   },
   "meta": {}
}

----------------------------------------------------------
Token: 375177db-e478-4db2-b85d-90d014a29ed1
Saving identity 'zititest' to /home/kbingham/.config/ziti/ziti-cli.json

This is a misleading message because it's not really an error, but expected. Here's the GH issue for improving or eliminating the log message.

99

I actually did have the password step broken up before :slight_smile:

Here is the extended output:

kubectl -n ziti get secret ziti-controller-admin-secret -o go-template='{{range $k,$v := .data}}{{printf "%s: " $k}}{{if not $v}}{{$v}}{{else}}{{$v | base64decode}}{{end}}{{"\n"}}{{end}}'

admin-password: PASSWORD_HERE
admin-user: admin

ziti edge login ziti-controller.domain.com:8441 --yes --username admin --password PASSWORD_HERE --verbose --output-request-json

RESTY 2024/10/21 17:28:07 
---------------------- REQUEST LOG -----------------------
POST  /edge/management/v1/authenticate?method=password  HTTP/1.1
HOST   : ziti-controller.domain.com:8441
HEADERS:
                   Accept: application/json
             Content-Type: application/json
               User-Agent: go-resty/1.12.0 (https://github.com/go-resty/resty)
BODY   :
{
   "password": "PASSWORD_HERE",
   "username": "admin"
}
----------------------------------------------------------
RESTY 2024/10/21 17:28:07 
---------------------- RESPONSE LOG -----------------------
STATUS 		: 401 Unauthorized
RECEIVED AT	: 2024-10-21T17:28:07.207110424Z
RESPONSE TIME	: 18.013827ms
HEADERS:
             Content-Type: application/json
                     Date: Mon, 21 Oct 2024 17:28:07 GMT
                   Server: ziti-controller/v1.1.9
         Ziti-Instance-Id: cm2jahsku00000da19ttzgton
BODY   :
{
   "error": {
      "code": "INVALID_AUTH",
      "message": "The authentication request failed",
      "requestId": "9vAfNVFyS"
   },
   "meta": {
      "apiEnrollmentVersion": "0.0.1",
      "apiVersion": "0.0.1"
   }
}

----------------------------------------------------------
error: unable to authenticate to https://ziti-controller.domain.com:8441/edge/management/v1. Status code: 401 Unauthorized, Server returned: {
    "error": {
        "code": "INVALID_AUTH",
        "message": "The authentication request failed",
        "requestId": "9vAfNVFyS"
    },
    "meta": {
        "apiEnrollmentVersion": "0.0.1",
        "apiVersion": "0.0.1"
    }
}

It seems to be throwing a 401.

100

Thanks for confirming the password is being retrieved and included in the login command. Does this fail in the same way, after adjusting it to reflect the namespace and Helm release name of your controller installation?

kubectl get pods --selector app.kubernetes.io/component=ziti-controller --output jsonpath="{.items[0].metadata.name}" \
| xargs -IPOD kubectl exec POD --container ziti-controller -- zitiLogin

This runs the zitiLogin script inside the controller container which should have the correct username and password assigned to env vars used by that script.