Docker compose issues

Support
1

Hey,

I’m trying to spin up a network locally using Local - Docker Compose | OpenZiti.

But it looks like it’s not working since the last update.

Logs of one of the routers container :

_ZITI_ROUTER_NAME set to: ziti-edge-router
NOT OVERRIDING: env var ZITI_BIN_DIR already set. using existing value
NOT OVERRIDING: env var ZITI_BIN_ROOT already set. using existing value
NOT OVERRIDING: env var ZITI_CTRL_ADVERTISED_ADDRESS already set. using existing value
NOT OVERRIDING: env var ZITI_CTRL_ADVERTISED_PORT already set. using existing value
NOT OVERRIDING: env var ZITI_CTRL_EDGE_ADVERTISED_ADDRESS already set. using existing value
NOT OVERRIDING: env var ZITI_CTRL_EDGE_ADVERTISED_PORT already set. using existing value
NOT OVERRIDING: env var ZITI_CTRL_NAME already set. using existing value
NOT OVERRIDING: env var ZITI_EDGE_IDENTITY_ENROLLMENT_DURATION already set. using existing value
NOT OVERRIDING: env var ZITI_ENV_FILE already set. using existing value
NOT OVERRIDING: env var ZITI_HOME already set. using existing value
NOT OVERRIDING: env var ZITI_IMAGE already set. using existing value
NOT OVERRIDING: env var ZITI_NETWORK already set. using existing value
NOT OVERRIDING: env var ZITI_ROUTER_ENROLLMENT_DURATION already set. using existing value
NOT OVERRIDING: env var ZITI_ROUTER_LISTENER_BIND_PORT already set. using existing value
NOT OVERRIDING: env var ZITI_ROUTER_NAME already set. using existing value
NOT OVERRIDING: env var ZITI_ROUTER_PORT already set. using existing value
NOT OVERRIDING: env var ZITI_SCRIPTS already set. using existing value
NOT OVERRIDING: env var ZITI_SHARED already set. using existing value
NOT OVERRIDING: env var ZITI_USER already set. using existing value
NOT OVERRIDING: env var ZITI_VERSION already set. using existing value
 
adding /var/openziti/ziti-bin to the path
ZITI_ROUTER_NAME set to: ziti-edge-router
system has not been initialized. initializing...
Untrusted certificate authority retrieved from server
Verified that server supplied certificates are trusted by server
Server supplied 5 certificates
Server certificate chain written to /home/ziti/.config/ziti/certs/ziti-edge-controller
error: unable to authenticate to https://ziti-edge-controller:1280/edge/management/v1. Status code: 401 Unauthorized, Server returned: {
    "error": {
        "code": "INVALID_AUTH",
        "message": "The authentication request failed",
        "requestId": "xu1scM-5-"
    },
    "meta": {
        "apiEnrollmentVersion": "0.0.1",
        "apiVersion": "0.0.1"
    }
}
----------  Creating edge-router ziti-edge-router....
CREATING EDGE ROUTER CONFIG: ziti-edge-router
This will overwrite the existing file, continue? (y/N) public router configuration file written to: /persistent/ziti-edge-router.yaml
error: no identity 'default' found in cli config /home/ziti/.config/ziti/ziti-cli.json
error: no identity 'default' found in cli config /home/ziti/.config/ziti/ziti-cli.json
---------- Enrolling edge-router ziti-edge-router....
[   0.000]   PANIC ziti/ziti/router.enrollGw: could not load JWT file from path [/persistent/ziti-edge-router.jwt]
panic: (*logrus.Entry) 0xc000181180

goroutine 1 [running]:
github.com/sirupsen/logrus.(*Entry).log(0xc00061dce0, 0x0, {0xc0005d8640, 0x44})
	github.com/sirupsen/logrus@v1.9.3/entry.go:260 +0x4d6
github.com/sirupsen/logrus.(*Entry).Log(0xc00061dce0, 0x0, {0xc00061dbd0?, 0x1?, 0x1?})
	github.com/sirupsen/logrus@v1.9.3/entry.go:304 +0x4f
github.com/sirupsen/logrus.(*Entry).Logf(0xc00061dce0, 0x0, {0x3748f62?, 0x0?}, {0xc00061dc80?, 0x0?, 0x0?})
	github.com/sirupsen/logrus@v1.9.3/entry.go:349 +0x85
github.com/sirupsen/logrus.(*Entry).Panicf(...)
	github.com/sirupsen/logrus@v1.9.3/entry.go:387
github.com/openziti/ziti/ziti/router.enrollGw(0xc000a92000?, {0xc000bbbfb0, 0x1, 0x3?})
	github.com/openziti/ziti/ziti/router/enrollgw.go:64 +0x2a5
github.com/spf13/cobra.(*Command).execute(0xc000a92000, {0xc000bbbf50, 0x3, 0x3})
	github.com/spf13/cobra@v1.7.0/command.go:944 +0x847
github.com/spf13/cobra.(*Command).ExecuteC(0x5281c60)
	github.com/spf13/cobra@v1.7.0/command.go:1068 +0x3bd
github.com/spf13/cobra.(*Command).Execute(...)
	github.com/spf13/cobra@v1.7.0/command.go:992
github.com/openziti/ziti/ziti/cmd.Execute()
	github.com/openziti/ziti/ziti/cmd/cmd.go:79 +0x25
main.main()
	github.com/openziti/ziti/ziti/main.go:51 +0x17

system initialized. writing marker file
[   0.000]   ERROR ziti/ziti/router.run: {build-date=[2023-08-10T17:19:11Z] error=[unable to load identity (open /persistent/ziti-edge-router.key: no such file or directory)] revision=[2adab4fc9127] configFile=[/persistent/ziti-edge-router.yaml] os=[linux] arch=[amd64] version=[v0.30.0] go-version=[go1.20.7]} error loading ziti-router config
panic: unable to load identity (open /persistent/ziti-edge-router.key: no such file or directory)

goroutine 1 [running]:
github.com/openziti/ziti/ziti/router.run(0xc000b67b00?, {0xc000c3d630, 0x1, 0x1?})
	github.com/openziti/ziti/ziti/router/run.go:67 +0x138e
github.com/spf13/cobra.(*Command).execute(0xc000b67b00, {0xc000c3d600, 0x1, 0x1})
	github.com/spf13/cobra@v1.7.0/command.go:944 +0x847
github.com/spf13/cobra.(*Command).ExecuteC(0x5281c60)
	github.com/spf13/cobra@v1.7.0/command.go:1068 +0x3bd
github.com/spf13/cobra.(*Command).Execute(...)
	github.com/spf13/cobra@v1.7.0/command.go:992
github.com/openziti/ziti/ziti/cmd.Execute()
	github.com/openziti/ziti/ziti/cmd/cmd.go:79 +0x25
main.main()
	github.com/openziti/ziti/ziti/main.go:51 +0x17

Already tried to docker system prune -a -f, still not working :confused:

2

This looks like you’re experiencing a bug I recently fixed. Can you confirm you’ve run docker compose pull to get the latest?

3

Yeah, I have the same problem, I’m pruning everything,even images before running the compose

4

Is it always the same router(s)? Can you provide the output from docker inspect <container-name> of a bad router container?

5

Yup all routers are down. The only working containers are these :

0563d44532b1   openziti/zac                 "docker-entrypoint.s…"    About a minute ago   Up 58 seconds                 1408/tcp, 0.0.0.0:8443->8443/tcp, :::8443->8443/tcp                                    openziti-ziti-console-1
b875524699b1   openziti/quickstart:latest   "/var/openziti/scrip…"    About a minute ago   Up About a minute             0.0.0.0:1280->1280/tcp, :::1280->1280/tcp, 0.0.0.0:6262->6262/tcp, :::6262->6262/tcp   openziti-ziti-controller-1
2e952235fc67   crccheck/hello-world         "/bin/sh -c 'echo \"h…"   About a minute ago   Up About a minute (healthy)   0.0.0.0:80->8000/tcp, :::80->8000/tcp

Inspect of openziti-ziti-edge-router-1 :

[
    {
        "Id": "dd0db808a4a5ad482e0f444677b7b1b96d076dbb558f15b832f0ce7a51a81289",
        "Created": "2023-08-19T09:31:41.315759717Z",
        "Path": "/bin/bash",
        "Args": [
            "/var/openziti/scripts/run-router.sh",
            "edge"
        ],
        "State": {
            "Status": "exited",
            "Running": false,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 0,
            "ExitCode": 2,
            "Error": "",
            "StartedAt": "2023-08-19T09:31:50.212365116Z",
            "FinishedAt": "2023-08-19T09:31:53.914696287Z"
        },
        "Image": "sha256:643987d97ca1f34e86ec4a51444e098b2487fd49d47bb2d27ac1bc87bf67786a",
        "ResolvConfPath": "/var/lib/docker/containers/dd0db808a4a5ad482e0f444677b7b1b96d076dbb558f15b832f0ce7a51a81289/resolv.conf",
        "HostnamePath": "/var/lib/docker/containers/dd0db808a4a5ad482e0f444677b7b1b96d076dbb558f15b832f0ce7a51a81289/hostname",
        "HostsPath": "/var/lib/docker/containers/dd0db808a4a5ad482e0f444677b7b1b96d076dbb558f15b832f0ce7a51a81289/hosts",
        "LogPath": "/var/lib/docker/containers/dd0db808a4a5ad482e0f444677b7b1b96d076dbb558f15b832f0ce7a51a81289/dd0db808a4a5ad482e0f444677b7b1b96d076dbb558f15b832f0ce7a51a81289-json.log",
        "Name": "/openziti-ziti-edge-router-1",
        "RestartCount": 0,
        "Driver": "overlay2",
        "Platform": "linux",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "docker-default",
        "ExecIDs": null,
        "HostConfig": {
            "Binds": [],
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "json-file",
                "Config": {}
            },
            "NetworkMode": "openziti_zitired",
            "PortBindings": {
                "10080/tcp": [
                    {
                        "HostIp": "",
                        "HostPort": "10080"
                    }
                ],
                "3022/tcp": [
                    {
                        "HostIp": "",
                        "HostPort": "3022"
                    }
                ]
            },
            "RestartPolicy": {
                "Name": "",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": null,
            "CapAdd": null,
            "CapDrop": null,
            "CgroupnsMode": "host",
            "Dns": null,
            "DnsOptions": null,
            "DnsSearch": null,
            "ExtraHosts": [],
            "GroupAdd": null,
            "IpcMode": "private",
            "Cgroup": "",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "",
            "Privileged": false,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": null,
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 67108864,
            "Runtime": "runc",
            "ConsoleSize": [
                0,
                0
            ],
            "Isolation": "",
            "CpuShares": 0,
            "Memory": 0,
            "NanoCpus": 0,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": null,
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": null,
            "DeviceCgroupRules": null,
            "DeviceRequests": null,
            "KernelMemory": 0,
            "KernelMemoryTCP": 0,
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": null,
            "OomKillDisable": false,
            "PidsLimit": null,
            "Ulimits": null,
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0,
            "Mounts": [
                {
                    "Type": "volume",
                    "Source": "openziti_ziti-fs",
                    "Target": "/persistent",
                    "VolumeOptions": {}
                }
            ],
            "MaskedPaths": [
                "/proc/asound",
                "/proc/acpi",
                "/proc/kcore",
                "/proc/keys",
                "/proc/latency_stats",
                "/proc/timer_list",
                "/proc/timer_stats",
                "/proc/sched_debug",
                "/proc/scsi",
                "/sys/firmware"
            ],
            "ReadonlyPaths": [
                "/proc/bus",
                "/proc/fs",
                "/proc/irq",
                "/proc/sys",
                "/proc/sysrq-trigger"
            ]
        },
        "GraphDriver": {
            "Data": {
                "LowerDir": "/var/lib/docker/overlay2/c0a3a5dc83257cac1b9f97c307dcd8df1c0dc3e942feda29af45056a67eb811e-init/diff:/var/lib/docker/overlay2/b4932eaff0da690a0f1f3343a7a4cd168b45293a5f787d656e6f3076981564bb/diff:/var/lib/docker/overlay2/01e33e296f970dd40da93b1793b4fb06f569552c1636738afd4289b1823feea0/diff:/var/lib/docker/overlay2/20ac4eac44e82a4c5d9489dbed49836cef586c019fc630206bd85766535eb7be/diff:/var/lib/docker/overlay2/38f2adb3b4641464b587e778cac7560eec89763f2264f8969890596d72471319/diff:/var/lib/docker/overlay2/0315f7a08f2b6b7ae9f1af008fba0b4e76ff820653199dcdc20e8191c175d4c5/diff:/var/lib/docker/overlay2/0c48593550ec4fd053efc25ea4e407c64ad8dcd96476625858a34105851b3f4e/diff:/var/lib/docker/overlay2/27030219445bdf9414b7810acddbd3e9f3cbdc466793ba3fabd7e9f8828e8154/diff:/var/lib/docker/overlay2/33d645733e56c59b93a7a54c2a1847336721b0dfa4b116901b59a8df095b0818/diff:/var/lib/docker/overlay2/78bb69a9e2d5a249275eaf8d5052d73fa7045dc4600e187b9426109a3dc1394b/diff:/var/lib/docker/overlay2/ad7434da844100e4c12c2913d6e43e229d246adb7d293c396c9856dc7869d9a4/diff:/var/lib/docker/overlay2/89cdfcbfe6e069c2f877c27d99c32c476aa4efa2ca945295aeec06e76ef72fff/diff:/var/lib/docker/overlay2/f0a26129b18d170402a37a36bea72ce582fc198eaa612a33e38ffc593c612478/diff",
                "MergedDir": "/var/lib/docker/overlay2/c0a3a5dc83257cac1b9f97c307dcd8df1c0dc3e942feda29af45056a67eb811e/merged",
                "UpperDir": "/var/lib/docker/overlay2/c0a3a5dc83257cac1b9f97c307dcd8df1c0dc3e942feda29af45056a67eb811e/diff",
                "WorkDir": "/var/lib/docker/overlay2/c0a3a5dc83257cac1b9f97c307dcd8df1c0dc3e942feda29af45056a67eb811e/work"
            },
            "Name": "overlay2"
        },
        "Mounts": [
            {
                "Type": "volume",
                "Name": "openziti_ziti-fs",
                "Source": "/var/lib/docker/volumes/openziti_ziti-fs/_data",
                "Destination": "/persistent",
                "Driver": "local",
                "Mode": "z",
                "RW": true,
                "Propagation": ""
            }
        ],
        "Config": {
            "Hostname": "dd0db808a4a5",
            "Domainname": "",
            "User": "ziti",
            "AttachStdin": false,
            "AttachStdout": true,
            "AttachStderr": true,
            "ExposedPorts": {
                "10080/tcp": {},
                "3022/tcp": {}
            },
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "ZITI_CTRL_ADVERTISED_PORT=6262",
                "ZITI_CTRL_NAME=ziti-controller",
                "ZITI_ROUTER_ROLES=public",
                "ZITI_USER=admin",
                "ZITI_ROUTER_ENROLLMENT_DURATION=10080",
                "ZITI_ROUTER_LISTENER_BIND_PORT=10080",
                "ZITI_ROUTER_NAME=ziti-edge-router",
                "ZITI_ROUTER_ADVERTISED_ADDRESS=ziti-edge-router",
                "ZITI_CTRL_EDGE_ADVERTISED_PORT=1280",
                "ZITI_EDGE_IDENTITY_ENROLLMENT_DURATION=10080",
                "ZITI_PWD=",
                "ZITI_IMAGE=openziti/quickstart",
                "ZITI_ROUTER_PORT=3022",
                "ZITI_VERSION=latest",
                "ZITI_CTRL_ADVERTISED_ADDRESS=ziti-controller",
                "ZITI_CTRL_EDGE_ADVERTISED_ADDRESS=ziti-edge-controller",
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "ZITI_HOME=/persistent",
                "ZITI_SHARED=/persistent/shared",
                "ZITI_ENV_FILE=/persistent/ziti.env",
                "ZITI_NETWORK=ziti",
                "ZITI_BIN_DIR=/var/openziti/ziti-bin",
                "ZITI_BIN_ROOT=/var/openziti/ziti-bin",
                "ZITI_SCRIPTS=/var/openziti/scripts"
            ],
            "Cmd": [
                "/var/openziti/scripts/run-router.sh",
                "edge"
            ],
            "Image": "openziti/quickstart:latest",
            "Volumes": {
                "/persistent": {}
            },
            "WorkingDir": "/persistent",
            "Entrypoint": [
                "/bin/bash"
            ],
            "OnBuild": null,
            "Labels": {
                "com.docker.compose.config-hash": "cfb2af0b12205961ec9a51b197558e53b80b125b5d4c763f778845133b1396e3",
                "com.docker.compose.container-number": "1",
                "com.docker.compose.depends_on": "ziti-controller:service_started",
                "com.docker.compose.oneoff": "False",
                "com.docker.compose.project": "openziti",
                "com.docker.compose.project.config_files": "/home/ares/dev/docker/openziti/docker-compose.yaml",
                "com.docker.compose.project.working_dir": "/home/ares/dev/docker/openziti",
                "com.docker.compose.service": "ziti-edge-router",
                "com.docker.compose.version": "2.12.2",
                "org.opencontainers.image.ref.name": "ubuntu",
                "org.opencontainers.image.version": "23.04"
            }
        },
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "009a96947b48dab6a79141ffb96673dabba2f407f0afebfbb1315a98b3b0eef9",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {},
            "SandboxKey": "/var/run/docker/netns/009a96947b48",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "",
            "Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "MacAddress": "",
            "Networks": {
                "openziti_zitiblue": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": [
                        "openziti-ziti-edge-router-1",
                        "ziti-edge-router",
                        "dd0db808a4a5"
                    ],
                    "NetworkID": "f6e6a4f18a66e0fcea044e3a3701f662bfb3ac76a3d3f2b1b2a443e5acd79164",
                    "EndpointID": "",
                    "Gateway": "",
                    "IPAddress": "",
                    "IPPrefixLen": 0,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "",
                    "DriverOpts": null
                },
                "openziti_zitired": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": [
                        "openziti-ziti-edge-router-1",
                        "ziti-edge-router",
                        "dd0db808a4a5"
                    ],
                    "NetworkID": "fbe53296f722fcb86100db71eefe56043be726602d6e7fe249dcae03b5d2d15a",
                    "EndpointID": "",
                    "Gateway": "",
                    "IPAddress": "",
                    "IPPrefixLen": 0,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "",
                    "DriverOpts": null
                }
            }
        }
    }
]

Logs of the same router:

_ZITI_ROUTER_NAME set to: ziti-edge-router
NOT OVERRIDING: env var ZITI_BIN_DIR already set. using existing value
NOT OVERRIDING: env var ZITI_BIN_ROOT already set. using existing value
NOT OVERRIDING: env var ZITI_CTRL_ADVERTISED_ADDRESS already set. using existing value
NOT OVERRIDING: env var ZITI_CTRL_ADVERTISED_PORT already set. using existing value
NOT OVERRIDING: env var ZITI_CTRL_EDGE_ADVERTISED_ADDRESS already set. using existing value
NOT OVERRIDING: env var ZITI_CTRL_EDGE_ADVERTISED_PORT already set. using existing value
NOT OVERRIDING: env var ZITI_CTRL_NAME already set. using existing value
NOT OVERRIDING: env var ZITI_EDGE_IDENTITY_ENROLLMENT_DURATION already set. using existing value
NOT OVERRIDING: env var ZITI_ENV_FILE already set. using existing value
NOT OVERRIDING: env var ZITI_HOME already set. using existing value
NOT OVERRIDING: env var ZITI_IMAGE already set. using existing value
NOT OVERRIDING: env var ZITI_NETWORK already set. using existing value
NOT OVERRIDING: env var ZITI_ROUTER_ENROLLMENT_DURATION already set. using existing value
NOT OVERRIDING: env var ZITI_ROUTER_LISTENER_BIND_PORT already set. using existing value
NOT OVERRIDING: env var ZITI_ROUTER_NAME already set. using existing value
NOT OVERRIDING: env var ZITI_ROUTER_PORT already set. using existing value
NOT OVERRIDING: env var ZITI_SCRIPTS already set. using existing value
NOT OVERRIDING: env var ZITI_SHARED already set. using existing value
NOT OVERRIDING: env var ZITI_USER already set. using existing value
NOT OVERRIDING: env var ZITI_VERSION already set. using existing value
 
adding /var/openziti/ziti-bin to the path
ZITI_ROUTER_NAME set to: ziti-edge-router
system has not been initialized. initializing...
Untrusted certificate authority retrieved from server
Verified that server supplied certificates are trusted by server
Server supplied 5 certificates
Server certificate chain written to /home/ziti/.config/ziti/certs/ziti-edge-controller
error: unable to authenticate to https://ziti-edge-controller:1280/edge/management/v1. Status code: 401 Unauthorized, Server returned: {
    "error": {
        "code": "INVALID_AUTH",
        "message": "The authentication request failed",
        "requestId": "UihGco5U5"
    },
    "meta": {
        "apiEnrollmentVersion": "0.0.1",
        "apiVersion": "0.0.1"
    }
}
----------  Creating edge-router ziti-edge-router....
CREATING EDGE ROUTER CONFIG: ziti-edge-router
This will overwrite the existing file, continue? (y/N) public router configuration file written to: /persistent/ziti-edge-router.yaml
error: no identity 'default' found in cli config /home/ziti/.config/ziti/ziti-cli.json
error: no identity 'default' found in cli config /home/ziti/.config/ziti/ziti-cli.json
---------- Enrolling edge-router ziti-edge-router....
[   0.247]   PANIC ziti/ziti/router.enrollGw: could not load JWT file from path [/persistent/ziti-edge-router.jwt]
panic: (*logrus.Entry) 0xc0000ff180

goroutine 1 [running]:
github.com/sirupsen/logrus.(*Entry).log(0xc0009d1ce0, 0x0, {0xc0009aff40, 0x44})
	github.com/sirupsen/logrus@v1.9.3/entry.go:260 +0x4d6
github.com/sirupsen/logrus.(*Entry).Log(0xc0009d1ce0, 0x0, {0xc0009d1bd0?, 0x1?, 0x1?})
	github.com/sirupsen/logrus@v1.9.3/entry.go:304 +0x4f
github.com/sirupsen/logrus.(*Entry).Logf(0xc0009d1ce0, 0x0, {0x3748f62?, 0x0?}, {0xc0009d1c80?, 0x0?, 0x0?})
	github.com/sirupsen/logrus@v1.9.3/entry.go:349 +0x85
github.com/sirupsen/logrus.(*Entry).Panicf(...)
	github.com/sirupsen/logrus@v1.9.3/entry.go:387
github.com/openziti/ziti/ziti/router.enrollGw(0xc000bd2000?, {0xc000cfdfb0, 0x1, 0x3?})
	github.com/openziti/ziti/ziti/router/enrollgw.go:64 +0x2a5
github.com/spf13/cobra.(*Command).execute(0xc000bd2000, {0xc000cfdf50, 0x3, 0x3})
	github.com/spf13/cobra@v1.7.0/command.go:944 +0x847
github.com/spf13/cobra.(*Command).ExecuteC(0x5281c60)
	github.com/spf13/cobra@v1.7.0/command.go:1068 +0x3bd
github.com/spf13/cobra.(*Command).Execute(...)
	github.com/spf13/cobra@v1.7.0/command.go:992
github.com/openziti/ziti/ziti/cmd.Execute()
	github.com/openziti/ziti/ziti/cmd/cmd.go:79 +0x25
main.main()
	github.com/openziti/ziti/ziti/main.go:51 +0x17

system initialized. writing marker file
[   0.000]   ERROR ziti/ziti/router.run: {revision=[2adab4fc9127] configFile=[/persistent/ziti-edge-router.yaml] os=[linux] arch=[amd64] version=[v0.30.0] error=[unable to load identity (open /persistent/ziti-edge-router.key: no such file or directory)] go-version=[go1.20.7] build-date=[2023-08-10T17:19:11Z]} error loading ziti-router config
panic: unable to load identity (open /persistent/ziti-edge-router.key: no such file or directory)

goroutine 1 [running]:
github.com/openziti/ziti/ziti/router.run(0xc000b1fb00?, {0xc000bef7c0, 0x1, 0x1?})
	github.com/openziti/ziti/ziti/router/run.go:67 +0x138e
github.com/spf13/cobra.(*Command).execute(0xc000b1fb00, {0xc000bef790, 0x1, 0x1})
	github.com/spf13/cobra@v1.7.0/command.go:944 +0x847
github.com/spf13/cobra.(*Command).ExecuteC(0x5281c60)
	github.com/spf13/cobra@v1.7.0/command.go:1068 +0x3bd
github.com/spf13/cobra.(*Command).Execute(...)
	github.com/spf13/cobra@v1.7.0/command.go:992
github.com/openziti/ziti/ziti/cmd.Execute()
	github.com/openziti/ziti/ziti/cmd/cmd.go:79 +0x25
main.main()
	github.com/openziti/ziti/ziti/main.go:51 +0x17
6

@arslane another dev found another race condition in the docker quickstart and is working on the fix. It causes the env file to be read possibly when it's not fully populated therefore leaving some environment values blank or unset entirely.

I'll update you when the fix is pushed out and a new image is built.

7

Hello @gberl002,
I believe I may be experiencing a similar issue. Has a fix been pushed out yet?

8

Hi @phenomenomenom. Welcome to the community and to OpenZiti. I've not seen this particular problem myself yet.

@gberl002 can you comment more? Who's working this? I see this is from Aug 2023! nevermind @gberl002 lol

@phenomenomenom what problem are you seeing? can you make a new topic?

9

Absolutely! I think a new topic is appropriate. I'll gather the appropriate info and get this up in a day or so.

In short, I'm unable to complete the initialisation of a new docker-compose instance. The following error occurs:

[   1.633]   FATAL ziti/controller/subcmd.NewEdgeInitializeCmd.func2: already initialized: Ziti Edge default admin already defined
  --- There was an error while initializing the controller ---

This happens both when using defaults (commenting out the admin user and password) or when specifying those details.

10

I tried it myself and it failed for me. I think we must have introduced a problem. @gberl002 can you take a look please?

11

@phenomenomenom I have been unable to reproduce this issue. I'll get in touch with some others to see if I can work on this.

12

@phenomenomenom The only possibilities we could come up with to replicate this was to remove the access-control.init file and try running the docker environment again. This would cause the controller to try and initialize when it was already initialized once before.

So, we're thinking, either this file failed to write for some reason or it was somehow deleted.

With that being the current assumption, could you try running one of the following depending on whether you are running the simplified or non-simplified compose?

Simplified

docker exec -it simp-ziti-controller-1 ls access-control.init

Non-Simplified

docker exec -it ziti-controller ls access-control.init

13

@gberl002 , a quick update. I'm having trouble myself replicating the issues I had yesterday.

I suspect that healthcheck was causing at least some of my issues. I'm running on a low power system and, the ziti-controller doesn't finish launching before the other containers attempt to start with the following requirement:

    depends_on:
      ziti-controller:
        condition: service_healthy

Simply waiting a while and re-running docker-compose up -d fixes things for me. Or, I'm sure the healthcheck timeout can be increased.

Once it's up, inspecting the logs reveals nothing of interest. No fatal errors are to be found. Why this is different to yesterday (I also waited and re-ran compose up then), I have no idea.

However, following your guidance and trying to list access-control.init returns No such file or directory. I exec'd in to inspect:

ziti@29f18b82f186:/persistent$ ls -a
.  ..  db  pki  scripts  ziti-controller.yaml  ziti.env

I cleaned up and did a control test on a different system (Oracle). My local system is a Synology, if it matters.

The Oracle based instance started up with no issues whatsoever, despite also having the default admin already defined issue yesterday. I changed nothing except a delete and re-upload of the docker-compose and .env, and a volume prune. Colour me confused.

Exec'ing in reveals

ziti@f5af9e74cbcf:/persistent$ ls
access-control.init  pki      ziti-controller.yaml  ziti-edge-router.cert  ziti-edge-router.key  ziti-edge-router.server.chain.cert  ziti.env
db                   scripts  ziti-edge-router.cas  ziti-edge-router.jwt   ziti-edge-router.log  ziti-edge-router.yaml

which looks better.

My apologies if this is a user error on my part. Unfortunately, I won't have time to dive back into this for a couple of days but I will give another update when I can.

Thanks for your support.