Docker-Compose Quickstart Setup - Edge Controller Issue

Hi All

I am trying to use the docker-compose quick-start to create my first network but have been waiting for the container to start for more than an hour. The terminal output continuously shows the below:

openziti-ziti-controller-init-container-1  | waiting for https://ziti-edge-controller:1280
openziti-ziti-edge-router-wss-1            | waiting for https://ziti-edge-controller:1280
openziti-ziti-private-blue-1               | waiting for https://ziti-edge-controller:1280
openziti-ziti-edge-router-1                | waiting for https://ziti-edge-controller:1280
openziti-ziti-fabric-router-br-1           | waiting for https://ziti-edge-controller:1280
openziti-ziti-edge-router-1                | waiting for https://ziti-edge-controller:1280
openziti-ziti-private-blue-1              | waiting for https://ziti-edge-controller:1280
openziti-ziti-private-red-1                | waiting for https://ziti-edge-controller:1280
openziti-ziti-edge-router-wss-1       | waiting for https://ziti-edge-controller:1280

have I missed something in the steps? Thanks for any help you can provide.

Hi @anarchomoh, welcome to OpenZiti! Nice to see you on the forums and thanks for the post.

That looks to me like the controller didnโ€™t start. I donโ€™t see it in the list there. Can you run

docker logs openziti-ziti-controller

(assuming that was the container name, replace the name of the container if thatโ€™s not correct but it looks like itโ€™ll be correct)

Can you share what you did to start the docker environment? Did you curl/wget down the docker-compose.yml and .env files and then use docker-compose -p openziti up?

The total time to start should be about 30 seconds to a minute depending on how fast your machine is.

Also can you tell me what OS/docker version youโ€™re using? Itโ€™s probably a situation we havenโ€™t run into yet.

Thanks!

Hi @TheLumberjack. No worries.

I ran the command and the output was:

Error: No such container: openziti-ziti-controller

I did curl the compose.yml and the .env file. I used the standard docker-compose up command. I am using Docker version 20.10.12, build tag v20.10.12 and my OS is Void Linux (no systemd).

can you try: docker ps -a | grep ziti-controller | grep -v init and find the right one. I assume you only have a single one (i have two shown below)

docker ps -a | grep ziti-controller | grep -v init
296982f26ac5   openziti/quickstart:latest   "/openziti/scripts/rโ€ฆ"    21 minutes ago   Exited (137) About a minute ago             openziti_ziti-controller_1
8e53367d54b7   openziti/quickstart:latest   "/openziti/scripts/rโ€ฆ"    5 days ago       Exited (137) 21 minutes ago                 docker_ziti-controller_1

Then run a docker logs __put_right_container_name_here

Thanks. The output was:
d4a561d517a3 openziti/quickstart:latest "/openziti/scripts/rโ€ฆ" 39 seconds ago Exited (2) 36 seconds ago openziti-ziti-controller-1

I attached the logs command output.

woah a segfault is REALLY strangeโ€ฆ iโ€™ve literally never seen that myself when trying to run the ziti pki.

You didnโ€™t modify the docker compose file at all, right? Can you try just running the โ€˜rawโ€™ container with:

docker run --rm -it openziti/quickstart bash

and then once in that container type:

ziti pki create ca whatever_this_doesnot_matter

image

You have some kind of strange โ€˜permissionโ€™ issue going on in there. I might need help from someone with a deeper understanding of linux to figure this one outโ€ฆ run that command and follow up please.

I donโ€™t use โ€œVoid Linuxโ€ but do you know what version of GLIBC it uses? This might be a tough oneโ€ฆ

Thanks, no I did not modify the compose file or the .env at all. I have attached the output.

The glibc version is 2.32_2

There are other โ€œunknown pcโ€ posts out there. "Exception 0xc0000005", "runtime: unknown pc" on program exit with loaded windows dll ยท Issue #42593 ยท golang/go ยท GitHub am wondering if this is somehow a go issueโ€ฆ Right now - I donโ€™t know what the issue is. weโ€™ll keep digging.

and a random reference here runtime/cgo: pthread_create failed: Operation not permitted ยท Issue #1501 ยท containers/skopeo ยท GitHub to try --security-opt seccomp=unconfined โ€ฆ

What happens when you run:

docker run --security-opt seccomp=unconfined --rm -it openziti/quickstart bash

and try ziti pki create ca whatever_this_doesnot_matter?

Thanks. I have attached the output:

Well that is quite promising. It would seem that Void Linux has some security settings that docker doesnโ€™t care for out of the boxโ€ฆ

Now I want you to modify the docker-compose.yml file. can you add this:

    security_opt:
      - seccomp:unconfined

to the ziti controller and see what happens? It should look like this in the file:

version: '2.4'
services:
  ziti-controller:
    image: "${ZITI_IMAGE}:${ZITI_VERSION}"
    security_opt:
      - seccomp:unconfined
    env_file:
      - ./.env
    ports:
      - "1280:1280"
    networks:
      zitiblue:
        aliases:
          - ziti-edge-controller
      zitired:
        aliases:
          - ziti-edge-controller
    volumes:
      - ziti-fs:/openziti
    entrypoint:
      - "/openziti/scripts/run-controller.sh"

then try a docker-compose up on that fileโ€ฆ and donโ€™t let it run for an hour if itโ€™s not running :slight_smile:

Thanks again. I modified the compose file but the result is the same:

openziti-ziti-controller-init-container-1  | waiting for https://ziti-edge-controller:1280
openziti-ziti-edge-router-wss-1            | waiting for https://ziti-edge-controller:1280
openziti-ziti-private-red-1                | waiting for https://ziti-edge-controller:1280
openziti-ziti-private-blue-1               | waiting for https://ziti-edge-controller:1280
openziti-ziti-edge-router-1                | waiting for https://ziti-edge-controller:1280
openziti-ziti-controller-init-container-1  | waiting for https://ziti-edge-controller:1280
openziti-ziti-fabric-router-br-1           | waiting for https://ziti-edge-controller:1280
openziti-ziti-edge-router-wss-1            | waiting for https://ziti-edge-controller:1280

Ok. I donโ€™t want to keep asking you to run commands. Iโ€™ll get with one of the people here who knows linux a bit better. They might know how to fix this immediately. They wonโ€™t be online for about 8+ hours so - weโ€™ll follow up in a bit.

Thanks! Weโ€™ll get it working, Iโ€™m sure of it! :smiley:

1 Like

Hello @anarchomoh

So, you can get this going. @TheLumberjack was correct - you need to add the

    security_opt:
      - seccomp:unconfined

against every service in the docker-compose file. Only doing the one server as @TheLumberjack has it will mean that the controller will start up, but the other components will fail with the same error. I have installed Void Linux and tested, and get to the first parts of the docker-compose example.

Step 1 is to ensure that you machine has enough memory. I installed within Virtual box and 1G, 1 Proc did not cut it. I went to dual proc, 3G and had success. For sake of simplicity, here is the docker compose file in full:

[root@void openziti]# cat docker-compose.yaml 
version: '2.4'
services:
  ziti-controller:
    image: "${ZITI_IMAGE}:${ZITI_VERSION}"
    env_file:
      - ./.env
    ports:
      - "1280:1280"
    networks:
      zitiblue:
        aliases:
          - ziti-edge-controller
      zitired:
        aliases:
          - ziti-edge-controller
    volumes:
      - ziti-fs:/openziti
    security_opt:
      - seccomp:unconfined
    entrypoint:
      - "/openziti/scripts/run-controller.sh"

  ziti-controller-init-container:
    image: "${ZITI_IMAGE}:${ZITI_VERSION}"
    depends_on:
      - ziti-controller
    environment:
      - ZITI_CONTROLLER_RAWNAME="${ZITI_CONTROLLER_RAWNAME}"
      - ZITI_EDGE_CONTROLLER_RAWNAME="${ZITI_EDGE_CONTROLLER_RAWNAME}"
    env_file:
      - ./.env
    networks:
      zitiblue:
        aliases:
          - ziti-edge-controller-init-container
      zitired:
        aliases:
          - ziti-edge-controller-init-container
    security_opt:
      - seccomp:unconfined

    volumes:
      - ziti-fs:/openziti
    entrypoint:
      - "/openziti/scripts/access-control.sh"

  ziti-edge-router:
    image: "${ZITI_IMAGE}:${ZITI_VERSION}"
    depends_on:
      - ziti-controller
    environment:
      - ZITI_CONTROLLER_RAWNAME="${ZITI_CONTROLLER_RAWNAME}"
      - ZITI_EDGE_CONTROLLER_RAWNAME="${ZITI_EDGE_CONTROLLER_RAWNAME}"
      - ZITI_EDGE_ROUTER_RAWNAME=ziti-edge-router
      - ZITI_EDGE_ROUTER_ROLES=public
    ports:
      - "3022:3022"
    security_opt:
      - seccomp:unconfined
    networks:
      - zitiblue
      - zitired
    volumes:
      - ziti-fs:/openziti
    entrypoint: /bin/bash
    command: "/openziti/scripts/run-router.sh edge"

  ziti-edge-router-wss:
    image: "${ZITI_IMAGE}:${ZITI_VERSION}"
    depends_on:
      - ziti-controller
    security_opt:
      - seccomp:unconfined
    environment:
      - ZITI_CONTROLLER_RAWNAME="${ZITI_CONTROLLER_RAWNAME}"
      - ZITI_EDGE_CONTROLLER_RAWNAME="${ZITI_EDGE_CONTROLLER_RAWNAME}"
      - ZITI_EDGE_ROUTER_RAWNAME=ziti-edge-router-wss
      - ZITI_EDGE_ROUTER_PORT=3023
      - ZITI_EDGE_ROUTER_ROLES=public
    ports:
      - "3023:3023"
    networks:
      - zitiblue
      - zitired
    volumes:
      - ziti-fs:/openziti
    entrypoint: /bin/bash
    command: "/openziti/scripts/run-router.sh wss"

  ziti-fabric-router-br:
    image: "${ZITI_IMAGE}:${ZITI_VERSION}"
    depends_on:
      - ziti-controller
    environment:
      - ZITI_CONTROLLER_RAWNAME="${ZITI_CONTROLLER_RAWNAME}"
      - ZITI_EDGE_CONTROLLER_RAWNAME="${ZITI_EDGE_CONTROLLER_RAWNAME}"
      - ZITI_EDGE_ROUTER_RAWNAME=ziti-fabric-router-br
    #ports:
    #  - 3024:3024
    security_opt:
      - seccomp:unconfined
    networks:
      - zitiblue
      - zitired
    volumes:
      - ziti-fs:/openziti
    entrypoint: /bin/bash
    command: "/openziti/scripts/run-router.sh fabric"

  ziti-private-red:
    image: "${ZITI_IMAGE}:${ZITI_VERSION}"
    depends_on:
      - ziti-controller
    environment:
      - ZITI_CONTROLLER_RAWNAME="${ZITI_CONTROLLER_RAWNAME}"
      - ZITI_EDGE_CONTROLLER_RAWNAME="${ZITI_EDGE_CONTROLLER_RAWNAME}"
      - ZITI_EDGE_ROUTER_RAWNAME=ziti-private-red
    #ports:
    #  - 3025:3025
    
    security_opt:
      - seccomp:unconfined
    networks:
      - zitired
    volumes:
      - ziti-fs:/openziti
    entrypoint: /bin/bash
    command: "/openziti/scripts/run-router.sh private"

  ziti-private-blue:
    image: "${ZITI_IMAGE}:${ZITI_VERSION}"
    depends_on:
      - ziti-controller
    environment:
      - ZITI_CONTROLLER_RAWNAME="${ZITI_CONTROLLER_RAWNAME}"
      - ZITI_EDGE_CONTROLLER_RAWNAME="${ZITI_EDGE_CONTROLLER_RAWNAME}"
      - ZITI_EDGE_ROUTER_RAWNAME=ziti-private-blue
    #ports:
    #  - 3026:3026
    security_opt:
      - seccomp:unconfined
    networks:
      - zitiblue
    volumes:
      - ziti-fs:/openziti
    entrypoint: /bin/bash
    command: "/openziti/scripts/run-router.sh private"

  web-test-blue:
    image: crccheck/hello-world
    ports:
      - "80:8000"
    security_opt:
      - seccomp:unconfined
    networks:
      zitiblue:
        aliases:
          - web-test-blue
          - web-test.blue
          - web.test.blue

  ziti-console:
    image: openziti/zac
    environment:
      - ZAC_SERVER_CERT_CHAIN=/openziti/pki/ziti-controller-intermediate/certs/ziti-controller-server.cert
      - ZAC_SERVER_KEY=/openziti/pki/ziti-controller-intermediate/keys/ziti-controller-server.key
    ports:
      - "1408:1408"
      - "8443:8443"
    working_dir: /usr/src/app
    volumes:
      - ziti-fs:/openziti
    security_opt:
      - seccomp:unconfined
    networks:
      - zitiblue
      - zitired

networks:
  zitired:
    driver: bridge
  zitiblue:
    driver: bridge
volumes:
  ziti-fs:

As can be seen, this is the stock standard docker-compose with the additional config files.

Before you start it all up, lets clear out your environment. I used docker-compose for everything, instead of using docker run, so if you have not installed docker-compose please do this (to follow along).

In the folder where the docker-compose file is, please run docker-compose down -v to clean the system. Then please run docker ps -a and ensure there are no stopped ziti containers hanging around. If there is, run docker-compose down again. This should clear it all out.

Now that is done, we should be good to go. Either type docker-compose up if you want to see all the logs scroll up, or docker-compose up -d to fork it into the background (can do docker logs -f <container name> to see a particular containers logs.

Wait a while (maybe a minute), then use docker exec -it *docker-ziti-controller-name-1* /bin/bash where docker-ziti-controller-name-1 is the name of the docker container, which can be found by docker ps | grep controller. Once there, you can continue with the tutorial. FWIW, here is the docker-exec from my console

[root@void kelvins]# docker exec -it openziti-ziti-controller-1 /bin/bash
 
adding /openziti/ziti-bin to the path
ziti@9bf1fc0c0bf8:/openziti$ zitiLogin
Token: 59260357-66b9-4157-8440-56af0abf0725
Saving identity 'default' to /openziti/ziti-cli.json
ziti@9bf1fc0c0bf8:/openziti$ ziti edge list identities
โ•ญโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ
โ”‚ ID         โ”‚ NAME                  โ”‚ TYPE   โ”‚ ATTRIBUTES โ”‚
โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค
โ”‚ .iq1lZDPVW โ”‚ ziti-edge-router-wss  โ”‚ Router โ”‚            โ”‚
โ”‚ KdU6lZ3QaW โ”‚ ziti-fabric-router-br โ”‚ Router โ”‚            โ”‚
โ”‚ Q1v1Oh3PaW โ”‚ ziti-private-blue     โ”‚ Router โ”‚            โ”‚
โ”‚ TcTvhVi2p  โ”‚ Default Admin         โ”‚ User   โ”‚            โ”‚
โ”‚ VTC1OZ3PV  โ”‚ ziti-edge-router      โ”‚ Router โ”‚            โ”‚
โ”‚ WYs1OhDQaW โ”‚ ziti-private-red      โ”‚ Router โ”‚            โ”‚
โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ
results: 1-6 of 6
ziti@9bf1fc0c0bf8:/openziti$ ziti edge list edge-routers
โ•ญโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ
โ”‚ ID         โ”‚ NAME                  โ”‚ ONLINE โ”‚ ALLOW TRANSIT โ”‚ COST โ”‚ ATTRIBUTES            โ”‚
โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค
โ”‚ .iq1lZDPVW โ”‚ ziti-edge-router-wss  โ”‚ true   โ”‚ true          โ”‚    0 โ”‚ public                โ”‚
โ”‚ KdU6lZ3QaW โ”‚ ziti-fabric-router-br โ”‚ true   โ”‚ true          โ”‚    0 โ”‚ ziti-fabric-router-br โ”‚
โ”‚ Q1v1Oh3PaW โ”‚ ziti-private-blue     โ”‚ true   โ”‚ true          โ”‚    0 โ”‚ ziti-private-blue     โ”‚
โ”‚ VTC1OZ3PV  โ”‚ ziti-edge-router      โ”‚ true   โ”‚ true          โ”‚    0 โ”‚ public                โ”‚
โ”‚ WYs1OhDQaW โ”‚ ziti-private-red      โ”‚ true   โ”‚ true          โ”‚    0 โ”‚ ziti-private-red      โ”‚
โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ

I have not gone past this test. Have fun, and lets know how you go.

3 Likes

Can you determine whether your Void Linux install is set up to use musl or GNU libc?

Also, please provide the output from docker --version. Apparently, older Docker doesnโ€™t allow the newer clone3 syscall thatโ€™s employed by cgo when spawning threads.

thanks @gooseleggs, that worked perfectly. I will continue the rest of the tutorial video and am excited to setup the network.

2 Likes

Hi @qrkourier, it is glibc and the docker version is: Docker version 20.10.12, build tag v20.10.12.

Hi Again

Iโ€™m trying to add ZAC using the instructions on docker-compose but getting the error: yaml: line 162: did not find expected key when i do docker-compose up -d. What did I do incorrectly?

Can you supply what is around line 162 and line 162 as well. Otherwise it is a bit hard to see what might be wrong.

Couple of gotchas that may or may not help. Donโ€™t use tabs to insert values unless it is done throughout the whole file. Always use spaces, and make sure that they line up correctly as indentation matters. That kind of error may be missing a : between name and variable. Some items requires quotes (") around the value, although I have used the code from that page recently and did not get a problem.

It sounds like there is some sort of indentation/formatting issue when the ziti-console service was copied and pasted the ziti-console service per the instructions.

Can you share your docker-compose source (the entire contents of the file would be preferable) and make sure to leave the formatting exactly as it is in your docker-compose file? This would be important in figuring out exactly what the formatting issue is.

In the meantime Iโ€™ll follow the instructions myself to see if I can replicate the issue as it may be some problem with the doc.