Troubleshooting slow response times of services through OpenZiti network

jruiz94 · April 18, 2023, 6:05pm

Hello, my setup is as follows, I’ll try to explain as best as I can the meaning of everything:

I have the classic setup ZitiEdgeController (with ZitiEdgeInitController) / ZitiEdgeRouter / ZitiConsole

My goal with this is to have a couple servers running Docker containers of various services and use OpenZiti as a way to easily and securely connect between them. Additionally, my plan is to use a Caddy instance on each server to handle HTTPS requests so I don’t get warnings about self signed certs and stuff like that.

In the above setup, the Holy Trinity of OpenZiti (Controller/Router/Console) is running in its own network, called ziti. Then I have another network called caddy where there is a simple whoami container (which is basically a simple Nginx that returns a static HTML) and a Caddy Reverse Proxy container. This caddy container is also connected to the ziti network. This is done this way so I can reverse proxy the ZitiConsole container too.

The Caddy container is setup in such a way that it gets certificates by doing DNS challenges (I hope I’m using the correct terminology) against Cloudflare. I have a domain my.domain.com and I’ve given Caddy a special token to perform these dns acme challenges so it can get certificates.

The Caddy container is exposing to the localhost the ports 80 and 443.

I also have 2 containers: Ziti Host and Ziti Tun. Ziti Host is being used to, well, host services. Since I have Caddy on this machine I’ve created a config to host a service called caddy in ports 80 and 443 and intercepts urls with the *.mymachine.my.domain.com. ZitiTun for now is not being used, but if I deploy a container that needs to contact another machine in my Ziti network I’ll need it

This is done so I can type https://ziti.mymachine.my.domain.com on my desktop, then the requests is captured by Ziti Edge Desktop app on Windows, redirected through MyMachine, it arrives to the ZitiHost container, it is then redirected to port 443 on localhost of MyMachine, it is captured by Caddy, Caddy then does its magic with the certificates stuff, and makes the redirection to the Ziti Console container.

I can also type https://whoami.mymachine.my.domain.com and the same will apply but Caddy will redirect it at the last step to the other container.

All of this is currently working nicely. I can access all services with HTTPS with no warnings whatsoever

However, when I reach, for example, https://ziti.mymachine.my.domain.com, the answer is painfully slow. I’m talking about 60 seconds to load the login page. Once that’s done it’s all cached and it works well but anything new is really slow.
I’ve also deployed other services to test (Syncthing for example) and it’s equally slow to access the UI.

So my question is, how can I debug and/or troubleshoot what might be causing this issue here? Perhaps it’s not even OpenZiti’s fault, but I don’t know what else could be?

I’ll leave my docker-compose.yml and .env files here in case that helps, but they might be a bit complex, idk.

Click here to see docker compose and .env

docker-compose.yml

# Sections prefixed by x- are not parsed by Docker Compose. This section is used to reuse common sections related to Caddy container (see below)
x-common-caddy: &common-caddy
  caddy.tls.protocols: "tls1.3" #### This is optional. Default it is tls1.2
  caddy.tls.dns: "cloudflare ${CLOUDFLARE_API_TOKEN}"

services:
  ziti-controller:
    image: "${ZITI_IMAGE}:${ZITI_VERSION}"
    ports:
      - ${ZITI_EDGE_CONTROLLER_PORT:-1280}:${ZITI_EDGE_CONTROLLER_PORT:-1280}
      - ${ZITI_CTRL_PORT:-6262}:${ZITI_CTRL_PORT:-6262}
    environment:
      - ZITI_EDGE_IDENTITY_ENROLLMENT_DURATION=${ZITI_EDGE_IDENTITY_ENROLLMENT_DURATION}
      - ZITI_EDGE_ROUTER_ENROLLMENT_DURATION=${ZITI_EDGE_ROUTER_ENROLLMENT_DURATION}
    env_file:
      - $MAIN_DIR/.env
    networks:
      ziti:
        aliases:
          - ziti-edge-controller
    volumes:
      - ziti-fs:/persistent
    entrypoint:
      - "/var/openziti/scripts/run-controller.sh"
    # Simple healthcheck to check open ports. This is just to make the startup a little bit more "step by step"
    healthcheck:
      test: ["CMD", "bash", "-c", "lsof -i -P -n | grep -q 'TCP.*:6262' && lsof -i -P -n | grep -q 'TCP.*:1280'"]
      interval: 10s
      timeout: 5s
      retries: 10

  ziti-controller-init-container:
    image: "${ZITI_IMAGE}:${ZITI_VERSION}"
    depends_on:
      ziti-controller:
        condition: service_healthy
    environment:
      - ZITI_CONTROLLER_RAWNAME="${ZITI_CONTROLLER_RAWNAME}"
      - ZITI_EDGE_CONTROLLER_RAWNAME="${ZITI_EDGE_CONTROLLER_RAWNAME}"
    env_file:
      - $MAIN_DIR/.env
    networks:
      ziti:
        aliases:
          - ziti-edge-controller-init-container
    volumes:
      - ziti-fs:/persistent
    entrypoint:
      - "/var/openziti/scripts/run-with-ziti-cli.sh"
    command:
      - "/var/openziti/scripts/access-control.sh"

  ziti-edge-router:
    image: "${ZITI_IMAGE}:${ZITI_VERSION}"
    depends_on:
      ziti-controller:
        condition: service_healthy
      ziti-controller-init-container:
        condition: service_completed_successfully
    environment:
      - ZITI_CONTROLLER_RAWNAME="${ZITI_CONTROLLER_RAWNAME}"
      - ZITI_EDGE_CONTROLLER_RAWNAME="${ZITI_EDGE_CONTROLLER_RAWNAME}"
      - ZITI_EDGE_ROUTER_RAWNAME=${ZITI_EDGE_ROUTER_RAWNAME:-ziti-edge-router}
      - ZITI_EDGE_ROUTER_ROLES=public
    env_file:
      - $MAIN_DIR/.env
    ports:
      - ${ZITI_EDGE_ROUTER_PORT:-3022}:${ZITI_EDGE_ROUTER_PORT:-3022}
    networks:
      - ziti
    volumes:
      - ziti-fs:/persistent
    entrypoint: /bin/bash
    command: "/var/openziti/scripts/run-router.sh edge"
    # Simple healthcheck to check open ports. This is just to make the startup a little bit more "step by step"
    healthcheck:
      test: ["CMD", "bash", "-c", "lsof -i -P -n | grep -q 'TCP.*:3022'"]
      interval: 10s
      timeout: 5s
      retries: 10

  ziti-console:
    image: openziti/zac:2.6.9
    depends_on:
      ziti-controller:
        condition: service_healthy
      ziti-controller-init-container:
        condition: service_completed_successfully
      ziti-edge-router:
        condition: service_healthy
    working_dir: /usr/src/app
    environment:
      - ZAC_SERVER_CERT_CHAIN=/persistent/pki/${ZITI_EDGE_CONTROLLER_HOSTNAME:-ziti-controller}-intermediate/certs/${ZITI_EDGE_CONTROLLER_HOSTNAME:-ziti-controller}-server.cert
      - ZAC_SERVER_KEY=/persistent/pki/${ZITI_EDGE_CONTROLLER_HOSTNAME:-ziti-controller}-intermediate/keys/${ZITI_EDGE_CONTROLLER_HOSTNAME:-ziti-controller}-server.key
      - PORTTLS=8443
    env_file:
      - $MAIN_DIR/.env
    # Labels are used by caddy container to know where are the upstreams and what url to use
    labels:
      <<: *common-caddy
      caddy: "ziti.${MACHINE_HOSTNAME}"
      caddy.reverse_proxy: "{{upstreams https 8443}}"
      caddy.reverse_proxy.transport: http
      # Tls insecure skip verify had to be used to prevent errors with https and caddy
      caddy.reverse_proxy.transport.tls_insecure_skip_verify:
    ports:
      # Port 8443 is not exposed, only through Caddy
      - 1408:1408
    networks:
      - ziti
    volumes:
      - ziti-fs:/persistent
  
  # This container will fail on first 'docker compose up -d' because you can't get the ZITI_ENROLL_TOKEN until you generate it
  # Once you have the enroll token, add it to .env then run again docker compose up -d
  ziti-host:
    image: openziti/ziti-host:0.21.0
    depends_on:
      ziti-controller:
        condition: service_healthy
      ziti-edge-router:
        condition: service_healthy
    restart: unless-stopped
    network_mode: "host"
    volumes:
        - ziti-identity:/ziti-edge-tunnel
    environment:
        - ZITI_IDENTITY_BASENAME=${ZITI_IDENTITY_BASENAME}
        - ZITI_ENROLL_TOKEN=${ZITI_ENROLL_TOKEN}
  
  # This container will fail on first 'docker compose up -d' because you can't get the ZITI_ENROLL_TOKEN until you generate it
  # Once you have the enroll token, add it to .env then run again docker compose up -d
  ziti-tun:
    image: openziti/ziti-edge-tunnel
    depends_on:
      ziti-controller:
        condition: service_healthy
      ziti-edge-router:
        condition: service_healthy
      ziti-host:
        condition: service_started
    restart: unless-stopped
    devices:
        - /dev/net/tun:/dev/net/tun
    volumes:
        - ziti-identity:/ziti-edge-tunnel
        - /var/run/dbus/system_bus_socket:/var/run/dbus/system_bus_socket
    environment:
        - ZITI_IDENTITY_BASENAME=${ZITI_IDENTITY_BASENAME}
        - PFXLOG_NO_JSON=true # suppress JSON logging
    network_mode: "host"
    privileged: true

  caddy:
    image: homeall/caddy-reverse-proxy-cloudflare:latest
    depends_on:
      ziti-host:
        condition: service_started
    restart: unless-stopped
    environment:
      TZ: 'Europe/Madrid'
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock"
      - caddy_data:/data
    ports:
      - 80:80
      - 443:443
    networks:
      caddy:
      ziti:
    labels:
      caddy.email: ${CLOUDFLARE_EMAIL}

  whoami:
    image: "nginxdemos/hello"
    depends_on:
      caddy:
        condition: service_started
    restart: unless-stopped
    networks:
      caddy:
    labels:
      <<: *common-caddy
      caddy: "whoami.${MACHINE_HOSTNAME}"
      caddy.reverse_proxy: "{{upstreams 80}}"

networks:
  ziti:
  caddy:

volumes:
  ziti-fs:
  caddy_data:
  ziti-identity:

.env

# Generic
MAIN_DIR=/home/ubuntu/yourdirectorywherethesefilesare
MACHINE_HOSTNAME=mymachine.mydomain.com
MACHINE_IP=1.2.3.4

# Caddy SSL
CLOUDFLARE_EMAIL=youremail@email.com
CLOUDFLARE_API_TOKEN=YOURCLOUDFLARETOKEN

# OpenZiti Variables
ZITI_IMAGE=openziti/quickstart
ZITI_VERSION=0.27.9

# The duration of the enrollment period (in minutes), default if not set
# shown - 7days
ZITI_EDGE_IDENTITY_ENROLLMENT_DURATION=10080
ZITI_EDGE_ROUTER_ENROLLMENT_DURATION=10080

# controller address/port information
ZITI_CONTROLLER_RAWNAME=ziti-controller
ZITI_CONTROLLER_HOSTNAME="${MACHINE_HOSTNAME}"
#ZITI_CTRL_PORT=8440

ZITI_EDGE_CONTROLLER_RAWNAME=ziti-edge-controller
ZITI_EDGE_CONTROLLER_HOSTNAME="${MACHINE_HOSTNAME}"
ZITI_EDGE_CONTROLLER_IP_OVERRIDE="${MACHINE_IP}"
#ZITI_EDGE_CONTROLLER_PORT=8441

# router address/port information
ZITI_EDGE_ROUTER_RAWNAME="${MACHINE_HOSTNAME}"
ZITI_EDGE_ROUTER_IP_OVERRIDE="${MACHINE_IP}"
#ZITI_EDGE_ROUTER_PORT=8442

# Ziti Host
ZITI_IDENTITY_BASENAME=mymachine
ZITI_ENROLL_TOKEN=

Thanks in advance!

TheLumberjack · April 18, 2023, 7:14pm

OUCH. That's obviously unacceptable and surprising. It's almost as though some timeout has to happen before traffic is flowing. Something is definitely strange.

The remote machine isn't hitting CPU file, or RAM bottlenecks right? When you access ZAC, do you do it entirely locally? Are you doing it through OpenZiti or directly? I assume via OpenZiti since port 8443 is not shown exposed? I've seen some cloud providers limit io/iops. Like on AWS, using a T-type instance, I've seen the cloud provider limit me and makes delays like this that I just can't explain.

If this were me, I would probably take parts out of the equation and bring it all the way back to the basics. I would put up the simple docker whale demo using crccheck/hello-world. I would make a service that takes https out of the equation and see if it's slow. Assuming it's NOT slow, that would rule OpenZiti out as the cause of the problem imo. After that I would layer on HTTPS and then add caddy back into the mix and see if any of that matters.

It's a really strange issue, tbh, I'm leaning towards throttling at the cloud level at the moment...

gooseleggs · April 18, 2023, 7:16pm

This looks like an interesting setup. Have you tried moving the whoami container to the ziti network and bypassing caddy altogether. Because you are using dns for certificates this is normally slower than http challenge response as it will sit there waiting for the dns to respond and depending upon the provider can take some time. That could explain the delay for the first connect if configured to do this on first connect. However not for the other connect delays.

Also I would use dev tools and see the waterfall of loading the page. It is only a simple page but it might give some insight.

TheLumberjack · April 18, 2023, 7:18pm

Oh that’s a good point. If caddy is resolving/reissuing/reverifying the DNS each time, that definitely makes sense since doing the DNS challenge requires updating DNS records which often have TTLs in the “60s” (or much much more) range.

That seems like a good thing to verify indeed

jruiz94 · April 18, 2023, 7:49pm

Nope just checked and it's like 600MB out of 6GB on RAM and around 10% CPU i think.

It's through OpenZiti, as I mentioned i'm using an intercept to "catch" the url https://ziti.mymachine.mydomain.com so that's routed to caddy and then caddy redirects to ZAC.

For the record I'm using Oracle cloud, those ARM based machines they recently have available.

Yeah I think I'm going to test that next, I'll try to set it up in such a way that I can revert to my current config easily, will update that when I have some time.

Well the whoami is actually fast, its only 215ms but I assumed its because it's just a single html page? I might be mistaken.

Noted, yeah it's weird because the only time I don't get these high times is when it's cached (after first request). If I force reload with CTRL + F5 then it's the same again.

BTW I didn't mention this because I thought it might not be related but since the DNS thing has come up in the conversation, I currently have another machine connected to my ziti network with a similar docker compose setup, in this case just ZitiHost/ZitiTun and a Caddy Container also that i was going to use to connect to another Syncthing UI.

I was using the default network for Caddy and Syncthing and Caddy couldn't get the certificates until I gave them their own network (I named it caddy again), and apparently it's related to DNS and the way docker compose handles this internally. I don't know if it's related but there are a plethora of issues with DNS and this Caddy proxy, it seems.

I've taken a couple quick screen captures for that. I'm not very webdev-savvy so I'm not fully confident on how to interpret the colours.

How can I verify if Caddy is reverifying DNS each time?

TheLumberjack · April 18, 2023, 7:59pm

If you hover over the bar, it'll tell you:

Looks like your request was waiting for a response for 40s... Which is an ETERNITY!!

Sadly, I don't know, that's a caddy type question. I did exactly this sort of thing but with nginx recently on a different discourse post... i used nginx to terminate TLS though and the overlay was not behaving like this is for you... I am interested in the "non-caddy-involved" path to see if that performance is acceptable.

You could test that easily by using the http port of your ZAC instead - generally 1408. That's a good test server to use -- probably better than the crccheck/hello-world since it's ZAC and a bit 'more' going on

jruiz94 · April 18, 2023, 8:10pm

Yeah I hovered on one of those
imagen
This one is the worst one, more than one minute to load hahaha

My bad for not reading the documentation, but you're telling me that I can access ZAC through http on port 1408 and skip https on port 8443? I think I'm going to try that, and also that would simplify my docker-compose.yml, i had a dirty skip_tls_verify around there.

Just to double check, what I should do is create a host config for tcp 1408 and 127.0.0.1 while exposing port 1408 on ZAC, and then an intercept config with whatever url i want to make and port 1408, right? (Then the service and bind and dial and that stuff, i know how to handle that so dw)

Btw the only reason I'm using caddy here is because it seemed like a super easy setup to get the automatic https certificates working, but if using Nginx or Traefik or whatever makes it a little bit more complex but at least works at acceptable speeds, I don't mind changing xD

TheLumberjack · April 18, 2023, 8:19pm

Yes. It's configurable but 1408 is the default. It should be exposed inside the docker container space already. You should see something like:

ZAC will use this key for TLS: /persistent/pki/ziti-controller-intermediate/keys/ziti-controller-server.key
ZAC will present this pem for TLS: /persistent/pki/ziti-controller-intermediate/certs/ziti-controller-server.cert
Loading Settings File From: /usr/src/app/../ziti/settings.json
TLS initialized on port: 8443
Ziti Admin Console is now listening on port 1408

just to double check, what I should do is create a host config for tcp 1408 and 127.0.0.1

Yep. You could also learn how to specifiy more than one port per service and then use "FORWARD PORT" on the config

Then you only have one config with two ports, but the way you're thinking is FINE too Just showing you a new trick here is all...

I’m using caddy here is because it seemed like a super easy setup to get the automatic https certificates working,

If you're not using it as an actual proxy, why just just use certbot? That's what I did for my DNS challenge... Then there's nothing else to install/run too? A cron to renew the cert is all you'd need?

jruiz94 · April 18, 2023, 9:06pm

Holy pepperonni I should read the docs... I didn't know that, thanks for pointing it out!

The only 2 things I want are:

The ability to route addresses like i've showcased before (ziti.mymachine.mydomain.com or syncthing.myothermachine.mydomain.com)
Using valid non-self signed certificates so me and future users don't get scary warnings showing up.

I know a bit of certbot, I think I made another post some time ago and we mentioned it and I managed to obtain a couple certificates, but I think there was some kind of issue with wildcard certificates? Which I kinda want for having that flexibility to get any subdomain certificated instead of having to issue manually certificates (I might be mistaken, forgive mi ignorance if that's the case)

I'll update when I have some time to test what we mentioned about doing direct http to ZAC through OpenZiti

TheLumberjack · April 18, 2023, 9:20pm

A while ago they wouldn't issue them but they do now. I made a wildcard cert for my self hosted zrok instance. If it turns out that caddy is somehow causing this issue somehow, I'd just swap it out for cerbot and nginx.

jruiz94 · April 19, 2023, 4:35pm

All right so I did the following commands

zitiLogin

ziti edge create config special.console.host host.v1 '{"protocol":"tcp","address":"127.0.0.1","port":1408}'

ziti edge create config special.console.intercept intercept.v1 '{"protocols":["tcp"],"addresses":["my.console.ziti"],"portRanges":[{"low":1408,"high":1408}]}'

ziti edge create service special.service --configs "special.console.host,special.console.intercept"

ziti edge create service-policy special.bind Bind --service-roles "@special.service" --identity-roles "#my.host"

ziti edge create service-policy special.dial Dial --service-roles "@special.service" --identity-roles "#special.my.clients"

I called them special because im bad at names and wanted to distinguish them from the other configs.

I can see my new service in my Ziti Edge Desktop UI in Windows, and when i get http://my.console.ziti:1408 in my browser, something weird happens.

So the times are pretty low, which is nice, but everything throws an error apparently

First 2 requests seem okay"

Then the rest are marked as error, with no apparent response whatsoever, but I think it’s weird that the url changed from http to https

However, it really does seem like it is a Caddy issue indeed, so I think it’s time to take a look at good old Nginx, see if I can reproduce this…

gooseleggs · April 19, 2023, 7:03pm

It will be ZAC that is doing the https prefix in the assets that it needs. Change your config to be 8443, the https port and should work. You haven’t exposed 8443 in the docker config and you don’t need too as on the same docker config. However to make it simple expose 8443 on the host like you have 1408 and change the above ports to 8443.

You will get a cert warning when going to the url but that is ok for testing ( I don’t bother about a valid SSL on the Zac as users don’t interact with it).

As for caddy, I am keen to have a play with this. Traditionally I have used nginx with nginx-dockergen and nginx-letsencrypt as a trio for container work - it listens for changes in containers and modifies the nginx config on the start/stop of containers and sorts certs out for me as well as renewals.

TheLumberjack · April 19, 2023, 7:07pm

something weird indeed! I must admit, I've never seen that particular issue just yet. I took your commands and I can reproduce the issue. I'll see if i can figure out what's going on there since that's definitely unexpected. @jeremy.tellier is probably not availalbe to troubleshoot for a bit so it might be a while to get this sorted...

My http zac:

TheLumberjack · April 19, 2023, 7:13pm

Yeah, it looks like if HTTPS is enabled, it triggers ZAC to redirect to the HTTPS port. That’s a bug in ZAC imo. I did not add 8443 to my.console.ziti and that’s why all those requests are failing…

I think we ruled ziti out in this situation though.

jruiz94 · April 20, 2023, 3:39pm

Yeah I also think that the bottom line of all this is that Ziti was not the issue so I'll have to check another reverse proxy to fix my super long response time problems

I'm not very expert with Caddy, I've mostly used Traefik both at home and at work but I kinda like the simplicity of Caddy at least in this variant where it uses the labels to do the config.

Before ditching it completely, I'm going to try something I just saw yesterday about generating wildcard certificates directly from Cloudflare's UI and paste them directly on a directory for Caddy to use them.

Otherwise if that doesn't work, I'll go back to Traefik or even Nginx, the good old reliable, hahaha.

I will update soon with what worked for me in the end!

jruiz94 · April 22, 2023, 11:23am

I’ve been these days doing the same that I had with Caddy but with Traefik, and the problem is still the same, I think the ACME DNS Challenge fails or takes too long to solve at some point but I don’t know why.

I wanted to test this ACME DNS Challenge with Traefik because I have this exact Traefik-ProxiedContainer setup on a public machine (to host a public webpage, no OpenZiti) and it works without issues.

I’ve simplified my docker compose in order to narrow down the issue but I still don’t know what might be.
Traefik has a dashboard so I’ve been using that to test instead of ZAC (it’s the same in the end, a UI to query to see the times to load). I’ve also removed for now ziti-tun and whoami, since i only need ziti-host and traefik

docker-compose.yml

services:
  ziti-controller:
    image: "${ZITI_IMAGE}:${ZITI_VERSION}"
    ports:
      - ${ZITI_EDGE_CONTROLLER_PORT:-1280}:${ZITI_EDGE_CONTROLLER_PORT:-1280}
      - ${ZITI_CTRL_PORT:-6262}:${ZITI_CTRL_PORT:-6262}
    environment:
      - ZITI_EDGE_IDENTITY_ENROLLMENT_DURATION=${ZITI_EDGE_IDENTITY_ENROLLMENT_DURATION}
      - ZITI_EDGE_ROUTER_ENROLLMENT_DURATION=${ZITI_EDGE_ROUTER_ENROLLMENT_DURATION}
    env_file:
      - $MAIN_DIR/.env
    networks:
      ziti:
        aliases:
          - ziti-edge-controller
    volumes:
      - ziti-fs:/persistent
    entrypoint:
      - "/var/openziti/scripts/run-controller.sh"
    healthcheck:
      test: ["CMD", "bash", "-c", "lsof -i -P -n | grep -q 'TCP.*:6262' && lsof -i -P -n | grep -q 'TCP.*:1280'"]
      interval: 10s
      timeout: 5s
      retries: 10

  ziti-controller-init-container:
    image: "${ZITI_IMAGE}:${ZITI_VERSION}"
    depends_on:
      ziti-controller:
        condition: service_healthy
    environment:
      - ZITI_CONTROLLER_RAWNAME="${ZITI_CONTROLLER_RAWNAME}"
      - ZITI_EDGE_CONTROLLER_RAWNAME="${ZITI_EDGE_CONTROLLER_RAWNAME}"
    env_file:
      - $MAIN_DIR/.env
    networks:
      ziti:
        aliases:
          - ziti-edge-controller-init-container
    volumes:
      - ziti-fs:/persistent
    entrypoint:
      - "/var/openziti/scripts/run-with-ziti-cli.sh"
    command:
      - "/var/openziti/scripts/access-control.sh"

  ziti-edge-router:
    image: "${ZITI_IMAGE}:${ZITI_VERSION}"
    depends_on:
      ziti-controller:
        condition: service_healthy
      ziti-controller-init-container:
        condition: service_completed_successfully
    environment:
      - ZITI_CONTROLLER_RAWNAME="${ZITI_CONTROLLER_RAWNAME}"
      - ZITI_EDGE_CONTROLLER_RAWNAME="${ZITI_EDGE_CONTROLLER_RAWNAME}"
      - ZITI_EDGE_ROUTER_RAWNAME=${ZITI_EDGE_ROUTER_RAWNAME:-ziti-edge-router}
      - ZITI_EDGE_ROUTER_ROLES=public
    env_file:
      - $MAIN_DIR/.env
    ports:
      - ${ZITI_EDGE_ROUTER_PORT:-3022}:${ZITI_EDGE_ROUTER_PORT:-3022}
    networks:
      - ziti
    volumes:
      - ziti-fs:/persistent
    entrypoint: /bin/bash
    command: "/var/openziti/scripts/run-router.sh edge"
    healthcheck:
      test: ["CMD", "bash", "-c", "lsof -i -P -n | grep -q 'TCP.*:3022'"]
      interval: 10s
      timeout: 5s
      retries: 10

  ziti-host:
    image: openziti/ziti-host:0.21.0
    depends_on:
      ziti-controller:
        condition: service_healthy
      ziti-edge-router:
        condition: service_healthy
    restart: unless-stopped
    network_mode: "host"
    volumes:
        - ziti-identity:/ziti-edge-tunnel
    environment:
        - ZITI_IDENTITY_BASENAME=${ZITI_IDENTITY_BASENAME}
        - ZITI_ENROLL_TOKEN=${ZITI_ENROLL_TOKEN}

  traefik:
    image: traefik:latest
    container_name: traefik
    restart: unless-stopped
    security_opt:
      - no-new-privileges:true
    depends_on:
      ziti-host:
        condition: service_started
      ziti-tun:
        condition: service_started
    networks:
      proxy:
    ports:
      - 80:80
      - 443:443
    environment:
      - CF_API_EMAIL=${CLOUDFLARE_EMAIL}
      - CF_DNS_API_TOKEN=${CLOUDFLARE_API_TOKEN}
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ${MAIN_DIR}/traefik/data/traefik.yml:/traefik.yml:ro
      - ${MAIN_DIR}/traefik/data/config.yml:/config.yml:ro
      - ${MAIN_DIR}/traefik/data/acme.json:/acme.json
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.traefik.entrypoints=http"
      - "traefik.http.routers.traefik.rule=Host(`traefik-dashboard.mymachine.mydomain.com`)"
      - "traefik.http.middlewares.traefik-auth.basicauth.users=${TRAEFIK_BASIC_AUTH}"
      - "traefik.http.middlewares.traefik-https-redirect.redirectscheme.scheme=https"
      - "traefik.http.middlewares.sslheader.headers.customrequestheaders.X-Forwarded-Proto=https"
      - "traefik.http.routers.traefik.middlewares=traefik-https-redirect"
      - "traefik.http.routers.traefik-secure.entrypoints=https"
      - "traefik.http.routers.traefik-secure.rule=Host(`traefik-dashboard.mymachine.mydomain.com`)"
      - "traefik.http.routers.traefik-secure.middlewares=traefik-auth"
      - "traefik.http.routers.traefik-secure.tls=true"
      - "traefik.http.routers.traefik-secure.tls.certresolver=cloudflare"
      - "traefik.http.routers.traefik-secure.tls.domains[0].main=mymachine.mydomain.com"
      - "traefik.http.routers.traefik-secure.tls.domains[0].sans=*.mymachine.mydomain.com"
      - "traefik.http.routers.traefik-secure.service=api@internal"

networks:
  ziti:
  proxy:

volumes:
  ziti-fs:
  ziti-identity:

Host/Intercept configs are the same, but I’ll leave here the commands just in case

Commands

zitiLogin

# I've run these commands so many times that i've created some shortcuts to quickly create configs and services
DOMAIN=mydomain.com
NODE=mymachine
SERVICE=traefik-dashboard
# Create identity for host, copy the token to environment variable so ziti-host can connect
ziti edge create identity device mymachine -o mymachine.jwt -a "$NODE.host"
# Create identity for client, copy and paste it into Ziti Edge Desktop
ziti edge create identity user jruiz -o jruiz.jwt -a "$SERVICE.$NODE.clients"

HOST_CONFIG_TCP_NAME=$NODE.tcp.hostv1

ziti edge create config $HOST_CONFIG_TCP_NAME host.v1 '{"protocol":"tcp","forwardPort":true,"allowedPortRanges":[{"low":80,"high":80},{"low":443,"high":443}],"address":"127.0.0.1"}'

INTERCEPT_CONFIG_TCP_NAME=$SERVICE.$NODE.http.interceptv1
SERVICE_NAME=$SERVICE.$NODE.service
BIND_NAME=$SERVICE.$NODE.bind
DIAL_NAME=$SERVICE.$NODE.dial
HOST_IDENTITY_NAME=$NODE.host
CLIENTS_IDENTITY_NAME=$SERVICE.$NODE.clients
ziti edge create config $INTERCEPT_CONFIG_TCP_NAME intercept.v1 '{"protocols":["tcp"],"addresses":["'$SERVICE'.'$NODE'.'$DOMAIN'"],"portRanges":[{"low":80,"high":80},{"low":443,"high":443}]}' && \
ziti edge create service $SERVICE_NAME --configs "$HOST_CONFIG_TCP_NAME,$INTERCEPT_CONFIG_TCP_NAME" && \
ziti edge create service-policy "$BIND_NAME" Bind --service-roles "@$SERVICE_NAME" --identity-roles "#$HOST_IDENTITY_NAME" && \
ziti edge create service-policy "$DIAL_NAME" Dial --service-roles "@$SERVICE_NAME" --identity-roles "#$CLIENTS_IDENTITY_NAME";

I’ve seen some folks online commenting that it might be that it tries to resolve first IPV6 and if it fails then it tries IPV4. Due to a recent issue I had with IPV6, I thought this might be affecting me as well here, but I’ve tried to disable IPV6 in every place I could think of and still no luck. I might have done it wrongly though but I don’t know.

I’ve been struggling with this for a while so I’m going to pause it for now, take a couple days off then I’ll come back and I’ll test it with a pre-generated certificate to see if that fixes everything or it’s something else.

TheLumberjack · April 22, 2023, 11:30am

Definitely interesting information. Thanks for following back up! It’s an interesting use case. I’m definitely interested to see what happens with a statically acquired/updated cert.

jruiz94 · April 23, 2023, 7:50pm

Status report: I couldn’t wait to keep experimenting, so I set my frustration apart and continued doing some tests.

First, I generated a wildcard certificate for *.mymachine.mydomain.com. I made it on a separate machine just to be sure I did it correctly xD.

Then, since I already had Traefik I thought “welp why not testing Traefik proxy with a static cert”, so I gave it a shot, but the problem persisted. I’d say that it was slightly faster, but we were talking about 30 seconds instead of 60 seconds, on every full reload. Still quite bad. Really weird.

Then I went to Nginx and the times started to look “fixed”. (all the times I’m posting here from now on are “full reloads”, CTRL + F5. Obviously regular reloads with the cache takes less than a second)
Minimum: 4 seconds
Approximate average: 6 seconds
But the maximum i registered was 15 seconds… which was very odd.

I had to leave to do some stuff outside home, then I came back and I thought “I know that my friend has a different ISP than me, and I know that my ISP loves ipv6 and I’m not 100% sure that doesn’t affect anything so what if I try to connect with his internet connection to see if that maybe affects something?”

But now we are both getting consistent 6 seconds average, I haven’t seen again 15 seconds in a while.

Sooo… yeah I don’t know, I guess it’s fixed? But it’s weird it seems like today the current time changed the result, so maybe the 15 second peaks were something else weird happening only in the morning…?

I’ll keep testing and re-trying to see if the bad load times somehow manage to come back but until then… I considered this solved, I guess

I’ll add a certbot container or something to statically get the certs and add them to a simple Nginx container. Good old Nginx always works.

Topic		Replies	Views
Router, Docker and DNS General Questions	7	213	November 14, 2023
Slow response and sometimes website inaccessible	9	141	December 12, 2023
Launched openziti network using docker compose, but client not able to fetch simple blue service Building/Development	40	1031	May 27, 2022
Best way to try openziti General Questions	14	410	May 23, 2023
DNS resolver not working with Docker Ziti Edge Tunnel	7	235	June 5, 2023

Troubleshooting slow response times of services through OpenZiti network

Related Topics