Troubleshooting slow response times of services through OpenZiti network

Support General Questions
16

I’ve been these days doing the same that I had with Caddy but with Traefik, and the problem is still the same, I think the ACME DNS Challenge fails or takes too long to solve at some point but I don’t know why.

I wanted to test this ACME DNS Challenge with Traefik because I have this exact Traefik-ProxiedContainer setup on a public machine (to host a public webpage, no OpenZiti) and it works without issues.

I’ve simplified my docker compose in order to narrow down the issue but I still don’t know what might be.
Traefik has a dashboard so I’ve been using that to test instead of ZAC (it’s the same in the end, a UI to query to see the times to load). I’ve also removed for now ziti-tun and whoami, since i only need ziti-host and traefik

docker-compose.yml 
services:
  ziti-controller:
    image: "${ZITI_IMAGE}:${ZITI_VERSION}"
    ports:
      - ${ZITI_EDGE_CONTROLLER_PORT:-1280}:${ZITI_EDGE_CONTROLLER_PORT:-1280}
      - ${ZITI_CTRL_PORT:-6262}:${ZITI_CTRL_PORT:-6262}
    environment:
      - ZITI_EDGE_IDENTITY_ENROLLMENT_DURATION=${ZITI_EDGE_IDENTITY_ENROLLMENT_DURATION}
      - ZITI_EDGE_ROUTER_ENROLLMENT_DURATION=${ZITI_EDGE_ROUTER_ENROLLMENT_DURATION}
    env_file:
      - $MAIN_DIR/.env
    networks:
      ziti:
        aliases:
          - ziti-edge-controller
    volumes:
      - ziti-fs:/persistent
    entrypoint:
      - "/var/openziti/scripts/run-controller.sh"
    healthcheck:
      test: ["CMD", "bash", "-c", "lsof -i -P -n | grep -q 'TCP.*:6262' && lsof -i -P -n | grep -q 'TCP.*:1280'"]
      interval: 10s
      timeout: 5s
      retries: 10

  ziti-controller-init-container:
    image: "${ZITI_IMAGE}:${ZITI_VERSION}"
    depends_on:
      ziti-controller:
        condition: service_healthy
    environment:
      - ZITI_CONTROLLER_RAWNAME="${ZITI_CONTROLLER_RAWNAME}"
      - ZITI_EDGE_CONTROLLER_RAWNAME="${ZITI_EDGE_CONTROLLER_RAWNAME}"
    env_file:
      - $MAIN_DIR/.env
    networks:
      ziti:
        aliases:
          - ziti-edge-controller-init-container
    volumes:
      - ziti-fs:/persistent
    entrypoint:
      - "/var/openziti/scripts/run-with-ziti-cli.sh"
    command:
      - "/var/openziti/scripts/access-control.sh"

  ziti-edge-router:
    image: "${ZITI_IMAGE}:${ZITI_VERSION}"
    depends_on:
      ziti-controller:
        condition: service_healthy
      ziti-controller-init-container:
        condition: service_completed_successfully
    environment:
      - ZITI_CONTROLLER_RAWNAME="${ZITI_CONTROLLER_RAWNAME}"
      - ZITI_EDGE_CONTROLLER_RAWNAME="${ZITI_EDGE_CONTROLLER_RAWNAME}"
      - ZITI_EDGE_ROUTER_RAWNAME=${ZITI_EDGE_ROUTER_RAWNAME:-ziti-edge-router}
      - ZITI_EDGE_ROUTER_ROLES=public
    env_file:
      - $MAIN_DIR/.env
    ports:
      - ${ZITI_EDGE_ROUTER_PORT:-3022}:${ZITI_EDGE_ROUTER_PORT:-3022}
    networks:
      - ziti
    volumes:
      - ziti-fs:/persistent
    entrypoint: /bin/bash
    command: "/var/openziti/scripts/run-router.sh edge"
    healthcheck:
      test: ["CMD", "bash", "-c", "lsof -i -P -n | grep -q 'TCP.*:3022'"]
      interval: 10s
      timeout: 5s
      retries: 10

  ziti-host:
    image: openziti/ziti-host:0.21.0
    depends_on:
      ziti-controller:
        condition: service_healthy
      ziti-edge-router:
        condition: service_healthy
    restart: unless-stopped
    network_mode: "host"
    volumes:
        - ziti-identity:/ziti-edge-tunnel
    environment:
        - ZITI_IDENTITY_BASENAME=${ZITI_IDENTITY_BASENAME}
        - ZITI_ENROLL_TOKEN=${ZITI_ENROLL_TOKEN}

  traefik:
    image: traefik:latest
    container_name: traefik
    restart: unless-stopped
    security_opt:
      - no-new-privileges:true
    depends_on:
      ziti-host:
        condition: service_started
      ziti-tun:
        condition: service_started
    networks:
      proxy:
    ports:
      - 80:80
      - 443:443
    environment:
      - CF_API_EMAIL=${CLOUDFLARE_EMAIL}
      - CF_DNS_API_TOKEN=${CLOUDFLARE_API_TOKEN}
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ${MAIN_DIR}/traefik/data/traefik.yml:/traefik.yml:ro
      - ${MAIN_DIR}/traefik/data/config.yml:/config.yml:ro
      - ${MAIN_DIR}/traefik/data/acme.json:/acme.json
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.traefik.entrypoints=http"
      - "traefik.http.routers.traefik.rule=Host(`traefik-dashboard.mymachine.mydomain.com`)"
      - "traefik.http.middlewares.traefik-auth.basicauth.users=${TRAEFIK_BASIC_AUTH}"
      - "traefik.http.middlewares.traefik-https-redirect.redirectscheme.scheme=https"
      - "traefik.http.middlewares.sslheader.headers.customrequestheaders.X-Forwarded-Proto=https"
      - "traefik.http.routers.traefik.middlewares=traefik-https-redirect"
      - "traefik.http.routers.traefik-secure.entrypoints=https"
      - "traefik.http.routers.traefik-secure.rule=Host(`traefik-dashboard.mymachine.mydomain.com`)"
      - "traefik.http.routers.traefik-secure.middlewares=traefik-auth"
      - "traefik.http.routers.traefik-secure.tls=true"
      - "traefik.http.routers.traefik-secure.tls.certresolver=cloudflare"
      - "traefik.http.routers.traefik-secure.tls.domains[0].main=mymachine.mydomain.com"
      - "traefik.http.routers.traefik-secure.tls.domains[0].sans=*.mymachine.mydomain.com"
      - "traefik.http.routers.traefik-secure.service=api@internal"

networks:
  ziti:
  proxy:

volumes:
  ziti-fs:
  ziti-identity:

Host/Intercept configs are the same, but I’ll leave here the commands just in case

Commands 
zitiLogin

# I've run these commands so many times that i've created some shortcuts to quickly create configs and services
DOMAIN=mydomain.com
NODE=mymachine
SERVICE=traefik-dashboard
# Create identity for host, copy the token to environment variable so ziti-host can connect
ziti edge create identity device mymachine -o mymachine.jwt -a "$NODE.host"
# Create identity for client, copy and paste it into Ziti Edge Desktop
ziti edge create identity user jruiz -o jruiz.jwt -a "$SERVICE.$NODE.clients"

HOST_CONFIG_TCP_NAME=$NODE.tcp.hostv1

ziti edge create config $HOST_CONFIG_TCP_NAME host.v1 '{"protocol":"tcp","forwardPort":true,"allowedPortRanges":[{"low":80,"high":80},{"low":443,"high":443}],"address":"127.0.0.1"}'

INTERCEPT_CONFIG_TCP_NAME=$SERVICE.$NODE.http.interceptv1
SERVICE_NAME=$SERVICE.$NODE.service
BIND_NAME=$SERVICE.$NODE.bind
DIAL_NAME=$SERVICE.$NODE.dial
HOST_IDENTITY_NAME=$NODE.host
CLIENTS_IDENTITY_NAME=$SERVICE.$NODE.clients
ziti edge create config $INTERCEPT_CONFIG_TCP_NAME intercept.v1 '{"protocols":["tcp"],"addresses":["'$SERVICE'.'$NODE'.'$DOMAIN'"],"portRanges":[{"low":80,"high":80},{"low":443,"high":443}]}' && \
ziti edge create service $SERVICE_NAME --configs "$HOST_CONFIG_TCP_NAME,$INTERCEPT_CONFIG_TCP_NAME" && \
ziti edge create service-policy "$BIND_NAME" Bind --service-roles "@$SERVICE_NAME" --identity-roles "#$HOST_IDENTITY_NAME" && \
ziti edge create service-policy "$DIAL_NAME" Dial --service-roles "@$SERVICE_NAME" --identity-roles "#$CLIENTS_IDENTITY_NAME";

I’ve seen some folks online commenting that it might be that it tries to resolve first IPV6 and if it fails then it tries IPV4. Due to a recent issue I had with IPV6, I thought this might be affecting me as well here, but I’ve tried to disable IPV6 in every place I could think of and still no luck. I might have done it wrongly though but I don’t know.

I’ve been struggling with this for a while so I’m going to pause it for now, take a couple days off then I’ll come back and I’ll test it with a pre-generated certificate to see if that fixes everything or it’s something else.