# Troubleshooting slow response times of services through OpenZiti network

**URL:** <https://openziti.discourse.group/t/troubleshooting-slow-response-times-of-services-through-openziti-network/1214>\
**Category:** General Questions\
**Created:** [April 18, 2023, 6:05pm UTC](https://openziti.discourse.group/t/troubleshooting-slow-response-times-of-services-through-openziti-network/1214 "2023-04-18T18:05:55Z")\
**Posts on this page:** 18\
**Page:** 1

<div class="post-metadata">

**Author:** ![jruiz94](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/jruiz94/32/652_2.png) [@jruiz94](https://openziti.discourse.group/u/jruiz94)\
**Post date:** [April 18, 2023, 6:05pm UTC](https://openziti.discourse.group/t/troubleshooting-slow-response-times-of-services-through-openziti-network/1214/1 "2023-04-18T18:05:55Z")

</div>

Hello, my setup is as follows, I’ll try to explain as best as I can the meaning of everything:

 ![Diagrama sin título.drawio](https://global.discourse-cdn.com/free1/uploads/netfoundry/original/1X/a3ab61f1d4937f857d9cdce490a65c6fc6f23e77.png)

I have the classic setup `ZitiEdgeController (with ZitiEdgeInitController) / ZitiEdgeRouter / ZitiConsole`

My goal with this is to have a couple servers running Docker containers of various services and use OpenZiti as a way to easily and securely connect between them. Additionally, my plan is to use a Caddy instance on each server to handle HTTPS requests so I don’t get warnings about self signed certs and stuff like that.

In the above setup, the Holy Trinity of OpenZiti (Controller/Router/Console) is running in its own network, called `ziti`. Then I have another network called `caddy` where there is a simple `whoami` container (which is basically a simple Nginx that returns a static HTML) and a Caddy Reverse Proxy container. This caddy container is also connected to the `ziti` network. This is done this way so I can reverse proxy the ZitiConsole container too.

The Caddy container is setup in such a way that it gets certificates by doing DNS challenges (I hope I’m using the correct terminology) against Cloudflare. I have a domain `my.domain.com` and I’ve given Caddy a special token to perform these dns acme challenges so it can get certificates.

The Caddy container is exposing to the localhost the ports 80 and 443.

I also have 2 containers: Ziti Host and Ziti Tun. Ziti Host is being used to, well, host services. Since I have Caddy on this machine I’ve created a config to host a service called caddy in ports 80 and 443 and intercepts urls with the `*.mymachine.my.domain.com`. ZitiTun for now is not being used, but if I deploy a container that needs to contact another machine in my Ziti network I’ll need it

This is done so I can type `https://ziti.mymachine.my.domain.com` on my desktop, then the requests is captured by Ziti Edge Desktop app on Windows, redirected through MyMachine, it arrives to the ZitiHost container, it is then redirected to port 443 on localhost of MyMachine, it is captured by Caddy, Caddy then does its magic with the certificates stuff, and makes the redirection to the Ziti Console container.

I can also type `https://whoami.mymachine.my.domain.com` and the same will apply but Caddy will redirect it at the last step to the other container.

All of this is **currently working nicely**. I can access all services with HTTPS with no warnings whatsoever

However, when I reach, for example, `https://ziti.mymachine.my.domain.com`, the answer is painfully slow. I’m talking about 60 seconds to load the login page. Once that’s done it’s all cached and it works well but anything new is really slow.  
I’ve also deployed other services to test (Syncthing for example) and it’s equally slow to access the UI.

So my question is, how can I debug and/or troubleshoot what might be causing this issue here? Perhaps it’s not even OpenZiti’s fault, but I don’t know what else could be?

I’ll leave my `docker-compose.yml` and `.env` files here in case that helps, but they might be a bit complex, idk.

> **Click here to see docker compose and .env**
>
> `docker-compose.yml`
> 
> ```auto
> # Sections prefixed by x- are not parsed by Docker Compose. This section is used to reuse common sections related to Caddy container (see below)
> x-common-caddy: &common-caddy
> caddy.tls.protocols: "tls1.3" #### This is optional. Default it is tls1.2
> caddy.tls.dns: "cloudflare ${CLOUDFLARE_API_TOKEN}"
> 
> services:
> ziti-controller:
> image: "${ZITI_IMAGE}:${ZITI_VERSION}"
> ports:
> - ${ZITI_EDGE_CONTROLLER_PORT:-1280}:${ZITI_EDGE_CONTROLLER_PORT:-1280}
> - ${ZITI_CTRL_PORT:-6262}:${ZITI_CTRL_PORT:-6262}
> environment:
> - ZITI_EDGE_IDENTITY_ENROLLMENT_DURATION=${ZITI_EDGE_IDENTITY_ENROLLMENT_DURATION}
> - ZITI_EDGE_ROUTER_ENROLLMENT_DURATION=${ZITI_EDGE_ROUTER_ENROLLMENT_DURATION}
> env_file:
> - $MAIN_DIR/.env
> networks:
> ziti:
> aliases:
> - ziti-edge-controller
> volumes:
> - ziti-fs:/persistent
> entrypoint:
> - "/var/openziti/scripts/run-controller.sh"
> # Simple healthcheck to check open ports. This is just to make the startup a little bit more "step by step"
> healthcheck:
> test: ["CMD", "bash", "-c", "lsof -i -P -n | grep -q 'TCP.*:6262' && lsof -i -P -n | grep -q 'TCP.*:1280'"]
> interval: 10s
> timeout: 5s
> retries: 10
> 
> ziti-controller-init-container:
> image: "${ZITI_IMAGE}:${ZITI_VERSION}"
> depends_on:
> ziti-controller:
> condition: service_healthy
> environment:
> - ZITI_CONTROLLER_RAWNAME="${ZITI_CONTROLLER_RAWNAME}"
> - ZITI_EDGE_CONTROLLER_RAWNAME="${ZITI_EDGE_CONTROLLER_RAWNAME}"
> env_file:
> - $MAIN_DIR/.env
> networks:
> ziti:
> aliases:
> - ziti-edge-controller-init-container
> volumes:
> - ziti-fs:/persistent
> entrypoint:
> - "/var/openziti/scripts/run-with-ziti-cli.sh"
> command:
> - "/var/openziti/scripts/access-control.sh"
> 
> ziti-edge-router:
> image: "${ZITI_IMAGE}:${ZITI_VERSION}"
> depends_on:
> ziti-controller:
> condition: service_healthy
> ziti-controller-init-container:
> condition: service_completed_successfully
> environment:
> - ZITI_CONTROLLER_RAWNAME="${ZITI_CONTROLLER_RAWNAME}"
> - ZITI_EDGE_CONTROLLER_RAWNAME="${ZITI_EDGE_CONTROLLER_RAWNAME}"
> - ZITI_EDGE_ROUTER_RAWNAME=${ZITI_EDGE_ROUTER_RAWNAME:-ziti-edge-router}
> - ZITI_EDGE_ROUTER_ROLES=public
> env_file:
> - $MAIN_DIR/.env
> ports:
> - ${ZITI_EDGE_ROUTER_PORT:-3022}:${ZITI_EDGE_ROUTER_PORT:-3022}
> networks:
> - ziti
> volumes:
> - ziti-fs:/persistent
> entrypoint: /bin/bash
> command: "/var/openziti/scripts/run-router.sh edge"
> # Simple healthcheck to check open ports. This is just to make the startup a little bit more "step by step"
> healthcheck:
> test: ["CMD", "bash", "-c", "lsof -i -P -n | grep -q 'TCP.*:3022'"]
> interval: 10s
> timeout: 5s
> retries: 10
> 
> ziti-console:
> image: openziti/zac:2.6.9
> depends_on:
> ziti-controller:
> condition: service_healthy
> ziti-controller-init-container:
> condition: service_completed_successfully
> ziti-edge-router:
> condition: service_healthy
> working_dir: /usr/src/app
> environment:
> - ZAC_SERVER_CERT_CHAIN=/persistent/pki/${ZITI_EDGE_CONTROLLER_HOSTNAME:-ziti-controller}-intermediate/certs/${ZITI_EDGE_CONTROLLER_HOSTNAME:-ziti-controller}-server.cert
> - ZAC_SERVER_KEY=/persistent/pki/${ZITI_EDGE_CONTROLLER_HOSTNAME:-ziti-controller}-intermediate/keys/${ZITI_EDGE_CONTROLLER_HOSTNAME:-ziti-controller}-server.key
> - PORTTLS=8443
> env_file:
> - $MAIN_DIR/.env
> # Labels are used by caddy container to know where are the upstreams and what url to use
> labels:
> <<: *common-caddy
> caddy: "ziti.${MACHINE_HOSTNAME}"
> caddy.reverse_proxy: "{{upstreams https 8443}}"
> caddy.reverse_proxy.transport: http
> # Tls insecure skip verify had to be used to prevent errors with https and caddy
> caddy.reverse_proxy.transport.tls_insecure_skip_verify:
> ports:
> # Port 8443 is not exposed, only through Caddy
> - 1408:1408
> networks:
> - ziti
> volumes:
> - ziti-fs:/persistent
>   
> # This container will fail on first 'docker compose up -d' because you can't get the ZITI_ENROLL_TOKEN until you generate it
> # Once you have the enroll token, add it to .env then run again docker compose up -d
> ziti-host:
> image: openziti/ziti-host:0.21.0
> depends_on:
> ziti-controller:
> condition: service_healthy
> ziti-edge-router:
> condition: service_healthy
> restart: unless-stopped
> network_mode: "host"
> volumes:
> - ziti-identity:/ziti-edge-tunnel
> environment:
> - ZITI_IDENTITY_BASENAME=${ZITI_IDENTITY_BASENAME}
> - ZITI_ENROLL_TOKEN=${ZITI_ENROLL_TOKEN}
>   
> # This container will fail on first 'docker compose up -d' because you can't get the ZITI_ENROLL_TOKEN until you generate it
> # Once you have the enroll token, add it to .env then run again docker compose up -d
> ziti-tun:
> image: openziti/ziti-edge-tunnel
> depends_on:
> ziti-controller:
> condition: service_healthy
> ziti-edge-router:
> condition: service_healthy
> ziti-host:
> condition: service_started
> restart: unless-stopped
> devices:
> - /dev/net/tun:/dev/net/tun
> volumes:
> - ziti-identity:/ziti-edge-tunnel
> - /var/run/dbus/system_bus_socket:/var/run/dbus/system_bus_socket
> environment:
> - ZITI_IDENTITY_BASENAME=${ZITI_IDENTITY_BASENAME}
> - PFXLOG_NO_JSON=true # suppress JSON logging
> network_mode: "host"
> privileged: true
> 
> caddy:
> image: homeall/caddy-reverse-proxy-cloudflare:latest
> depends_on:
> ziti-host:
> condition: service_started
> restart: unless-stopped
> environment:
> TZ: 'Europe/Madrid'
> volumes:
> - "/var/run/docker.sock:/var/run/docker.sock"
> - caddy_data:/data
> ports:
> - 80:80
> - 443:443
> networks:
> caddy:
> ziti:
> labels:
> caddy.email: ${CLOUDFLARE_EMAIL}
> 
> whoami:
> image: "nginxdemos/hello"
> depends_on:
> caddy:
> condition: service_started
> restart: unless-stopped
> networks:
> caddy:
> labels:
> <<: *common-caddy
> caddy: "whoami.${MACHINE_HOSTNAME}"
> caddy.reverse_proxy: "{{upstreams 80}}"
> 
> networks:
> ziti:
> caddy:
> 
> volumes:
> ziti-fs:
> caddy_data:
> ziti-identity:
> 
> ```
> 
> `.env`
> 
> ```auto
> # Generic
> MAIN_DIR=/home/ubuntu/yourdirectorywherethesefilesare
> MACHINE_HOSTNAME=mymachine.mydomain.com
> MACHINE_IP=1.2.3.4
> 
> # Caddy SSL
> CLOUDFLARE_EMAIL=youremail@email.com
> CLOUDFLARE_API_TOKEN=YOURCLOUDFLARETOKEN
> 
> # OpenZiti Variables
> ZITI_IMAGE=openziti/quickstart
> ZITI_VERSION=0.27.9
> 
> # The duration of the enrollment period (in minutes), default if not set
> # shown - 7days
> ZITI_EDGE_IDENTITY_ENROLLMENT_DURATION=10080
> ZITI_EDGE_ROUTER_ENROLLMENT_DURATION=10080
> 
> # controller address/port information
> ZITI_CONTROLLER_RAWNAME=ziti-controller
> ZITI_CONTROLLER_HOSTNAME="${MACHINE_HOSTNAME}"
> #ZITI_CTRL_PORT=8440
> 
> ZITI_EDGE_CONTROLLER_RAWNAME=ziti-edge-controller
> ZITI_EDGE_CONTROLLER_HOSTNAME="${MACHINE_HOSTNAME}"
> ZITI_EDGE_CONTROLLER_IP_OVERRIDE="${MACHINE_IP}"
> #ZITI_EDGE_CONTROLLER_PORT=8441
> 
> # router address/port information
> ZITI_EDGE_ROUTER_RAWNAME="${MACHINE_HOSTNAME}"
> ZITI_EDGE_ROUTER_IP_OVERRIDE="${MACHINE_IP}"
> #ZITI_EDGE_ROUTER_PORT=8442
> 
> # Ziti Host
> ZITI_IDENTITY_BASENAME=mymachine
> ZITI_ENROLL_TOKEN=
> 
> ```

Thanks in advance!

---

<div class="post-metadata">

**Author:** ![TheLumberjack](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/thelumberjack/32/113_2.png) [@TheLumberjack](https://openziti.discourse.group/u/TheLumberjack)\
**Post date:** [April 18, 2023, 7:14pm UTC](https://openziti.discourse.group/t/troubleshooting-slow-response-times-of-services-through-openziti-network/1214/2 "2023-04-18T19:14:12Z")

</div>

> [@jruiz94](#):
>
> I’m talking about 60 seconds to load the login page.

OUCH. That's obviously unacceptable and surprising. It's almost as though some timeout has to happen before traffic is flowing. Something is definitely strange.

The remote machine isn't hitting CPU file, or RAM bottlenecks right? When you access ZAC, do you do it entirely locally? Are you doing it through OpenZiti or directly? I assume via OpenZiti since port 8443 is not shown exposed? I've seen some cloud providers limit io/iops. Like on AWS, using a [T-type instance](https://aws.amazon.com/ec2/instance-types/t2/), I've seen the cloud provider limit me and makes delays like this that I just can't explain.

If this were me, I would probably take parts out of the equation and bring it all the way back to the basics. I would put up the simple [docker whale demo using crccheck/hello-world](https://github.com/openziti/ziti/blob/release-next/quickstart/docker/docker-compose.yml#L140). I would make a service that takes https out of the equation and see if it's slow. Assuming it's NOT slow, that would rule OpenZiti out as the cause of the problem imo. After that I would layer on HTTPS and then add caddy back into the mix and see if any of that matters.

It's a really strange issue, tbh, I'm leaning towards throttling at the cloud level at the moment...

---

<div class="post-metadata">

**Author:** ![gooseleggs](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/gooseleggs/32/450_2.png) [@gooseleggs](https://openziti.discourse.group/u/gooseleggs)\
**Post date:** [April 18, 2023, 7:16pm UTC](https://openziti.discourse.group/t/troubleshooting-slow-response-times-of-services-through-openziti-network/1214/3 "2023-04-18T19:16:25Z")

</div>

This looks like an interesting setup. Have you tried moving the whoami container to the ziti network and bypassing caddy altogether. Because you are using dns for certificates this is normally slower than http challenge response as it will sit there waiting for the dns to respond and depending upon the provider can take some time. That could explain the delay for the first connect if configured to do this on first connect. However not for the other connect delays.

Also I would use dev tools and see the waterfall of loading the page. It is only a simple page but it might give some insight.

---

<div class="post-metadata">

**Author:** ![TheLumberjack](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/thelumberjack/32/113_2.png) [@TheLumberjack](https://openziti.discourse.group/u/TheLumberjack)\
**Post date:** [April 18, 2023, 7:18pm UTC](https://openziti.discourse.group/t/troubleshooting-slow-response-times-of-services-through-openziti-network/1214/4 "2023-04-18T19:18:34Z")

</div>

Oh that’s a good point. If caddy is resolving/reissuing/reverifying the DNS each time, that definitely makes sense since doing the DNS challenge requires updating DNS records which often have TTLs in the “60s” (or much much more) range.

That seems like a good thing to verify indeed

---

<div class="post-metadata">

**Author:** ![jruiz94](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/jruiz94/32/652_2.png) [@jruiz94](https://openziti.discourse.group/u/jruiz94)\
**Post date:** [April 18, 2023, 7:49pm UTC](https://openziti.discourse.group/t/troubleshooting-slow-response-times-of-services-through-openziti-network/1214/5 "2023-04-18T19:49:45Z")

</div>

> [@TheLumberjack](#):
>
> The remote machine isn’t hitting CPU file, or RAM bottlenecks right?

Nope just checked and it's like 600MB out of 6GB on RAM and around 10% CPU i think.

> [@TheLumberjack](#):
>
> When you access ZAC, do you do it entirely locally? Are you doing it through OpenZiti or directly?

It's through OpenZiti, as I mentioned i'm using an intercept to "catch" the url `https://ziti.mymachine.mydomain.com` so that's routed to caddy and then caddy redirects to ZAC.

> [@TheLumberjack](#):
>
> I’ve seen some cloud providers limit io/iops. Like on AWS, using a [T-type instance](https://aws.amazon.com/ec2/instance-types/t2/), I’ve seen the cloud provider limit me and makes delays like this that I just can’t explain.

For the record I'm using Oracle cloud, those ARM based machines they recently have available.

> [@TheLumberjack](#):
>
> If this were me, I would probably take parts out of the equation and bring it all the way back to the basics. I would put up the simple [docker whale demo using crccheck/hello-world](https://github.com/openziti/ziti/blob/release-next/quickstart/docker/docker-compose.yml#L140).

Yeah I think I'm going to test that next, I'll try to set it up in such a way that I can revert to my current config easily, will update that when I have some time.

> [@gooseleggs](#):
>
> Have you tried moving the whoami container to the ziti network and bypassing caddy altogether.

Well the whoami is actually fast, its only 215ms but I assumed its because it's just a single html page? I might be mistaken.

> [@gooseleggs](#):
>
> Because you are using dns for certificates this is normally slower than http challenge response as it will sit there waiting for the dns to respond and depending upon the provider can take some time. That could explain the delay for the first connect if configured to do this on first connect. However not for the other connect delays.

Noted, yeah it's weird because the only time I don't get these high times is when it's cached (after first request). If I force reload with `CTRL + F5` then it's the same again.

BTW I didn't mention this because I thought it might not be related but since the DNS thing has come up in the conversation, I currently have another machine connected to my ziti network with a similar docker compose setup, in this case just ZitiHost/ZitiTun and a Caddy Container also that i was going to use to connect to another Syncthing UI.

I was using the default network for Caddy and Syncthing and Caddy couldn't get the certificates until I gave them their own network (I named it caddy again), and apparently it's related to DNS and the way docker compose handles this internally. I don't know if it's related but there are a plethora of issues with DNS and this Caddy proxy, it seems.

> [@gooseleggs](#):
>
> Also I would use dev tools and see the waterfall of loading the page. It is only a simple page but it might give some insight.

I've taken a couple quick screen captures for that. I'm not very webdev-savvy so I'm not fully confident on how to interpret the colours.

 ![imagen](https://global.discourse-cdn.com/free1/uploads/netfoundry/original/1X/3a5f324b4d83c9f795a79c42817d5a3d675fbbc4.png)

 ![imagen](https://global.discourse-cdn.com/free1/uploads/netfoundry/original/1X/50ed9f91ecddaf5cca03ecc39fb048ff2e17ae99.png)

> [@TheLumberjack](#):
>
> Oh that’s a good point. If caddy is resolving/reissuing/reverifying the DNS each time, that definitely makes sense since doing the DNS challenge requires updating DNS records which often have TTLs in the “60s” (or much much more) range.
> 
> That seems like a good thing to verify indeed

How can I verify if Caddy is reverifying DNS each time?

---

<div class="post-metadata">

**Author:** ![TheLumberjack](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/thelumberjack/32/113_2.png) [@TheLumberjack](https://openziti.discourse.group/u/TheLumberjack)\
**Post date:** [April 18, 2023, 7:59pm UTC](https://openziti.discourse.group/t/troubleshooting-slow-response-times-of-services-through-openziti-network/1214/6 "2023-04-18T19:59:44Z")

</div>

If you hover over the bar, it'll tell you:  
 ![image](https://global.discourse-cdn.com/free1/uploads/netfoundry/original/1X/da073c72d0d3ae87f4169a866cf86394bf589b9d.png)

Looks like your request was waiting for a response for 40s... Which is an _ETERNITY_!! 🙂

> [@jruiz94](#):
>
> How can I verify if Caddy is reverifying DNS each time?

Sadly, I don't know, that's a caddy type question. ☹ I did exactly this sort of thing but with nginx recently on a different discourse post... i used nginx to terminate TLS though and the overlay was not behaving like this is for you... I am interested in the "non-caddy-involved" path to see if that performance is acceptable.

You could test that easily by using the http port of your ZAC instead - generally 1408. That's a good test server to use -- probably better than the crccheck/hello-world since it's ZAC and a bit 'more' going on

---

<div class="post-metadata">

**Author:** ![jruiz94](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/jruiz94/32/652_2.png) [@jruiz94](https://openziti.discourse.group/u/jruiz94)\
**Post date:** [April 18, 2023, 8:10pm UTC](https://openziti.discourse.group/t/troubleshooting-slow-response-times-of-services-through-openziti-network/1214/7 "2023-04-18T20:10:09Z")

</div>

> [@TheLumberjack](#):
>
> Looks like your request was waiting for a response for 40s… Which is an _ETERNITY_!! 🙂

Yeah I hovered on one of those  
 ![imagen](https://global.discourse-cdn.com/free1/uploads/netfoundry/original/1X/c945728098e79da751166aa3ed9ad6459356103f.png)  
This one is the worst one, more than one minute to load hahaha

> [@TheLumberjack](#):
>
> You could test that easily by using the http port of your ZAC instead - generally 1408. That’s a good test server to use – probably better than the crccheck/hello-world since it’s ZAC and a bit ‘more’ going on

My bad for not reading the documentation, but you're telling me that I can access ZAC through http on port 1408 and skip https on port 8443? I think I'm going to try that, and also that would simplify my docker-compose.yml, i had a dirty skip\_tls\_verify around there.

Just to double check, what I should do is create a host config for tcp 1408 and `127.0.0.1` while exposing port 1408 on ZAC, and then an intercept config with whatever url i want to make and port 1408, right? (Then the service and bind and dial and that stuff, i know how to handle that so dw)

Btw the only reason I'm using caddy here is because it seemed like a super easy setup to get the automatic https certificates working, but if using Nginx or Traefik or whatever makes it a little bit more complex but at least works at acceptable speeds, I don't mind changing xD

---

<div class="post-metadata">

**Author:** ![TheLumberjack](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/thelumberjack/32/113_2.png) [@TheLumberjack](https://openziti.discourse.group/u/TheLumberjack)\
**Post date:** [April 18, 2023, 8:19pm UTC](https://openziti.discourse.group/t/troubleshooting-slow-response-times-of-services-through-openziti-network/1214/8 "2023-04-18T20:19:50Z")

</div>

> [@jruiz94](#):
>
> but you’re telling me that I can access ZAC through http on port 1408 and skip https on port 8443

Yes. It's configurable but 1408 is the default. It should be exposed inside the docker container space already. You should see something like:

```auto
ZAC will use this key for TLS: /persistent/pki/ziti-controller-intermediate/keys/ziti-controller-server.key
ZAC will present this pem for TLS: /persistent/pki/ziti-controller-intermediate/certs/ziti-controller-server.cert
Loading Settings File From: /usr/src/app/../ziti/settings.json
TLS initialized on port: 8443
Ziti Admin Console is now listening on port 1408

```

> just to double check, what I should do is create a host config for tcp 1408 and `127.0.0.1`

Yep. You could also learn how to specifiy more than one port per service and then use "FORWARD PORT" on the config  
 ![image](https://global.discourse-cdn.com/free1/uploads/netfoundry/original/1X/38b975e243a917f77c2a93006cf90e0085d3e96b.png)

Then you only have one config with two ports, but the way you're thinking is FINE too 🙂 Just showing you a new trick here is all...

> I’m using caddy here is because it seemed like a super easy setup to get the automatic https certificates working,

If you're not using it as an actual proxy, why just just use certbot? That's what I did for my DNS challenge... Then there's nothing else to install/run too? A cron to renew the cert is all you'd need?

> **[Certbot](https://certbot.eff.org/)**
>
> Tagline

---

<div class="post-metadata">

**Author:** ![jruiz94](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/jruiz94/32/652_2.png) [@jruiz94](https://openziti.discourse.group/u/jruiz94)\
**Post date:** [April 18, 2023, 9:06pm UTC](https://openziti.discourse.group/t/troubleshooting-slow-response-times-of-services-through-openziti-network/1214/9 "2023-04-18T21:06:54Z")

</div>

> [@TheLumberjack](#):
>
> Then you only have one config with two ports, but the way you’re thinking is FINE too 🙂 Just showing you a new trick here is all…

Holy pepperonni I should read the docs... I didn't know that, thanks for pointing it out! 😃

> [@TheLumberjack](#):
>
> If you’re not using it as an actual proxy, why just just use certbot? That’s what I did for my DNS challenge… Then there’s nothing else to install/run too? A cron to renew the cert is all you’d need?

The only 2 things I want are:

1. The ability to route addresses like i've showcased before (`ziti.mymachine.mydomain.com` or `syncthing.myothermachine.mydomain.com`)
2. Using valid non-self signed certificates so me and future users don't get scary warnings showing up.

I know a bit of certbot, I think I made another post some time ago and we mentioned it and I managed to obtain a couple certificates, but I think there was some kind of issue with wildcard certificates? Which I kinda want for having that flexibility to get any subdomain certificated instead of having to issue manually certificates (I might be mistaken, forgive mi ignorance if that's the case)

I'll update when I have some time to test what we mentioned about doing direct http to ZAC through OpenZiti

---

<div class="post-metadata">

**Author:** ![TheLumberjack](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/thelumberjack/32/113_2.png) [@TheLumberjack](https://openziti.discourse.group/u/TheLumberjack)\
**Post date:** [April 18, 2023, 9:20pm UTC](https://openziti.discourse.group/t/troubleshooting-slow-response-times-of-services-through-openziti-network/1214/10 "2023-04-18T21:20:19Z")

</div>

> [@jruiz94](#):
>
> think there was some kind of issue with wildcard certificates

A while ago they wouldn't issue them but they do now. I made a wildcard cert for my self hosted zrok instance. If it turns out that caddy is somehow causing this issue somehow, I'd just swap it out for cerbot and nginx. 😂

---

<div class="post-metadata">

**Author:** ![jruiz94](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/jruiz94/32/652_2.png) [@jruiz94](https://openziti.discourse.group/u/jruiz94)\
**Post date:** [April 19, 2023, 4:35pm UTC](https://openziti.discourse.group/t/troubleshooting-slow-response-times-of-services-through-openziti-network/1214/11 "2023-04-19T16:35:08Z")

</div>

All right so I did the following commands

```auto
zitiLogin

ziti edge create config special.console.host host.v1 '{"protocol":"tcp","address":"127.0.0.1","port":1408}'

ziti edge create config special.console.intercept intercept.v1 '{"protocols":["tcp"],"addresses":["my.console.ziti"],"portRanges":[{"low":1408,"high":1408}]}'

ziti edge create service special.service --configs "special.console.host,special.console.intercept"

ziti edge create service-policy special.bind Bind --service-roles "@special.service" --identity-roles "#my.host"

ziti edge create service-policy special.dial Dial --service-roles "@special.service" --identity-roles "#special.my.clients"

```

I called them `special` because im bad at names and wanted to distinguish them from the other configs.

I can see my new service in my Ziti Edge Desktop UI in Windows, and when i get `http://my.console.ziti:1408` in my browser, something weird happens.

 ![imagen](https://global.discourse-cdn.com/free1/uploads/netfoundry/original/1X/ff95c9b98d8364584477726ff38dd1423110c339.png)

So the times are pretty low, which is nice, but everything throws an error apparently

First 2 requests seem okay"

 ![imagen](https://global.discourse-cdn.com/free1/uploads/netfoundry/original/1X/bb763cf65e4e20ee1c9dd586ca784e881b115a3c.png)

 ![imagen](https://global.discourse-cdn.com/free1/uploads/netfoundry/original/1X/cc3a745cea8872adbd6e9af1ffd04359d614be85.png)

Then the rest are marked as error, with no apparent response whatsoever, but I think it’s weird that the url changed from http to https

 ![imagen](https://global.discourse-cdn.com/free1/uploads/netfoundry/original/1X/d021ea91bafd00729cad4752596f75bad4fa1d61.png)

However, it really does seem like it is a Caddy issue indeed, so I think it’s time to take a look at good old Nginx, see if I can reproduce this…

---

<div class="post-metadata">

**Author:** ![gooseleggs](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/gooseleggs/32/450_2.png) [@gooseleggs](https://openziti.discourse.group/u/gooseleggs)\
**Post date:** [April 19, 2023, 7:03pm UTC](https://openziti.discourse.group/t/troubleshooting-slow-response-times-of-services-through-openziti-network/1214/12 "2023-04-19T19:03:59Z")

</div>

It will be ZAC that is doing the https prefix in the assets that it needs. Change your config to be 8443, the https port and should work. You haven’t exposed 8443 in the docker config and you don’t need too as on the same docker config. However to make it simple expose 8443 on the host like you have 1408 and change the above ports to 8443.

You will get a cert warning when going to the url but that is ok for testing ( I don’t bother about a valid SSL on the Zac as users don’t interact with it).

As for caddy, I am keen to have a play with this. Traditionally I have used nginx with nginx-dockergen and nginx-letsencrypt as a trio for container work - it listens for changes in containers and modifies the nginx config on the start/stop of containers and sorts certs out for me as well as renewals.

---

<div class="post-metadata">

**Author:** ![TheLumberjack](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/thelumberjack/32/113_2.png) [@TheLumberjack](https://openziti.discourse.group/u/TheLumberjack)\
**Post date:** [April 19, 2023, 7:07pm UTC](https://openziti.discourse.group/t/troubleshooting-slow-response-times-of-services-through-openziti-network/1214/13 "2023-04-19T19:07:08Z")

</div>

> [@jruiz94](#):
>
> something weird happens.

something weird indeed! I must admit, I've never seen that particular issue just yet. I took your commands and I can reproduce the issue. I'll see if i can figure out what's going on there since that's definitely unexpected. @jeremy.tellier is probably not availalbe to troubleshoot for a bit so it might be a while to get this sorted...

My http zac:  
 ![image](https://global.discourse-cdn.com/free1/uploads/netfoundry/original/1X/4d1e21126d626617af9cdd6c7516b608542393ea.png)

---

<div class="post-metadata">

**Author:** ![TheLumberjack](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/thelumberjack/32/113_2.png) [@TheLumberjack](https://openziti.discourse.group/u/TheLumberjack)\
**Post date:** [April 19, 2023, 7:13pm UTC](https://openziti.discourse.group/t/troubleshooting-slow-response-times-of-services-through-openziti-network/1214/14 "2023-04-19T19:13:13Z")

</div>

Yeah, it looks like if HTTPS is enabled, it triggers ZAC to redirect to the HTTPS port. That’s a bug in ZAC imo. I did _not_ add 8443 to `my.console.ziti` and that’s why all those requests are failing…

 ![image](https://global.discourse-cdn.com/free1/uploads/netfoundry/original/1X/ba8c740deb802562939b3384e2246cab61b2f718.png)

I think we ruled ziti out in this situation though. 🙂

---

<div class="post-metadata">

**Author:** ![jruiz94](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/jruiz94/32/652_2.png) [@jruiz94](https://openziti.discourse.group/u/jruiz94)\
**Post date:** [April 20, 2023, 3:39pm UTC](https://openziti.discourse.group/t/troubleshooting-slow-response-times-of-services-through-openziti-network/1214/15 "2023-04-20T15:39:08Z")

</div>

> [@TheLumberjack](#):
>
> I think we ruled ziti out in this situation though. 🙂

Yeah I also think that the bottom line of all this is that Ziti was not the issue so I'll have to check another reverse proxy to fix my super long response time problems 😆

> [@gooseleggs](#):
>
> As for caddy, I am keen to have a play with this. Traditionally I have used nginx with nginx-dockergen and nginx-letsencrypt as a trio for container work - it listens for changes in containers and modifies the nginx config on the start/stop of containers and sorts certs out for me as well as renewals.

I'm not very expert with Caddy, I've mostly used Traefik both at home and at work but I kinda like the simplicity of Caddy at least in this variant where it uses the labels to do the config.

Before ditching it completely, I'm going to try something I just saw yesterday about generating wildcard certificates directly from Cloudflare's UI and paste them directly on a directory for Caddy to use them.

Otherwise if that doesn't work, I'll go back to Traefik or even Nginx, the good old reliable, hahaha.

I will update soon with what worked for me in the end!

---

<div class="post-metadata">

**Author:** ![jruiz94](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/jruiz94/32/652_2.png) [@jruiz94](https://openziti.discourse.group/u/jruiz94)\
**Post date:** [April 22, 2023, 11:23am UTC](https://openziti.discourse.group/t/troubleshooting-slow-response-times-of-services-through-openziti-network/1214/16 "2023-04-22T11:23:59Z")

</div>

I’ve been these days doing the same that I had with Caddy but with Traefik, and the problem is still the same, I think the ACME DNS Challenge fails or takes too long to solve at some point but I don’t know why.

I wanted to test this ACME DNS Challenge with Traefik because I have this exact Traefik-ProxiedContainer setup on a public machine (to host a public webpage, no OpenZiti) and it works without issues.

I’ve simplified my docker compose in order to narrow down the issue but I still don’t know what might be.  
Traefik has a dashboard so I’ve been using that to test instead of ZAC (it’s the same in the end, a UI to query to see the times to load). I’ve also removed for now `ziti-tun` and `whoami`, since i only need `ziti-host` and `traefik`

> **docker-compose.yml**
>
> ```yml
> services:
> ziti-controller:
> image: "${ZITI_IMAGE}:${ZITI_VERSION}"
> ports:
> - ${ZITI_EDGE_CONTROLLER_PORT:-1280}:${ZITI_EDGE_CONTROLLER_PORT:-1280}
> - ${ZITI_CTRL_PORT:-6262}:${ZITI_CTRL_PORT:-6262}
> environment:
> - ZITI_EDGE_IDENTITY_ENROLLMENT_DURATION=${ZITI_EDGE_IDENTITY_ENROLLMENT_DURATION}
> - ZITI_EDGE_ROUTER_ENROLLMENT_DURATION=${ZITI_EDGE_ROUTER_ENROLLMENT_DURATION}
> env_file:
> - $MAIN_DIR/.env
> networks:
> ziti:
> aliases:
> - ziti-edge-controller
> volumes:
> - ziti-fs:/persistent
> entrypoint:
> - "/var/openziti/scripts/run-controller.sh"
> healthcheck:
> test: ["CMD", "bash", "-c", "lsof -i -P -n | grep -q 'TCP.*:6262' && lsof -i -P -n | grep -q 'TCP.*:1280'"]
> interval: 10s
> timeout: 5s
> retries: 10
> 
> ziti-controller-init-container:
> image: "${ZITI_IMAGE}:${ZITI_VERSION}"
> depends_on:
> ziti-controller:
> condition: service_healthy
> environment:
> - ZITI_CONTROLLER_RAWNAME="${ZITI_CONTROLLER_RAWNAME}"
> - ZITI_EDGE_CONTROLLER_RAWNAME="${ZITI_EDGE_CONTROLLER_RAWNAME}"
> env_file:
> - $MAIN_DIR/.env
> networks:
> ziti:
> aliases:
> - ziti-edge-controller-init-container
> volumes:
> - ziti-fs:/persistent
> entrypoint:
> - "/var/openziti/scripts/run-with-ziti-cli.sh"
> command:
> - "/var/openziti/scripts/access-control.sh"
> 
> ziti-edge-router:
> image: "${ZITI_IMAGE}:${ZITI_VERSION}"
> depends_on:
> ziti-controller:
> condition: service_healthy
> ziti-controller-init-container:
> condition: service_completed_successfully
> environment:
> - ZITI_CONTROLLER_RAWNAME="${ZITI_CONTROLLER_RAWNAME}"
> - ZITI_EDGE_CONTROLLER_RAWNAME="${ZITI_EDGE_CONTROLLER_RAWNAME}"
> - ZITI_EDGE_ROUTER_RAWNAME=${ZITI_EDGE_ROUTER_RAWNAME:-ziti-edge-router}
> - ZITI_EDGE_ROUTER_ROLES=public
> env_file:
> - $MAIN_DIR/.env
> ports:
> - ${ZITI_EDGE_ROUTER_PORT:-3022}:${ZITI_EDGE_ROUTER_PORT:-3022}
> networks:
> - ziti
> volumes:
> - ziti-fs:/persistent
> entrypoint: /bin/bash
> command: "/var/openziti/scripts/run-router.sh edge"
> healthcheck:
> test: ["CMD", "bash", "-c", "lsof -i -P -n | grep -q 'TCP.*:3022'"]
> interval: 10s
> timeout: 5s
> retries: 10
> 
> ziti-host:
> image: openziti/ziti-host:0.21.0
> depends_on:
> ziti-controller:
> condition: service_healthy
> ziti-edge-router:
> condition: service_healthy
> restart: unless-stopped
> network_mode: "host"
> volumes:
> - ziti-identity:/ziti-edge-tunnel
> environment:
> - ZITI_IDENTITY_BASENAME=${ZITI_IDENTITY_BASENAME}
> - ZITI_ENROLL_TOKEN=${ZITI_ENROLL_TOKEN}
> 
> traefik:
> image: traefik:latest
> container_name: traefik
> restart: unless-stopped
> security_opt:
> - no-new-privileges:true
> depends_on:
> ziti-host:
> condition: service_started
> ziti-tun:
> condition: service_started
> networks:
> proxy:
> ports:
> - 80:80
> - 443:443
> environment:
> - CF_API_EMAIL=${CLOUDFLARE_EMAIL}
> - CF_DNS_API_TOKEN=${CLOUDFLARE_API_TOKEN}
> volumes:
> - /etc/localtime:/etc/localtime:ro
> - /var/run/docker.sock:/var/run/docker.sock:ro
> - ${MAIN_DIR}/traefik/data/traefik.yml:/traefik.yml:ro
> - ${MAIN_DIR}/traefik/data/config.yml:/config.yml:ro
> - ${MAIN_DIR}/traefik/data/acme.json:/acme.json
> labels:
> - "traefik.enable=true"
> - "traefik.http.routers.traefik.entrypoints=http"
> - "traefik.http.routers.traefik.rule=Host(`traefik-dashboard.mymachine.mydomain.com`)"
> - "traefik.http.middlewares.traefik-auth.basicauth.users=${TRAEFIK_BASIC_AUTH}"
> - "traefik.http.middlewares.traefik-https-redirect.redirectscheme.scheme=https"
> - "traefik.http.middlewares.sslheader.headers.customrequestheaders.X-Forwarded-Proto=https"
> - "traefik.http.routers.traefik.middlewares=traefik-https-redirect"
> - "traefik.http.routers.traefik-secure.entrypoints=https"
> - "traefik.http.routers.traefik-secure.rule=Host(`traefik-dashboard.mymachine.mydomain.com`)"
> - "traefik.http.routers.traefik-secure.middlewares=traefik-auth"
> - "traefik.http.routers.traefik-secure.tls=true"
> - "traefik.http.routers.traefik-secure.tls.certresolver=cloudflare"
> - "traefik.http.routers.traefik-secure.tls.domains[0].main=mymachine.mydomain.com"
> - "traefik.http.routers.traefik-secure.tls.domains[0].sans=*.mymachine.mydomain.com"
> - "traefik.http.routers.traefik-secure.service=api@internal"
> 
> networks:
> ziti:
> proxy:
> 
> volumes:
> ziti-fs:
> ziti-identity:
> 
> ```

Host/Intercept configs are the same, but I’ll leave here the commands just in case

> **Commands**
>
> ```bash
> zitiLogin
> 
> # I've run these commands so many times that i've created some shortcuts to quickly create configs and services
> DOMAIN=mydomain.com
> NODE=mymachine
> SERVICE=traefik-dashboard
> # Create identity for host, copy the token to environment variable so ziti-host can connect
> ziti edge create identity device mymachine -o mymachine.jwt -a "$NODE.host"
> # Create identity for client, copy and paste it into Ziti Edge Desktop
> ziti edge create identity user jruiz -o jruiz.jwt -a "$SERVICE.$NODE.clients"
> 
> HOST_CONFIG_TCP_NAME=$NODE.tcp.hostv1
> 
> ziti edge create config $HOST_CONFIG_TCP_NAME host.v1 '{"protocol":"tcp","forwardPort":true,"allowedPortRanges":[{"low":80,"high":80},{"low":443,"high":443}],"address":"127.0.0.1"}'
> 
> INTERCEPT_CONFIG_TCP_NAME=$SERVICE.$NODE.http.interceptv1
> SERVICE_NAME=$SERVICE.$NODE.service
> BIND_NAME=$SERVICE.$NODE.bind
> DIAL_NAME=$SERVICE.$NODE.dial
> HOST_IDENTITY_NAME=$NODE.host
> CLIENTS_IDENTITY_NAME=$SERVICE.$NODE.clients
> ziti edge create config $INTERCEPT_CONFIG_TCP_NAME intercept.v1 '{"protocols":["tcp"],"addresses":["'$SERVICE'.'$NODE'.'$DOMAIN'"],"portRanges":[{"low":80,"high":80},{"low":443,"high":443}]}' && \
> ziti edge create service $SERVICE_NAME --configs "$HOST_CONFIG_TCP_NAME,$INTERCEPT_CONFIG_TCP_NAME" && \
> ziti edge create service-policy "$BIND_NAME" Bind --service-roles "@$SERVICE_NAME" --identity-roles "#$HOST_IDENTITY_NAME" && \
> ziti edge create service-policy "$DIAL_NAME" Dial --service-roles "@$SERVICE_NAME" --identity-roles "#$CLIENTS_IDENTITY_NAME";
> 
> ```

I’ve seen some folks online commenting that it might be that it tries to resolve first IPV6 and if it fails then it tries IPV4. [Due to a recent issue I had with IPV6](https://openziti.discourse.group/t/problems-with-docker-compose-setup-and-minecraft-example/1018/32), I thought this might be affecting me as well here, but I’ve tried to disable IPV6 in every place I could think of and still no luck. I might have done it wrongly though but I don’t know.

I’ve been struggling with this for a while so I’m going to pause it for now, take a couple days off then I’ll come back and I’ll test it with a pre-generated certificate to see if that fixes everything or it’s something else.

---

<div class="post-metadata">

**Author:** ![TheLumberjack](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/thelumberjack/32/113_2.png) [@TheLumberjack](https://openziti.discourse.group/u/TheLumberjack)\
**Post date:** [April 22, 2023, 11:30am UTC](https://openziti.discourse.group/t/troubleshooting-slow-response-times-of-services-through-openziti-network/1214/17 "2023-04-22T11:30:14Z")

</div>

Definitely interesting information. Thanks for following back up! It’s an interesting use case. I’m definitely interested to see what happens with a statically acquired/updated cert.

---

<div class="post-metadata">

**Author:** ![jruiz94](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/jruiz94/32/652_2.png) [@jruiz94](https://openziti.discourse.group/u/jruiz94)\
**Post date:** [April 23, 2023, 7:50pm UTC](https://openziti.discourse.group/t/troubleshooting-slow-response-times-of-services-through-openziti-network/1214/18 "2023-04-23T19:50:01Z")

</div>

Status report: I couldn’t wait to keep experimenting, so I set my frustration apart and continued doing some tests.

First, I generated a wildcard certificate for `*.mymachine.mydomain.com`. I made it on a separate machine just to be sure I did it correctly xD.

Then, since I already had Traefik I thought “welp why not testing Traefik proxy with a static cert”, so I gave it a shot, but the problem persisted. I’d say that it was slightly faster, but we were talking about 30 seconds instead of 60 seconds, on every full reload. Still quite bad. Really weird.

Then I went to Nginx and the times started to look “fixed”. (all the times I’m posting here from now on are “full reloads”, CTRL + F5. Obviously regular reloads with the cache takes less than a second)  
Minimum: 4 seconds  
Approximate average: 6 seconds  
But the maximum i registered was 15 seconds… which was very odd.

I had to leave to do some stuff outside home, then I came back and I thought “I know that my friend has a different ISP than me, and I know that my ISP loves ipv6 and I’m not 100% sure that doesn’t affect anything so what if I try to connect with his internet connection to see if that maybe affects something?”

But now we are both getting consistent 6 seconds average, I haven’t seen again 15 seconds in a while.

Sooo… yeah I don’t know, I guess it’s fixed? But it’s weird it seems like today the current time changed the result, so maybe the 15 second peaks were something else weird happening only in the morning…?

I’ll keep testing and re-trying to see if the bad load times somehow manage to come back but until then… I considered this solved, I guess 😛

I’ll add a certbot container or something to statically get the certs and add them to a simple Nginx container. Good old Nginx always works.