# Blue not running in Docker Compose Quickstart

**URL:** <https://openziti.discourse.group/t/blue-not-running-in-docker-compose-quickstart/1473>\
**Category:** Building/Development\
**Created:** [August 10, 2023, 5:27pm UTC](https://openziti.discourse.group/t/blue-not-running-in-docker-compose-quickstart/1473 "2023-08-10T17:27:51Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![bearrito](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/bearrito/32/1189_2.png) [@bearrito](https://openziti.discourse.group/u/bearrito)\
**Post date:** [August 10, 2023, 5:27pm UTC](https://openziti.discourse.group/t/blue-not-running-in-docker-compose-quickstart/1473/1 "2023-08-10T17:27:51Z")

</div>

Hello,

I'm attempting to run the Docker Compose quickstart guide. I downloaded both the compose file and .env file.

Looking at

> ziti edge list edge-routers

I could see that Blue was missing

The logs show

> ziti-private-blue\_1 | [0.000] ERROR ziti/ziti/router.run: {error=[unable to load identity (open /persistent/ziti-private-blue.key: no such file or directory)] os=[linux] version=[v0.29.0] revision=[3ca2dd2f4e7b] configFile=[/persistent/ziti-private-blue.yaml] arch=[amd64] build-date=[2023-07-13T15:53:37Z] go-version=[go1.20.5]} error loading ziti-router config  
> open-ziti\_ziti-controller-init-container\_1 exited with code 0  
> ziti-private-blue\_1 | panic: unable to load identity (open /persistent/ziti-private-blue.key: no such file or directory)

Red runs fine.

---

<div class="post-metadata">

**Author:** ![qrkourier](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/qrkourier/32/52_2.png) [@qrkourier](https://openziti.discourse.group/u/qrkourier)\
**Post date:** [August 10, 2023, 5:33pm UTC](https://openziti.discourse.group/t/blue-not-running-in-docker-compose-quickstart/1473/2 "2023-08-10T17:33:20Z")

</div>

Welcome to the fun @bearrito 🙂

I couldn’t get the same result, so I’m trying to think of what could be different in your environment. My blue router seems happy with the latest. Any chance you have an old container image in your local cache? `docker compose pull` will freshen if needed.

```bash
$ ziti edge list edge-routers
╭────────────┬───────────────────────┬────────┬───────────────┬──────┬─────────────╮
│ ID │ NAME │ ONLINE │ ALLOW TRANSIT │ COST │ ATTRIBUTES │
├────────────┼───────────────────────┼────────┼───────────────┼──────┼─────────────┤
│ 0e3oC6cBCg │ ziti-edge-router │ true │ true │ 0 │ public │
│ 5ceBC6cBCg │ ziti-fabric-router-br │ true │ true │ 0 │ fabric-only │
│ AfaBC6voCZ │ ziti-private-blue │ true │ true │ 0 │ zitiblue │
│ tzLok-vBC │ ziti-edge-router-wss │ true │ true │ 0 │ public │
│ zMeBk-cBCg │ ziti-private-red │ true │ true │ 0 │ zitired │
╰────────────┴───────────────────────┴────────┴───────────────┴──────┴─────────────╯
results: 1-5 of 5

```

---

<div class="post-metadata">

**Author:** ![gberl002](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/gberl002/32/333_2.png) [@gberl002](https://openziti.discourse.group/u/gberl002)\
**Post date:** [August 10, 2023, 5:36pm UTC](https://openziti.discourse.group/t/blue-not-running-in-docker-compose-quickstart/1473/3 "2023-08-10T17:36:28Z")

</div>

@bearrito There is a race condition I’m working on a fix for, it’s possible you are experiencing this issue. Can you try deploying the compose environment again and see if it’s a recurring issue with just blue or if it intermittently happens with others?

---

<div class="post-metadata">

**Author:** ![qrkourier](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/qrkourier/32/52_2.png) [@qrkourier](https://openziti.discourse.group/u/qrkourier)\
**Post date:** [August 10, 2023, 5:37pm UTC](https://openziti.discourse.group/t/blue-not-running-in-docker-compose-quickstart/1473/4 "2023-08-10T17:37:26Z")

</div>

I made sure I do actually have that key file.

```bash
$ docker compose exec ziti-private-blue ls -lh /persistent/ziti-private-blue.key 
-rw------- 1 ziti ziti 3.2K Aug 10 17:29 /persistent/ziti-private-blue.key

```

If you need to destroy the data in the quickstart project to try again:

```bash
docker compose down --volumes           

```

---

<div class="post-metadata">

**Author:** ![bearrito](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/bearrito/32/1189_2.png) [@bearrito](https://openziti.discourse.group/u/bearrito)\
**Post date:** [August 10, 2023, 5:47pm UTC](https://openziti.discourse.group/t/blue-not-running-in-docker-compose-quickstart/1473/5 "2023-08-10T17:47:08Z")

</div>

Both,

Yep. I just recycled it a few times and eventually got a good deploy. Thanks for the extremely fast feedback I appreciate it.

I have an unrelated question, but I’m attempting to prove out something quickly within the quickstart.

My OpenZiti use in the past has been against fairly standard services. Http/Websockers/GRPC etc.

I have an app with lots of dynamic ports (please dont ask 😉 ). Only in the context of the quickstart, if add that app alongside blue web server. Is there anything I need to do policy wise or is the quickstart fairly open?

---

<div class="post-metadata">

**Author:** ![qrkourier](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/qrkourier/32/52_2.png) [@qrkourier](https://openziti.discourse.group/u/qrkourier)\
**Post date:** [August 10, 2023, 7:37pm UTC](https://openziti.discourse.group/t/blue-not-running-in-docker-compose-quickstart/1473/6 "2023-08-10T19:37:48Z")

</div>

Your quick start network can provide any type of Ziti service and Ziti config. You need a Ziti config for your service that describes the range of ports the app might need to allocate dynamically. I’ll whip up an example, but the concept is a config with a port range on the client/dialing side and a config set to forward the same port on the server/hosting side, assuming you’re using tunnelers on both sides.

intercept config example [from doc](https://openziti.io/docs/learn/core-concepts/config-store/config-type-intercept-v1#examples)

```json
{
    "addresses": [
        "acme.ziti",
        "*.dazzle.acme.ziti",
        "10.0.0.0/8"
    ],
    "portRanges": [
        {
            "low": 1025,
            "high": 1999
        }
    ],
    "protocols": [
        "tcp"
    ]
}

```

host config [from doc](https://openziti.io/docs/learn/core-concepts/config-store/config-type-host-v1#examples)

```json
{
    "address": "localhost",
    "forwardPort": true,
    "allowedPortRanges": [ 
        {
             "low": 1025, 
             "high": 1999
         }
    ],
    "protocol": "tcp"
}

```

Taken together, these configs means that a range of TCP ports are allowed for any destination that matches the list of intercept addresses.

The host config example always sends the packets to the matching port on address `localhost`.

---

<div class="post-metadata">

**Author:** ![bearrito](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/bearrito/32/1189_2.png) [@bearrito](https://openziti.discourse.group/u/bearrito)\
**Post date:** [August 11, 2023, 6:14pm UTC](https://openziti.discourse.group/t/blue-not-running-in-docker-compose-quickstart/1473/7 "2023-08-11T18:14:20Z")

</div>

@qrkourier

Thanks. I think I'm very/almost/not quite there. I appreciate the help so far.  
I largely followed the steps at [Your First Service | OpenZiti](https://openziti.io/docs/learn/quickstarts/services/ztha#http-server-identity)  
I _think/guess_ my issues are now related to my system Ubuntu 20 and DNS

Primary Questions:

1. How can I access the new service using the overlay
2. Can I verify with logs I am in fact using the overlay and not routing user underlay (maybe this is paranoid)

Couple points of clarification.

1. I am attempting to access via the overlay. That's probably obvious just pointing it out.
2. I deployed a container with my app.
3. I can exec into the container. I see that my "control plane port" is open

> root@88763fe0d946:/# netstat -nltp  
> Active Internet connections (only servers)  
> Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name  
> tcp 0 0 0.0.0.0:11311 0.0.0.0:\* LISTEN 32/python3

1. Under the red network, I aliased my app as my-app-red

2. I created the tunneler on my client machine. I can see relevant logs

3. If I attempt to use the overlay name. Doesn't work

> nc -zv my-app-red 11311  
> nc: getaddrinfo for host "my-app-red" port 11311: Temporary failure in name resolution  
> INFO tunnel-cbs:ziti\_dns.c:296 new\_ipv4\_entry() registered DNS entry my-app-red -\> 100.64.0.3

1. Using the underlay does work

> nc -zv 100.64.0.3 11311  
> Connection to 100.64.0.3 11311 port [tcp/\*] succeeded!

---

<div class="post-metadata">

**Author:** ![qrkourier](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/qrkourier/32/52_2.png) [@qrkourier](https://openziti.discourse.group/u/qrkourier)\
**Post date:** [August 11, 2023, 6:36pm UTC](https://openziti.discourse.group/t/blue-not-running-in-docker-compose-quickstart/1473/8 "2023-08-11T18:36:42Z")

</div>

You do seem to be quite close to a working Ziti service and I agree it looks like DNS isn’t set up on your Ubuntu 20.\* host. Are you running a stock DNS configuration or is it customized, if you happen to know?

I believe you are running `ziti-edge-tunnel` on the Ubuntu host as a tunneler/proxy. If it’s running then it should have created a `tun0` device and started listening for DNS queries. You might have to choose a Ziti service address with at least one stop character `.` to ensure the OS recognizes it as a domain name so it can send the query to the Ziti nameserver.

Is Ziti’s tun interface UP?

```auto
$ ip link sh up tun0
19: tun0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UNKNOWN mode DEFAULT group default qlen 500 link/none 

```

What is Ziti’s nameserver IP?

```auto
$ resolvectl --interface=tun0 dns
Link 19 (tun0): 100.64.0.2

```

Is Ziti DNS answering?

```auto
$ nslookup my-app-red 100.64.0.2
Server: 100.64.0.2
Address: 100.64.0.2#53

Non-authoritative answer:
Name: my-app-red
Address: 100.64.0.3

```

Is my Ziti service working? If my-app-red were an SSH server:

```auto
$ ncat 100.64.0.3 22 </dev/null
SSH-2.0-OpenSSH_7.4

```

Is systemd-resolved configured to try all queries on the Ziti nameserver?

```auto
# Ziti resolver has wildcard routing domain
$ resolvectl --interface=tun0 domain
Link 24 (tun0): ~.

```

Is Linux configured to use systemd-resolved?

```auto
# this symlink is one of the ways systemd-resolved can be enabled
$ ls -l /etc/resolv.conf 
lrwxrwxrwx 1 root root 37 Mar 1 2021 /etc/resolv.conf -> /run/systemd/resolve/stub-resolv.conf

```

Re: “stock” vs “custom” DNS  
I’m peeking to see how Ubuntu 20 (20.04 Focal?) stock DNS is set up. I’m pretty sure it’s using NetworkManager, but unsure if systemd-resolved or dnsmasq are enabled.

Ultimately, you have to configure Linux to try the Ziti nameserver first then fall back to a recursive/global/next nameserver.

Here’s how a fresh Ubuntu 20.04 ([Vagrant box](https://github.com/openziti-test-kitchen/vagrant/blob/main/ubuntu-20-focal-ziti-edge-tunnel-deb-repo/Vagrantfile)) was set up.

```bash
vagrant@ubuntu2004:~$ ll /etc/resolv.conf
lrwxrwxrwx 1 root root 39 Mar 29 22:19 /etc/resolv.conf -> ../run/systemd/resolve/stub-resolv.conf

vagrant@ubuntu2004:~$ grep nameserver /etc/resolv.conf
nameserver 127.0.0.53

vagrant@ubuntu2004:~$ sudo ss -lnup|grep 127.0.0.53
UNCONN 0 0 127.0.0.53%lo:53 0.0.0.0:* users:(("systemd-resolve",pid=505,fd=12))                                      

vagrant@ubuntu2004:~$ resolvectl --interface=tun0 dns
Link 3 (tun0): 100.64.0.2

# no wildcard, but it works anyway?
vagrant@ubuntu2004:~$ resolvectl --interface=tun0 domain
Link 3 (tun0):

vagrant@ubuntu2004:~$ nslookup httpbin.miniziti
Server: 127.0.0.53
Address: 127.0.0.53#53

Non-authoritative answer:
Name: httpbin.miniziti
Address: 100.64.0.3

vagrant@ubuntu2004:~$ curl http://httpbin.miniziti/ip
{"origin":"ziti-edge-router connId=2147483652, logical=ziti-sdk[router=tls://miniziti-router.192.168.49.2.sslip.io:443]"}

```

---

<div class="post-metadata">

**Author:** ![bearrito](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/bearrito/32/1189_2.png) [@bearrito](https://openziti.discourse.group/u/bearrito)\
**Post date:** [August 15, 2023, 5:30pm UTC](https://openziti.discourse.group/t/blue-not-running-in-docker-compose-quickstart/1473/9 "2023-08-15T17:30:48Z")

</div>

@qrkourier

I think my initial problem was a mismatch between the aliases I was using in docker compose. I sorted that out. But something is still wonky, I'm guessing this OS specific . Seems like some entries aren't respected?

Am I correct in that creating a host config is when DNS entries get created?

Here is what's going on.

> ros-offline-mapping-red:  
> image: my local-image: latests  
> hostname: offline-mapping.lan  
> expose:  
> - 32768-60999  
> - 11311  
> networks:  
> zitired:  
> aliases:  
> - offline-mapping  
> - offline-mapping.lan  
> - offline-mapping-red  
> - offline-mapping.red  
> - offline.mapping.red

> ziti edge create config ros.intercept.v1 intercept.v1 '{"addresses": ["offline-mapping.red", "offline-mapping", "offline-mapping.lan"],"portRanges": [{"low": 32768,"high": 60999}, {"low": 11311,"high": 11311}],"protocols": ["tcp"]}'

> ziti edge create config ros.host.v1 host.v1 '{"allowedAddresses": ["offline-mapping.red", "offline-mapping", "offline-mapping.lan"],"forwardAddress": true ,"forwardPort": true,"allowedPortRanges": [{"low": 32768,"high": 60999},{"low": 11311,"high": 11311}],"protocol": "tcp"}'

> $ nslookup offline-mapping.red 100.64.0.2  
> Server: 100.64.0.2  
> Address: 100.64.0.2#53  
> Non-authoritative answer:  
> Name: offline-mapping.red  
> Address: 100.64.0.8

> $ nslookup offline-mapping.lan 100.64.0.2  
> Server: 100.64.0.2  
> Address: 100.64.0.2#53  
> Non-authoritative answer:  
> Name: offline-mapping.lan  
> Address: 100.64.0.6

> nc -zv offline-mapping.lan 11311  
> nc: getaddrinfo for host "offline-mapping.lan" port 11311: Name or service not known

> nc -zv offline-mapping.red 11311  
> Connection to offline-mapping.red 11311 port [tcp/\*] succeeded!

# HOST DETAILS

> ll /etc/resolv.conf  
> lrwxrwxrwx 1 root root 39 Feb 17 12:34 /etc/resolv.conf -\> ../run/systemd/resolve/stub-resolv.conf

> grep nameserver /etc/resolv.conf  
> nameserver 127.0.0.53

> sudo ss -lnup|grep 127.0.0.53  
> UNCONN 0 0 127.0.0.53%lo:53 0.0.0.0:\* users:(("systemd-resolve",pid=1524,fd=12))

> resolvectl --interface=tun0 dns  
> Link 2533 (tun0): 100.64.0.2

> resolvectl --interface=tun0 domain  
> Link 2533 (tun0): ~.

---

<div class="post-metadata">

**Author:** ![qrkourier](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/qrkourier/32/52_2.png) [@qrkourier](https://openziti.discourse.group/u/qrkourier)\
**Post date:** [August 22, 2023, 4:43am UTC](https://openziti.discourse.group/t/blue-not-running-in-docker-compose-quickstart/1473/10 "2023-08-22T04:43:49Z")

</div>

The issue seems to be that only one of the two allowed intercept addresses is actually working, despite the tunneler's nameserver successfully resolving both to distinct IP addresses.

I too see distinct IP addresses for multiple allowed addresses on the same Ziti service, but I'm able to connect to the service with either address.

You host details look good too. I can see the tunneler was able to auto-configure `systemd-resolved`.

> [@bearrito](#):
>
> Am I correct in that creating a host config is when DNS entries get created?

A DNS record for each address from the intercept config will appear in the tunneler's nameserver a few seconds after a Ziti service is authorized. The host config has no bearing on the availability of the DNS record because it's based solely on the intercept config.

Did you happen to figure out why getaddrinfo for host “offline-mapping.lan" failed despite the DNS record being available according to nslookup? I couldn't trigger the same problem with cURL or Netcat.