Blue not running in Docker Compose Quickstart

Building/Development
1

Hello,

I'm attempting to run the Docker Compose quickstart guide. I downloaded both the compose file and .env file.

Looking at

ziti edge list edge-routers

I could see that Blue was missing

The logs show

ziti-private-blue_1 | [ 0.000] ERROR ziti/ziti/router.run: {error=[unable to load identity (open /persistent/ziti-private-blue.key: no such file or directory)] os=[linux] version=[v0.29.0] revision=[3ca2dd2f4e7b] configFile=[/persistent/ziti-private-blue.yaml] arch=[amd64] build-date=[2023-07-13T15:53:37Z] go-version=[go1.20.5]} error loading ziti-router config
open-ziti_ziti-controller-init-container_1 exited with code 0
ziti-private-blue_1 | panic: unable to load identity (open /persistent/ziti-private-blue.key: no such file or directory)

Red runs fine.

2

Welcome to the fun @bearrito :slightly_smiling_face:

I couldn’t get the same result, so I’m trying to think of what could be different in your environment. My blue router seems happy with the latest. Any chance you have an old container image in your local cache? docker compose pull will freshen if needed.

$ ziti edge list edge-routers
╭────────────┬───────────────────────┬────────┬───────────────┬──────┬─────────────╮
│ ID         │ NAME                  │ ONLINE │ ALLOW TRANSIT │ COST │ ATTRIBUTES  │
├────────────┼───────────────────────┼────────┼───────────────┼──────┼─────────────┤
│ 0e3oC6cBCg │ ziti-edge-router      │ true   │ true          │    0 │ public      │
│ 5ceBC6cBCg │ ziti-fabric-router-br │ true   │ true          │    0 │ fabric-only │
│ AfaBC6voCZ │ ziti-private-blue     │ true   │ true          │    0 │ zitiblue    │
│ tzLok-vBC  │ ziti-edge-router-wss  │ true   │ true          │    0 │ public      │
│ zMeBk-cBCg │ ziti-private-red      │ true   │ true          │    0 │ zitired     │
╰────────────┴───────────────────────┴────────┴───────────────┴──────┴─────────────╯
results: 1-5 of 5
3

@bearrito There is a race condition I’m working on a fix for, it’s possible you are experiencing this issue. Can you try deploying the compose environment again and see if it’s a recurring issue with just blue or if it intermittently happens with others?

4

I made sure I do actually have that key file.

$ docker compose exec ziti-private-blue ls -lh /persistent/ziti-private-blue.key 
-rw------- 1 ziti ziti 3.2K Aug 10 17:29 /persistent/ziti-private-blue.key

If you need to destroy the data in the quickstart project to try again:

docker compose down --volumes
5

Both,

Yep. I just recycled it a few times and eventually got a good deploy. Thanks for the extremely fast feedback I appreciate it.

I have an unrelated question, but I’m attempting to prove out something quickly within the quickstart.

My OpenZiti use in the past has been against fairly standard services. Http/Websockers/GRPC etc.

I have an app with lots of dynamic ports (please dont ask :wink: ). Only in the context of the quickstart, if add that app alongside blue web server. Is there anything I need to do policy wise or is the quickstart fairly open?

6

Your quick start network can provide any type of Ziti service and Ziti config. You need a Ziti config for your service that describes the range of ports the app might need to allocate dynamically. I’ll whip up an example, but the concept is a config with a port range on the client/dialing side and a config set to forward the same port on the server/hosting side, assuming you’re using tunnelers on both sides.

intercept config example from doc

{
    "addresses": [
        "acme.ziti",
        "*.dazzle.acme.ziti",
        "10.0.0.0/8"
    ],
    "portRanges": [
        {
            "low": 1025,
            "high": 1999
        }
    ],
    "protocols": [
        "tcp"
    ]
}

host config from doc

{
    "address": "localhost",
    "forwardPort": true,
    "allowedPortRanges": [ 
        {
             "low": 1025, 
             "high": 1999
         }
    ],
    "protocol": "tcp"
}

Taken together, these configs means that a range of TCP ports are allowed for any destination that matches the list of intercept addresses.

The host config example always sends the packets to the matching port on address localhost.

7

@qrkourier

Thanks. I think I'm very/almost/not quite there. I appreciate the help so far.
I largely followed the steps at Your First Service | OpenZiti
I think/guess my issues are now related to my system Ubuntu 20 and DNS

Primary Questions:

  1. How can I access the new service using the overlay
  2. Can I verify with logs I am in fact using the overlay and not routing user underlay (maybe this is paranoid)

Couple points of clarification.

  1. I am attempting to access via the overlay. That's probably obvious just pointing it out.
  2. I deployed a container with my app.
  3. I can exec into the container. I see that my "control plane port" is open

root@88763fe0d946:/# netstat -nltp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:11311 0.0.0.0:* LISTEN 32/python3

  1. Under the red network, I aliased my app as my-app-red

  2. I created the tunneler on my client machine. I can see relevant logs

  3. If I attempt to use the overlay name. Doesn't work

nc -zv my-app-red 11311
nc: getaddrinfo for host "my-app-red" port 11311: Temporary failure in name resolution
INFO tunnel-cbs:ziti_dns.c:296 new_ipv4_entry() registered DNS entry my-app-red -> 100.64.0.3

  1. Using the underlay does work

nc -zv 100.64.0.3 11311
Connection to 100.64.0.3 11311 port [tcp/*] succeeded!

8

You do seem to be quite close to a working Ziti service and I agree it looks like DNS isn’t set up on your Ubuntu 20.* host. Are you running a stock DNS configuration or is it customized, if you happen to know?

I believe you are running ziti-edge-tunnel on the Ubuntu host as a tunneler/proxy. If it’s running then it should have created a tun0 device and started listening for DNS queries. You might have to choose a Ziti service address with at least one stop character . to ensure the OS recognizes it as a domain name so it can send the query to the Ziti nameserver.

Is Ziti’s tun interface UP?

$ ip link sh up tun0
19: tun0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UNKNOWN mode DEFAULT group default qlen 500    link/none

What is Ziti’s nameserver IP?

$ resolvectl --interface=tun0 dns
Link 19 (tun0): 100.64.0.2

Is Ziti DNS answering?

$ nslookup my-app-red 100.64.0.2
Server:         100.64.0.2
Address:        100.64.0.2#53

Non-authoritative answer:
Name:   my-app-red
Address: 100.64.0.3

Is my Ziti service working? If my-app-red were an SSH server:

$ ncat 100.64.0.3 22 </dev/null
SSH-2.0-OpenSSH_7.4

Is systemd-resolved configured to try all queries on the Ziti nameserver?

# Ziti resolver has wildcard routing domain
$ resolvectl --interface=tun0 domain
Link 24 (tun0): ~.

Is Linux configured to use systemd-resolved?

# this symlink is one of the ways systemd-resolved can be enabled
$ ls -l /etc/resolv.conf 
lrwxrwxrwx 1 root root 37 Mar  1  2021 /etc/resolv.conf -> /run/systemd/resolve/stub-resolv.conf

Re: “stock” vs “custom” DNS
I’m peeking to see how Ubuntu 20 (20.04 Focal?) stock DNS is set up. I’m pretty sure it’s using NetworkManager, but unsure if systemd-resolved or dnsmasq are enabled.

Ultimately, you have to configure Linux to try the Ziti nameserver first then fall back to a recursive/global/next nameserver.

Here’s how a fresh Ubuntu 20.04 (Vagrant box) was set up.

vagrant@ubuntu2004:~$ ll /etc/resolv.conf
lrwxrwxrwx 1 root root 39 Mar 29 22:19 /etc/resolv.conf -> ../run/systemd/resolve/stub-resolv.conf

vagrant@ubuntu2004:~$ grep nameserver /etc/resolv.conf
nameserver 127.0.0.53

vagrant@ubuntu2004:~$ sudo ss -lnup|grep 127.0.0.53
UNCONN   0        0                 127.0.0.53%lo:53             0.0.0.0:*       users:(("systemd-resolve",pid=505,fd=12))                                      

vagrant@ubuntu2004:~$ resolvectl --interface=tun0 dns
Link 3 (tun0): 100.64.0.2

# no wildcard, but it works anyway?
vagrant@ubuntu2004:~$ resolvectl --interface=tun0 domain
Link 3 (tun0):

vagrant@ubuntu2004:~$ nslookup httpbin.miniziti
Server:         127.0.0.53
Address:        127.0.0.53#53

Non-authoritative answer:
Name:   httpbin.miniziti
Address: 100.64.0.3

vagrant@ubuntu2004:~$ curl http://httpbin.miniziti/ip
{"origin":"ziti-edge-router connId=2147483652, logical=ziti-sdk[router=tls://miniziti-router.192.168.49.2.sslip.io:443]"}
9

@qrkourier

I think my initial problem was a mismatch between the aliases I was using in docker compose. I sorted that out. But something is still wonky, I'm guessing this OS specific . Seems like some entries aren't respected?

Am I correct in that creating a host config is when DNS entries get created?

Here is what's going on.

ros-offline-mapping-red:
image: my local-image: latests
hostname: offline-mapping.lan
expose:
- 32768-60999
- 11311
networks:
zitired:
aliases:
- offline-mapping
- offline-mapping.lan
- offline-mapping-red
- offline-mapping.red
- offline.mapping.red

ziti edge create config ros.intercept.v1 intercept.v1 '{"addresses": ["offline-mapping.red", "offline-mapping", "offline-mapping.lan"],"portRanges": [{"low": 32768,"high": 60999}, {"low": 11311,"high": 11311}],"protocols": ["tcp"]}'

ziti edge create config ros.host.v1 host.v1 '{"allowedAddresses": ["offline-mapping.red", "offline-mapping", "offline-mapping.lan"],"forwardAddress": true ,"forwardPort": true,"allowedPortRanges": [{"low": 32768,"high": 60999},{"low": 11311,"high": 11311}],"protocol": "tcp"}'

$ nslookup offline-mapping.red 100.64.0.2
Server: 100.64.0.2
Address: 100.64.0.2#53
Non-authoritative answer:
Name: offline-mapping.red
Address: 100.64.0.8

$ nslookup offline-mapping.lan 100.64.0.2
Server: 100.64.0.2
Address: 100.64.0.2#53
Non-authoritative answer:
Name: offline-mapping.lan
Address: 100.64.0.6

nc -zv offline-mapping.lan 11311
nc: getaddrinfo for host "offline-mapping.lan" port 11311: Name or service not known

nc -zv offline-mapping.red 11311
Connection to offline-mapping.red 11311 port [tcp/*] succeeded!

HOST DETAILS

ll /etc/resolv.conf
lrwxrwxrwx 1 root root 39 Feb 17 12:34 /etc/resolv.conf -> ../run/systemd/resolve/stub-resolv.conf

grep nameserver /etc/resolv.conf
nameserver 127.0.0.53

sudo ss -lnup|grep 127.0.0.53
UNCONN 0 0 127.0.0.53%lo:53 0.0.0.0:* users:(("systemd-resolve",pid=1524,fd=12))

resolvectl --interface=tun0 dns
Link 2533 (tun0): 100.64.0.2

resolvectl --interface=tun0 domain
Link 2533 (tun0): ~.

10

The issue seems to be that only one of the two allowed intercept addresses is actually working, despite the tunneler's nameserver successfully resolving both to distinct IP addresses.

I too see distinct IP addresses for multiple allowed addresses on the same Ziti service, but I'm able to connect to the service with either address.

You host details look good too. I can see the tunneler was able to auto-configure systemd-resolved.

A DNS record for each address from the intercept config will appear in the tunneler's nameserver a few seconds after a Ziti service is authorized. The host config has no bearing on the availability of the DNS record because it's based solely on the intercept config.

Did you happen to figure out why getaddrinfo for host “offline-mapping.lan" failed despite the DNS record being available according to nslookup? I couldn't trigger the same problem with cURL or Netcat.