# Help designing OpenZiti network

**URL:** <https://openziti.discourse.group/t/help-designing-openziti-network/3486>\
**Category:** Uncategorized\
**Created:** [November 26, 2024, 12:10am UTC](https://openziti.discourse.group/t/help-designing-openziti-network/3486 "2024-11-26T00:10:14Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![bivalve.felt](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/bivalve.felt/32/2792_2.png) [@bivalve.felt](https://openziti.discourse.group/u/bivalve.felt)\
**Post date:** [November 26, 2024, 12:10am UTC](https://openziti.discourse.group/t/help-designing-openziti-network/3486/1 "2024-11-26T00:10:14Z")

</div>

Hello!

I think OpenZiti is really cool and I want to build something beyond the docker compose quickstart. I have a couple machines running macOS and I want to create an OpenZiti network with a service running the [open-webui](https://github.com/open-webui/open-webui) for ollama in docker compose, so that others in my network can access it. I am here because I got stuck trying to set it up 😃

Below is the basic idea of what I want to do.

 ![Screenshot 2024-11-25 at 4.47.58 PM](https://global.discourse-cdn.com/free1/uploads/netfoundry/original/2X/5/556e414adc082107a14775e0f4ea5e02222a0cf9.png)

I already have a controller set up in AWS and am able to access the admin console.

What I am not sure how to do is set up the service machine in docker compose such that the docker tunneler gives access to open-webui also running in the same docker compose.

I started by trying to following [these instructions](https://openziti.io/docs/reference/tunnelers/docker/#enroll-and-persist-identity-in-a-volume) to set up the tunneler using the `ZITI_ENROLL_TOKEN` but the container couldn't initialize properly and just kept restarting citing the following errors:

```auto
2024-11-25 17:04:33 ERROR: failed to enroll with token from /ziti-edge-tunnel/ziti_id.jwt (1033B)
2024-11-25 17:04:40 WARN: clobbering non-empty Ziti enrollment token file /ziti-edge-tunnel/ziti_id.jwt with contents of env var ZITI_ENROLL_TOKEN
2024-11-25 17:04:40 (7)[0.000] INFO ziti-sdk:utils.c:198 ziti_log_set_level() set log level: root=3/INFO
2024-11-25 17:04:40 (7)[0.000] INFO ziti-sdk:utils.c:169 ziti_log_init() Ziti C SDK version 1.2.0 @g6f3ae93(HEAD) starting at (2024-11-26T00:04:40.080)
2024-11-25 17:04:40 (7)[0.000] INFO ziti-sdk:ziti_enroll.c:141 ziti_enroll() Ziti C SDK version 1.2.0 @g6f3ae93(HEAD) starting enrollment at (2024-11-26T00:04:40.080)

2024-11-25 17:05:18 DEBUG: waiting 3s for /ziti-edge-tunnel/ziti_id.json (or token) to appear
2024-11-25 17:05:18 DEBUG: identity file /ziti-edge-tunnel/ziti_id.json not found

```

---

<div class="post-metadata">

**Author:** ![TheLumberjack](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/thelumberjack/32/113_2.png) [@TheLumberjack](https://openziti.discourse.group/u/TheLumberjack)\
**Post date:** [November 26, 2024, 2:27am UTC](https://openziti.discourse.group/t/help-designing-openziti-network/3486/2 "2024-11-26T02:27:18Z")

</div>

I'll follow up here on your other thread, if that's ok? You [started it here](https://openziti.discourse.group/t/add-identity-to-new-conatiner-in-docker-compose-quickstart/2840/13), Ken replied and you replied back... If you don't mind, let's pick it up from here.

Sorry about Discourse blocking your post! I have cleared the "is this spam" flag. If you write your post "too fast" Discourse thinks you're a bot. I understand though, I often compose my text in a different/bigger window first too...

As to that question you asked over there, that i think is relevant here, you had asked:

> Could you give an example of a docker compose yaml file demonstrating this?

referring to using one docker container as another containers 'network' and melding them together, kubernetes "pod-like".

It'll take me a bit but I'll cook up an example and follow up tonight or tomorrow. Welcome to OpenZiti and to the community! I'll follow up shortly

---

<div class="post-metadata">

**Author:** ![TheLumberjack](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/thelumberjack/32/113_2.png) [@TheLumberjack](https://openziti.discourse.group/u/TheLumberjack)\
**Post date:** [November 26, 2024, 2:40am UTC](https://openziti.discourse.group/t/help-designing-openziti-network/3486/3 "2024-11-26T02:40:23Z")

</div>

> [@Add identity to new conatiner in docker-compose quickstart](https://openziti.discourse.group/t/add-identity-to-new-conatiner-in-docker-compose-quickstart/2840/15):
>
> I discovered that I cannot even register an identity with my macOS tunneler to my controller, so I think my controller configuration might be off?

I was rereading your post and this caught my eye. Yes, this makes me think the deployment has a problem. Generally, this is caused by not having the proper variables set when doing the "Host It Anywhere" installation. Notably these variables:

 ![image](https://global.discourse-cdn.com/free1/uploads/netfoundry/original/2X/2/2403845aad4f7b3e6cb65d9ddf25d94777a95784.png)

The `ziti` CLI has built into it a mechanism to verify your network for traffic. Can you run the command from the mac?

First, use the ziti CLI and authenticate to your controller from the mac:

```auto
ziti edge login "${ZITI_EDGE_CONTROLLER_API}" -u "${ZITI_USER-}" -p "${ZITI_PWD}" -y

```

Then run,

```auto
ziti ops verify-traffic

```

You'll see something that looks like this. Let's make sure that works and succeeds

 ![image](https://global.discourse-cdn.com/free1/uploads/netfoundry/original/2X/c/ce331edb4da97566520895ea01377ea09c2a54ba.png)

---

<div class="post-metadata">

**Author:** ![bivalve.felt](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/bivalve.felt/32/2792_2.png) [@bivalve.felt](https://openziti.discourse.group/u/bivalve.felt)\
**Post date:** [November 26, 2024, 3:19pm UTC](https://openziti.discourse.group/t/help-designing-openziti-network/3486/4 "2024-11-26T15:19:21Z")

</div>

How do you install the cli without setting up a local network? That is one of the things that has confused me a bit too. It seems like the cli is useful for configuring any ziti network that I could authenticate to, but the installation docs that I have seen all show that installing the cli includes setting up a local test network, which I may not want every time.

---

<div class="post-metadata">

**Author:** ![TheLumberjack](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/thelumberjack/32/113_2.png) [@TheLumberjack](https://openziti.discourse.group/u/TheLumberjack)\
**Post date:** [November 26, 2024, 3:26pm UTC](https://openziti.discourse.group/t/help-designing-openziti-network/3486/5 "2024-11-26T15:26:25Z")

</div>

> [@bivalve.felt](#):
>
> How do you install the cli without setting up a local network?

Oh there are a myriad of ways. Ranging from "you don't", just download the exe from the releases page and run it, to install the package for your os if you're on linux.

The root download page is at [Downloads | OpenZiti](https://openziti.io/docs/downloads)

You're on macOS, we don't have a brew command for that yet. You can source the `ziti-cli-functions.sh` helper script if you want (standard disclosure: you should read the script first, as a best-practice of course)

Then you can run `getZiti`. for example, I often will run:

```auto
source /dev/stdin <<< "$(wget -qO- https://get.openziti.io/ziti-cli-functions.sh)"; getZiti yes

```

You'll see something like:

 ![image](https://global.discourse-cdn.com/free1/uploads/netfoundry/original/2X/b/b67363387379d1c88349d971a0f7e244dd64e3ef.png)

Then ziti is on your path for that session...  
 ![image](https://global.discourse-cdn.com/free1/uploads/netfoundry/original/2X/a/ad14fa617985449665b34a13b02c13b6ef1cc09f.png)

For windows, there's no choco/scoop/winget package yet but you can run a powershell:

```auto
iex(iwr -Uri https://get.openziti.io/quick/getZiti.ps1)

```

For linux, you can run the same:

```auto
source /dev/stdin <<< "$(wget -qO- https://get.openziti.io/ziti-cli-functions.sh)"; getZiti yes

```

or you can install it as a package an manage it that way [Downloads | OpenZiti](https://openziti.io/docs/downloads?os=Linux)

---

<div class="post-metadata">

**Author:** ![bivalve.felt](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/bivalve.felt/32/2792_2.png) [@bivalve.felt](https://openziti.discourse.group/u/bivalve.felt)\
**Post date:** [November 26, 2024, 6:01pm UTC](https://openziti.discourse.group/t/help-designing-openziti-network/3486/6 "2024-11-26T18:01:46Z")

</div>

Thanks! Yeah, a brew command would be awesome.

> [@TheLumberjack](#):
>
> First, use the ziti CLI and authenticate to your controller from the mac:
> 
> ```auto
> ziti edge login "${ZITI_EDGE_CONTROLLER_API}" -u "${ZITI_USER-}" -p "${ZITI_PWD}" -y
> 
> ```

Okay, so I tried the above and it failed with a bunch of errors like this:

```auto
RESTY 2024/11/26 10:46:10 ERROR Post "https://ec2-3-142-189-241.us-east-2.compute.amazonaws.com:8441/edge/management/v1/identities/KCRBwLJ09/authenticate?method=password": tls: failed to verify certificate: x509: certificate is valid for ip-172-31-21-181.us-east-2.compute.internal, localhost, not ec2-3-142-189-241.us-east-2.compute.amazonaws.com, Attempt 1

```

I think that somewhere in the expressInstall script, it is using `hostname` instead of the `EXTERNAL_DNS` value I set.

`hostname` produces the `ip-172-31-21-181.us-east-2.compute.internal`

but my public DNS in AWS is `ec2-3-142-189-241.us-east-2.compute.amazonaws.com`

---

<div class="post-metadata">

**Author:** ![TheLumberjack](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/thelumberjack/32/113_2.png) [@TheLumberjack](https://openziti.discourse.group/u/TheLumberjack)\
**Post date:** [November 26, 2024, 6:04pm UTC](https://openziti.discourse.group/t/help-designing-openziti-network/3486/7 "2024-11-26T18:04:31Z")

</div>

I'm confident you didn't set the environment variables correctly. I would say "start over". it's quick enough...

You can reset the quickstart pretty easily...

> **[Reset the Quickstart | OpenZiti](https://openziti.io/docs/learn/quickstarts/network/help/reset-quickstart/)**
>
> You may want to re-run expressInstall with different parameters or because a readiness check failed. You could run it again with a new ZITI\_HOME without changing the current installation or delete the directory and begin again with these steps:

Those instructions should also start with "if you enabled systemd, stop the units: sudo systemctl stop ziti-controller; sudo systemctl stop ziti-router"

---

<div class="post-metadata">

**Author:** ![bivalve.felt](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/bivalve.felt/32/2792_2.png) [@bivalve.felt](https://openziti.discourse.group/u/bivalve.felt)\
**Post date:** [November 26, 2024, 6:08pm UTC](https://openziti.discourse.group/t/help-designing-openziti-network/3486/8 "2024-11-26T18:08:37Z")

</div>

Ok, I nuked my install. However, I am still not sure what to do about the `EXTERNAL_DNS`. In my AWS console, they give me a public dns that is slightly different from what `hostname` produces like I commented above. Which one do I use?

---

<div class="post-metadata">

**Author:** ![TheLumberjack](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/thelumberjack/32/113_2.png) [@TheLumberjack](https://openziti.discourse.group/u/TheLumberjack)\
**Post date:** [November 26, 2024, 6:12pm UTC](https://openziti.discourse.group/t/help-designing-openziti-network/3486/9 "2024-11-26T18:12:10Z")

</div>

I should have expanded... You really need to restart. The setting of the PKI is a vital first step. If it's not done correctly, it all needs to be redone and starting over is the easiest way...

> [@bivalve.felt](#):
>
> However, I am still not sure what to do about the `EXTERNAL_DNS`

Since you're using AWS it's actually quite easy. AWS provides a handy url you can curl to set it:

```auto
export EXTERNAL_DNS="$(curl -sk http://169.254.169.254/latest/meta-data/public-hostname)"
echo "This server external DNS: $EXTERNAL_DNS"
. $HOME/.ziti/quickstart/$(hostname)/$(hostname).env

```

☝ I put that in all my AWS .bashrc files so the moment i log in it tells me:

```auto
Last login: Tue Nov 26 18:09:05 2024 from 67.246.244.61
This server external DNS: ec2-3-18-113-172.us-east-2.compute.amazonaws.com

adding /home/ubuntu/.ziti/quickstart/ip-172-31-47-200/ziti-bin/ziti-v1.1.15 to the path

```

---

<div class="post-metadata">

**Author:** ![TheLumberjack](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/thelumberjack/32/113_2.png) [@TheLumberjack](https://openziti.discourse.group/u/TheLumberjack)\
**Post date:** [November 26, 2024, 6:12pm UTC](https://openziti.discourse.group/t/help-designing-openziti-network/3486/10 "2024-11-26T18:12:45Z")

</div>

Or you can just

```auto
export EXTERNAL_DNS="ec2-3-142-189-241.us-east-2.compute.amazonaws.com"

```

too. 🙂

---

<div class="post-metadata">

**Author:** ![bivalve.felt](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/bivalve.felt/32/2792_2.png) [@bivalve.felt](https://openziti.discourse.group/u/bivalve.felt)\
**Post date:** [November 26, 2024, 6:20pm UTC](https://openziti.discourse.group/t/help-designing-openziti-network/3486/11 "2024-11-26T18:20:42Z")

</div>

This is what I had done the first time... so I am a little confused why it didn't work. I started off by doing the following, just like in the guide:

```bash
export EXTERNAL_DNS="ec2-3-142-189-241.us-east-2.compute.amazonaws.com"
export EXTERNAL_IP="$(curl -s eth0.me)"
export ZITI_CTRL_EDGE_IP_OVERRIDE="${EXTERNAL_IP}"
export ZITI_ROUTER_IP_OVERRIDE="${EXTERNAL_IP}"
export ZITI_CTRL_ADVERTISED_ADDRESS="${EXTERNAL_DNS:-${EXTERNAL_IP}}"
export ZITI_ROUTER_ADVERTISED_ADDRESS="${EXTERNAL_DNS:-${EXTERNAL_IP}}"
export ZITI_CTRL_ADVERTISED_PORT=8440
export ZITI_CTRL_EDGE_ADVERTISED_PORT=8441
export ZITI_ROUTER_PORT=8442

```

---

<div class="post-metadata">

**Author:** ![bivalve.felt](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/bivalve.felt/32/2792_2.png) [@bivalve.felt](https://openziti.discourse.group/u/bivalve.felt)\
**Post date:** [November 26, 2024, 6:24pm UTC](https://openziti.discourse.group/t/help-designing-openziti-network/3486/12 "2024-11-26T18:24:57Z")

</div>

Also, curling [http://169.254.169.254/latest/meta-data/public-hostname](http://169.254.169.254/latest/meta-data/public-hostname) doesn't return anything for me in my AWS instance. 😕

Edit: Ah, I just had to turn it on at Instance \> Actions \> Instance Settings \> Modify instance metadata options \> Instance metata data service enabled and IMDSv2 set to optional.

Not sure why that wasn't the default.

---

<div class="post-metadata">

**Author:** ![TheLumberjack](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/thelumberjack/32/113_2.png) [@TheLumberjack](https://openziti.discourse.group/u/TheLumberjack)\
**Post date:** [November 26, 2024, 6:32pm UTC](https://openziti.discourse.group/t/help-designing-openziti-network/3486/13 "2024-11-26T18:32:47Z")

</div>

Often, it's easy to exit a shell and forget you exited the shell, only to ssh back there expecting the variables to be set... but they aren't any more. I've done that countless times... It's equally easy to have more than one shell open and do a thing in one shell, move to the other and forget you moved... One of those two things would be my guess. Remove the environment and then run the quickstart again as you have it set. When it's done, enable/restart systemd if you are using it and then run the verify-traffic command again. It should pass.

> [@bivalve.felt](#):
>
> Also, curling [http://169.254.169.254/latest/meta-data/public-hostname](http://169.254.169.254/latest/meta-data/public-hostname) doesn't return anything for me in my AWS instance. 😕

That's a bummer! I wonder if it's only on certain instance types? I found the information back in the day from [Use instance metadata to manage your EC2 instance - Amazon Elastic Compute Cloud](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html) but I didn't see anything about it not being available. ChatGPT thinks perhap a network ACL blocked it. It's not important really, you can just set it manually like you did. It's just convinient imo 🙂

EDIT: I see you got the metadata sorted 🙂 cool

---

<div class="post-metadata">

**Author:** ![bivalve.felt](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/bivalve.felt/32/2792_2.png) [@bivalve.felt](https://openziti.discourse.group/u/bivalve.felt)\
**Post date:** [November 26, 2024, 6:51pm UTC](https://openziti.discourse.group/t/help-designing-openziti-network/3486/14 "2024-11-26T18:51:32Z")

</div>

Ok, I ran through the install process again from zero, making sure to maintain the same environment and variables, but when I try to run

```bash
ziti edge login "${ZITI_EDGE_CONTROLLER_API}" -u "${ZITI_USER-}" -p "${ZITI_PWD}" -y

```

I get the same error about the certs...

```bash
RESTY 2024/11/26 11:50:53 ERROR Post "https://ec2-3-142-189-241.us-east-2.compute.amazonaws.com:8441/edge/management/v1/identities/KCRBwLJ09/authenticate?method=password": tls: failed to verify certificate: x509: certificate is valid for ip-172-31-21-181.us-east-2.compute.internal, localhost, not ec2-3-142-189-241.us-east-2.compute.amazonaws.com, Attempt 1

```

---

<div class="post-metadata">

**Author:** ![TheLumberjack](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/thelumberjack/32/113_2.png) [@TheLumberjack](https://openziti.discourse.group/u/TheLumberjack)\
**Post date:** [November 26, 2024, 6:54pm UTC](https://openziti.discourse.group/t/help-designing-openziti-network/3486/15 "2024-11-26T18:54:10Z")

</div>

Strange.... Will you capture the full output from the expressInstall and DM them to me here on discourse? Do you maybe have two different instances running? Before running the expressInstall, can you verify there are no processes?

```auto
sudo ps -ef | grep ziti

```

Should return no results.

Maybe there's something going on with the linux version you're using, what version is the OS? I tend to use ubuntu for the OS, what are you using?

---

<div class="post-metadata">

**Author:** ![bivalve.felt](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/bivalve.felt/32/2792_2.png) [@bivalve.felt](https://openziti.discourse.group/u/bivalve.felt)\
**Post date:** [November 26, 2024, 6:55pm UTC](https://openziti.discourse.group/t/help-designing-openziti-network/3486/16 "2024-11-26T18:55:50Z")

</div>

I am using the default Amazon Linux.

---

<div class="post-metadata">

**Author:** ![TheLumberjack](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/thelumberjack/32/113_2.png) [@TheLumberjack](https://openziti.discourse.group/u/TheLumberjack)\
**Post date:** [November 26, 2024, 6:56pm UTC](https://openziti.discourse.group/t/help-designing-openziti-network/3486/17 "2024-11-26T18:56:13Z")

</div>

I'll try with that, gimme a few to make a VM and run it through...

---

<div class="post-metadata">

**Author:** ![bivalve.felt](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/bivalve.felt/32/2792_2.png) [@bivalve.felt](https://openziti.discourse.group/u/bivalve.felt)\
**Post date:** [November 26, 2024, 6:57pm UTC](https://openziti.discourse.group/t/help-designing-openziti-network/3486/18 "2024-11-26T18:57:25Z")

</div>

This is my output from the ps command **after** running expressInstall:

```bash
root 53497 1 0 18:36 ? 00:00:00 /home/ec2-user/.ziti/quickstart/ip-172-31-21-181.us-east-2.compute.internal/ziti-bin/ziti-v1.1.15/ziti router run /home/ec2-user/.ziti/quickstart/ip-172-31-21-181.us-east-2.compute.internal/ip-172-31-21-181.us-east-2.compute.internal-edge-router.yaml
root 53894 1 0 18:44 ? 00:00:02 /home/ec2-user/.ziti/quickstart/ip-172-31-21-181.us-east-2.compute.internal/ziti-bin/ziti-v1.1.15/ziti controller run /home/ec2-user/.ziti/quickstart/ip-172-31-21-181.us-east-2.compute.internal/ip-172-31-21-181.us-east-2.compute.internal.yaml
ec2-user 54300 54275 0 18:56 pts/3 00:00:00 grep --color=auto ziti

```

---

<div class="post-metadata">

**Author:** ![TheLumberjack](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/thelumberjack/32/113_2.png) [@TheLumberjack](https://openziti.discourse.group/u/TheLumberjack)\
**Post date:** [November 26, 2024, 7:37pm UTC](https://openziti.discourse.group/t/help-designing-openziti-network/3486/19 "2024-11-26T19:37:43Z")

</div>

@bivalve.felt DM'ed for a bit and this was a quickstart-related bug. It should be resolved with [fix quickstart doc by dovholuknf · Pull Request #986 · openziti/ziti-doc · GitHub](https://github.com/openziti/ziti-doc/pull/986)

The doc site should be rebuilding and deployed soon with the updates.

Thanks again @bivalve.felt

---

<div class="post-metadata">

**Author:** ![bivalve.felt](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/bivalve.felt/32/2792_2.png) [@bivalve.felt](https://openziti.discourse.group/u/bivalve.felt)\
**Post date:** [November 26, 2024, 11:18pm UTC](https://openziti.discourse.group/t/help-designing-openziti-network/3486/20 "2024-11-26T23:18:14Z")

</div>

Thanks for the help fixing my controller installation! I am back to trying to configure the network.

What is meant by "variable" [here](https://openziti.io/docs/learn/quickstarts/services/ztha#http-server-ipdns)? Is that an environment variable?

> This variable represents the location of the HTTP server relative to the OpenZiti tunneler you're using.

My http server is running in a docker compose project that defines a bridge network. For my tunneler, I am using the `openziti:/ziti-host:latest` image. If `http_server` from the linked doc is an environment variable, then do I just need to define it in the `compose.yaml` and provide the docker compose network identifier for my server as the value?

[Next page](https://openziti.discourse.group/t/help-designing-openziti-network/3486.md?page=2)