# Problem initializing HA cluster

**URL:** <https://openziti.discourse.group/t/problem-initializing-ha-cluster/3288>\
**Category:** Support\
**Created:** [October 21, 2024, 2:34pm UTC](https://openziti.discourse.group/t/problem-initializing-ha-cluster/3288 "2024-10-21T14:34:16Z")\
**Posts on this page:** 15\
**Page:** 1

<div class="post-metadata">

**Author:** ![mkuhlmann](https://avatars.discourse-cdn.com/v4/letter/m/5f9b8f/32.png) [@mkuhlmann](https://openziti.discourse.group/u/mkuhlmann)\
**Post date:** [October 21, 2024, 2:34pm UTC](https://openziti.discourse.group/t/problem-initializing-ha-cluster/3288/1 "2024-10-21T14:34:16Z")

</div>

Hello everyone,  
I was just trying to setup a HA cluster following [this guide](https://github.com/openziti/ziti/blob/v1.1.7/doc/ha/overview.md).  
The problem is with initializing the controllers.

I run this command:

> ziti agent controller init admin REDACTED MK

And I recieve this output:

> Error: no processes found matching filter, use 'ziti agent list' to list candidates  
> Usage:  
> ziti agent controller init [flags]
> 
> Flags:  
> -a, --app-alias string Alias of host application to talk to (specified in host application)  
> -i, --app-id string Id of host application to talk to (like controller or router id)  
> -t, --app-type string Type of host application to talk to (like controller or router)  
> -h, --help help for init  
> -p, --pid uint32 Process ID of host application to talk to  
> -n, --process-name string Process name of host application to talk to  
> --tcp-addr string Type of host application to talk to (like controller or router)  
> --timeout duration Operation timeout (default 5s)
> 
> no processes found matching filter, use 'ziti agent list' to list candidates

Im not sure what I messed up here. Any help is highly appreciated.

---

<div class="post-metadata">

**Author:** ![plorenz](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/plorenz/32/54_2.png) [@plorenz](https://openziti.discourse.group/u/plorenz)\
**Post date:** [October 21, 2024, 3:26pm UTC](https://openziti.discourse.group/t/problem-initializing-ha-cluster/3288/2 "2024-10-21T15:26:20Z")

</div>

Are you running the `ziti agent` command as the same user as the controller is running? If not, you may need to do `sudo -u <ziti user> ziti agent ...`.

Let me know if that helps.  
Paul

---

<div class="post-metadata">

**Author:** ![mkuhlmann](https://avatars.discourse-cdn.com/v4/letter/m/5f9b8f/32.png) [@mkuhlmann](https://openziti.discourse.group/u/mkuhlmann)\
**Post date:** [October 21, 2024, 3:34pm UTC](https://openziti.discourse.group/t/problem-initializing-ha-cluster/3288/3 "2024-10-21T15:34:20Z")

</div>

Thanks for your quick reply!  
Unfortunately, this yields the same output:

> sudo -u ziti-controller ziti agent controller init admin REDACTED MK  
> Error: no processes found matching filter, use 'ziti agent list' to list candidates  
> Usage:  
> ziti agent controller init [flags]
> 
> Flags:  
> -a, --app-alias string Alias of host application to talk to (specified in host application)  
> -i, --app-id string Id of host application to talk to (like controller or router id)  
> -t, --app-type string Type of host application to talk to (like controller or router)  
> -h, --help help for init  
> -p, --pid uint32 Process ID of host application to talk to  
> -n, --process-name string Process name of host application to talk to  
> --tcp-addr string Type of host application to talk to (like controller or router)  
> --timeout duration Operation timeout (default 5s)
> 
> no processes found matching filter, use 'ziti agent list' to list candidates

---

<div class="post-metadata">

**Author:** ![plorenz](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/plorenz/32/54_2.png) [@plorenz](https://openziti.discourse.group/u/plorenz)\
**Post date:** [October 21, 2024, 3:39pm UTC](https://openziti.discourse.group/t/problem-initializing-ha-cluster/3288/4 "2024-10-21T15:39:12Z")

</div>

Just to confirm, the controller is running? You should see something like this in the controller log:

```auto
[0.351] INFO ziti/controller/network.(*Network).Run: started
[1.986] WARNING github.com/hashicorp/raft.(*Raft).runFollower: no known peers, aborting election
[3.288] WARNING ziti/controller/server.(*Controller).checkEdgeInitialized: the Ziti Edge has not been initialized, no default admin exists. Please run 'ziti agent controller init' to configure the default admin'

```

The last message will be repeated on an interval until the controller is initialized.

Paul

---

<div class="post-metadata">

**Author:** ![mkuhlmann](https://avatars.discourse-cdn.com/v4/letter/m/5f9b8f/32.png) [@mkuhlmann](https://openziti.discourse.group/u/mkuhlmann)\
**Post date:** [October 21, 2024, 3:41pm UTC](https://openziti.discourse.group/t/problem-initializing-ha-cluster/3288/5 "2024-10-21T15:41:21Z")

</div>

> [@plorenz](#):
>
> confirm, the controller is running? You should see something lik

I think so, yes. This is the latest line in the journal:

> Oct 21 17:39:51 REDACTED ziti[14897]: {"file":"[github.com/openziti/ziti/controller/server/controller.go:294","func":"github.com/openziti/ziti/controller/server.(\*Controller).checkEdgeInitialized","level":"warning","msg":"the](http://github.com/openziti/ziti/controller/server/controller.go:294%22,%22func%22:%22github.com/openziti/ziti/controller/server.(*Controller).checkEdgeInitialized%22,%22level%22:%22warning%22,%22msg%22:%22the) Ziti Edge has not been initialized, no default admin exists. Please run 'ziti agent controller init' to configure the default admin'","time":"2024-10-21T17:39:51.875Z"}

---

<div class="post-metadata">

**Author:** ![plorenz](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/plorenz/32/54_2.png) [@plorenz](https://openziti.discourse.group/u/plorenz)\
**Post date:** [October 21, 2024, 3:52pm UTC](https://openziti.discourse.group/t/problem-initializing-ha-cluster/3288/6 "2024-10-21T15:52:50Z")

</div>

Do you see any of the unix pipes used for the `ziti agent` IPC? They're in `/tmp`.

```auto
$ sudo /home/plorenz/go/bin/ziti agent list
╭───────┬────────────┬────────┬────────────────────────────┬────────────┬─────────────┬───────────╮
│ PID │ EXECUTABLE │ APP ID │ UNIX SOCKET │ APP TYPE │ APP VERSION │ APP ALIAS │
├───────┼────────────┼────────┼────────────────────────────┼────────────┼─────────────┼───────────┤
│ 36637 │ ziti │ ctrl1 │ /tmp/gops-agent.36637.sock │ controller │ v0.0.0 │ │
│ 5740 │ ziti │ │ /tmp/gops-agent.5740.sock │ │ │ │
╰───────┴────────────┴────────┴────────────────────────────┴────────────┴─────────────┴───────────╯
$ ls -l /tmp/gops-agent.*
srwx------ 1 plorenz plorenz 0 Oct 21 11:37 /tmp/gops-agent.36637.sock=
srwx------ 1 root root 0 Oct 21 09:22 /tmp/gops-agent.5740.sock=

```

Paul

---

<div class="post-metadata">

**Author:** ![mkuhlmann](https://avatars.discourse-cdn.com/v4/letter/m/5f9b8f/32.png) [@mkuhlmann](https://openziti.discourse.group/u/mkuhlmann)\
**Post date:** [October 21, 2024, 3:58pm UTC](https://openziti.discourse.group/t/problem-initializing-ha-cluster/3288/7 "2024-10-21T15:58:25Z")

</div>

> [@plorenz](#):
>
> `ls -l /tmp/gops-agent.*`

There are no entries called /tmp/gops\*

> sudo -u ziti-controller ziti agent list  
> yields an empty list

---

<div class="post-metadata">

**Author:** ![plorenz](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/plorenz/32/54_2.png) [@plorenz](https://openziti.discourse.group/u/plorenz)\
**Post date:** [October 21, 2024, 4:32pm UTC](https://openziti.discourse.group/t/problem-initializing-ha-cluster/3288/8 "2024-10-21T16:32:29Z")

</div>

Can you run `sudo ps -Af | grep ziti` just so we can verify that it's running, check the flags and see what user it's running as?

---

<div class="post-metadata">

**Author:** ![mkuhlmann](https://avatars.discourse-cdn.com/v4/letter/m/5f9b8f/32.png) [@mkuhlmann](https://openziti.discourse.group/u/mkuhlmann)\
**Post date:** [October 21, 2024, 4:40pm UTC](https://openziti.discourse.group/t/problem-initializing-ha-cluster/3288/9 "2024-10-21T16:40:12Z")

</div>

> [@plorenz](#):
>
> sudo ps -Af | grep ziti

Thats the output of the command:

> [root@ctrl01 tmp]# sudo ps -Af | grep ziti  
> ziti-co+ 14897 1 1 17:35 ? 00:00:59 /opt/openziti/bin/ziti controller run config.yml --

---

<div class="post-metadata">

**Author:** ![plorenz](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/plorenz/32/54_2.png) [@plorenz](https://openziti.discourse.group/u/plorenz)\
**Post date:** [October 21, 2024, 6:14pm UTC](https://openziti.discourse.group/t/problem-initializing-ha-cluster/3288/10 "2024-10-21T18:14:11Z")

</div>

Thank you. Do you see `unable to start CLI agent` in the log, by any chance? If it's there, it should include the error which prevented the CLI agent from starting. I'm wondering if maybe there's a permission issue?

I've not seen this issue before. We do have an option to place the named pipe somewhere else when starting the controller, but I'm noticing that we don't have a way to specify it when using the agent, which is a bug. I'll file an issue so I remember to fix that.

Paul

---

<div class="post-metadata">

**Author:** ![mkuhlmann](https://avatars.discourse-cdn.com/v4/letter/m/5f9b8f/32.png) [@mkuhlmann](https://openziti.discourse.group/u/mkuhlmann)\
**Post date:** [October 22, 2024, 10:09am UTC](https://openziti.discourse.group/t/problem-initializing-ha-cluster/3288/11 "2024-10-22T10:09:25Z")

</div>

I dont see any errors in the logs. Maybe pasting my config can help here:

> v: 3
> 
> db: "/var/lib/private/ziti-controller/bbolt.db"
> 
> raft:  
> dataDir: "/var/lib/private/ziti-controller/raft"  
> minClusterSize: 2  
> bootstrapMembers:  
> - tls:{Second Controllers DNS Name}:6262
> 
> identity:  
> cert: "pki/intermediate/certs/ctrl01.cert"  
> server\_cert: "pki/intermediate/certs/ctrl01.chain.pem"  
> key: "pki/intermediate/keys/ctrl01.key"  
> ca: "pki/ca/certs/ca.cert"
> 
> trustDomain: {My Domain Name}
> 
> ctrl:  
> options:  
> advertiseAddress: tls:{My Internal DNS Name}:6262  
> listener: tls:0.0.0.0:6262
> 
> healthChecks:  
> boltCheck:  
> interval: 30s  
> timeout: 20s  
> initialDelay: 30s
> 
> edge:  
> api:  
> sessionTimeout: 30m  
> enrollment:  
> signingCert:  
> cert: pki/intermediate/certs/intermediate.cert  
> key: pki/intermediate/keys/intermediate.key  
> edgeIdentity:  
> duration: 180m  
> edgeRouter:  
> duration: 180m
> 
> web:
> 
> - name: client-management  
> bindPoints:
> - interface: 0.0.0.0:1280  
> address: {My Public DNS Name}:1280  
> identity:  
> ca: "pki/ca/certs/ca.cert"  
> key: "pki/intermediate/keys/ctrl01.key"  
> server\_cert: "pki/intermediate/certs/ctrl01.chain.pem"  
> cert: "pki/intermediate/certs/client.cert"  
> options:  
> idleTimeout: 5000ms  
> readTimeout: 5000ms  
> writeTimeout: 100000ms  
> minTLSVersion: TLS1.2  
> maxTLSVersion: TLS1.3  
> apis:
> - binding: edge-management  
> options: { }
> - binding: edge-client  
> options: { }
> - binding: fabric  
> options: { }

The config is basically a result of the bootstrapping process. The only this I changed was the raft part.

Thanks in advance!

---

<div class="post-metadata">

**Author:** ![mkuhlmann](https://avatars.discourse-cdn.com/v4/letter/m/5f9b8f/32.png) [@mkuhlmann](https://openziti.discourse.group/u/mkuhlmann)\
**Post date:** [October 22, 2024, 3:16pm UTC](https://openziti.discourse.group/t/problem-initializing-ha-cluster/3288/12 "2024-10-22T15:16:38Z")

</div>

I just gave it a fresh start: This time I did not use the package repo and simply used the binary and my config files. It seems I tripped over the extra complexity of the systemd unit file, the entrypoint script and all that stuff.  
It works now. Case closed.

Thank you so much for your input!

---

<div class="post-metadata">

**Author:** ![plorenz](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/plorenz/32/54_2.png) [@plorenz](https://openziti.discourse.group/u/plorenz)\
**Post date:** [October 22, 2024, 3:54pm UTC](https://openziti.discourse.group/t/problem-initializing-ha-cluster/3288/13 "2024-10-22T15:54:05Z")

</div>

Thank you for the update! I'll let our package maintainer know about the issue.

Paul

---

<div class="post-metadata">

**Author:** ![qrkourier](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/qrkourier/32/52_2.png) [@qrkourier](https://openziti.discourse.group/u/qrkourier)\
**Post date:** [October 22, 2024, 3:57pm UTC](https://openziti.discourse.group/t/problem-initializing-ha-cluster/3288/14 "2024-10-22T15:57:10Z")

</div>

Please let me know if this explanation is sufficient. It's about accessing the agent when running a sandboxed systemd service provided by the openziti-controller or openziti-router Linux packages.

> **[Interprocess Communication Agent | OpenZiti](https://openziti.io/docs/guides/troubleshooting/agent/#accessing-the-agent-on-linux)**
>
> The controller and router provide an IPC agent for administration. The agent listens on a Unix domain socket. Here's an example for querying the controller's agent for statistics.

e.g.,

```bash
systemctl show -p MainPID --value ziti-controller.service \
| xargs -rIPID sudo nsenter --target PID --mount -- \
    ziti agent stats

```

---

<div class="post-metadata">

**Author:** ![mkuhlmann](https://avatars.discourse-cdn.com/v4/letter/m/5f9b8f/32.png) [@mkuhlmann](https://openziti.discourse.group/u/mkuhlmann)\
**Post date:** [February 4, 2025, 9:58am UTC](https://openziti.discourse.group/t/problem-initializing-ha-cluster/3288/15 "2025-02-04T09:58:58Z")

</div>

Sorry for the late response! That has helped indeed!