Unable to run HA Controller as a service

1

Hey All,

I've been attempting to set up some VMs for testing the HA controller functionality in OpenZiti. I have a reasonably automatable process going to generate a config file with the settings as desired, and I'm able to at least start up a controller cluster.

However, my problem lies in making the controller application as a service, like can be done after bootstrapping a single controller. My current setup relies on being run manually in a scratch folder. If I run the command line ziti controller run ./config.yml , then everything boots up as expected. Obviously though this means my controller does not stay alive after disconnecting my SSH session.

Firstly, I tried running the bootstrap process from the Controller Deployment Page (link). I had hoped this might pick up on the environment variables I had set to generate my controller configuration, but it just seemed to power ahead and create a default single-controller setup.

Next, I tried following the instructions to migrate an existing setup. Similarly, I followed the instructions from the documentation (link), excluding in Step 4, where I copied the raft folder instead of db.

However, on running the command to enable the service, it fails to start. Examining the logs shows the following error:

Jul 23 14:58:26 ha-controller-1 systemd[1]: ziti-controller.service: Scheduled restart job, restart counter is at 411.
Jul 23 14:58:26 ha-controller-1 systemd[1]: Starting ziti-controller.service - OpenZiti Controller...
Jul 23 14:58:26 ha-controller-1 entrypoint.bash[9683]: realpath: missing operand
Jul 23 14:58:26 ha-controller-1 entrypoint.bash[9683]: Try 'realpath --help' for more information.
Jul 23 14:58:26 ha-controller-1 entrypoint.bash[9687]: realpath: missing operand
Jul 23 14:58:26 ha-controller-1 entrypoint.bash[9687]: Try 'realpath --help' for more information.
Jul 23 14:58:26 ha-controller-1 entrypoint.bash[9676]: ERROR: database file '' is not writable
Jul 23 14:58:26 ha-controller-1 entrypoint.bash[9676]: Provide a configuration in '/var/lib/private/ziti-controller' or generate with:
Jul 23 14:58:26 ha-controller-1 entrypoint.bash[9676]: * Set vars in'/opt/openziti/etc/controller/bootstrap.env'
Jul 23 14:58:26 ha-controller-1 entrypoint.bash[9676]: * Run '/opt/openziti/etc/controller/bootstrap.bash'
Jul 23 14:58:26 ha-controller-1 entrypoint.bash[9676]: * Run 'systemctl enable --now ziti-controller.service'
Jul 23 14:58:26 ha-controller-1 entrypoint.bash[9676]: WARN: set VERBOSE=1 or DEBUG=1 for more output
Jul 23 14:58:26 ha-controller-1 systemd[1]: ziti-controller.service: Control process exited, code=exited, status=1/FAILURE
Jul 23 14:58:26 ha-controller-1 systemd[1]: ziti-controller.service: Failed with result 'exit-code'.
Jul 23 14:58:26 ha-controller-1 systemd[1]: Failed to start ziti-controller.service - OpenZiti Controller.

I have checked, and my config file is definitely in the location which the service seems to be asking for:

$ sudo ls -la /var/lib/private/ziti-controller
total 28
drwxr-xr-x 4 64093 64093  4096 Jul 23 15:02 .
drwx------ 3 root  root   4096 Jul 23 14:36 ..
-rw-r--r-- 1 64093 64093 10790 Jul 23 14:35 config.yml
drwxr-xr-x 4 root  root   4096 Jul 23 15:02 pki
drwx------ 3 root  root   4096 Jul 23 15:02 raft

I struggled to get my head around the documentation for HA so I'm not ruling out the chance I've overlooked something simple. Any advice or ideas would be greatly appreciated.

Thanks!

2

Why not you share the configuration of your controller service?
My working service example (Note: workingdirectory should be your root folder of your raft and controller config.yaml)

nano /etc/systemd/system/ziti-controller.service

[Unit]
Description=OpenZiti Controller
After=network.target

[Service]
Type=simple
ExecStart=/usr/local/bin/ziti controller run /etc/ziti-ha/controller.yaml
Restart=always
RestartSec=5
User=root
WorkingDirectory=/etc/ziti-ha
StandardOutput=append:/var/log/ziti-controller.log
StandardError=append:/var/log/ziti-controller.log

[Install]
WantedBy=multi-user.target

If you can the command manually probably your service file is incorrectly set.

worst case you can run ziti controller run ./config.yml &
this will run it in background process until next reboot

3

Took me a second to find it, as the path is different under Ubuntu.

This is the default file added presumably from installing the package - I've not changed this at all.
I have some extra options compared to yours, but nothing here looks out of the ordinary to me.

Specifically, the working directory is the same one that I copied all the files to while following the docs.

[Unit]
Description=OpenZiti Controller
After=network-online.target

[Service]
Type=simple

# manage the user and permissions for the service automatically
DynamicUser=yes

# this env file configures the service, including whether or not to perform bootstrapping
EnvironmentFile=/opt/openziti/etc/controller/service.env

# relative to /var/lib
StateDirectory=ziti-controller
WorkingDirectory=/var/lib/ziti-controller
ReadOnlyPaths=/opt/openziti/share/console

ExecStartPre=/opt/openziti/etc/controller/entrypoint.bash check config.yml
ExecStart=/opt/openziti/bin/ziti controller run config.yml ${ZITI_ARGS}

Restart=always
RestartSec=3

LimitNOFILE=65535
UMask=0007

[Install]
4

I suggest you change the user from dynamic user to root or ziti with proper permission to ziti folder.

Which user you used when execute ziti run manually?