The docs at Controller Backup and Recovery | OpenZiti gives use a command to create a db snapshot.
ziti edge db snapshot
The snapshot operation creates a snapshot file in the same filesystem directory as the main database file on the controller host, so you can periodically back up the directory containing the configuration file and database and snapshots.
Does this mean backup up a controller involves keep track of three resources?
- configuration file
- database
- snapshots
Is there a suggestion or best practice for backing these resources up in a cron manner? Would a simple lambda function suffice? Hosting the controller on aws eks for context.
You're correct about the database and configuration being essential for backup, except you don't need the BBolt database file if you have a current snapshot. Each snapshot stands alone and is sufficient to restore the database. You also need to ensure you back up your root CA. See the PKI section of the backup article about that.
BTW, are you using Linux, Docker, or Kubernetes?
The snapshot command you referenced is the most portable way to trigger the snapshot operation. That approach uses the management API, so you must be logged in.
If you're automating this with cron or systemd, then it's better to not store an admin credential if it can be avoided. Instead, you can use the IPC socket which is implemented by the ziti agent command.
On Linux, the controller service runs in a namespace managed by systemd. Here's an example for mounting the namespace to invoke the agent.
systemctl show -p MainPID --value ziti-controller.service \
| xargs -rIPID sudo nsenter --target PID --mount -- \
ziti agent controller snapshot-db
I'm contemplating adding a scheduled backup job in the Linux controller package and the Docker controller image.
I'm using Kubernetes. Is threre a specific recommendatoin for it as opposed to other?
you don't need the BBolt database file if you have a current snapshot. Each snapshot stands alone and is sufficient to restore the database
So as long as snapshots are being made. Am I reight to say the list of resources to keep track of is reduced to:
- configuratoin file
- snapshot
- root CA
I'd say "mostly" yes. You should ensure you've backed up any/all PKI related files, not just the root CA. Any cert/key referenced from the config file and/or CA's should be backed up and any relevant CA key/cert. Backing up the config file, while not strictly required since you could recreate it, is definitely a good idea.
Fair enough. It's probably simpler to restore everything, even the stuff that's replaceable.
I like the idea of securing the root CA separately from the operational environment of the controller because it's used for recovery and not needed for routine ops. I wonder if that's viable for a K8s controller, though. Cert Manager might not allow removing an Issuer or its backing secret.
I'd be interested in what you come up with for a Kubernetes backup. I imagine I'd use something like Velero to discover manifests in a namespace and copy them to the backup storage location.