What is the requirement / process around backup. Using docker container, but any documentation about locally hosted or docker would be great. I could not see it, but are there commands that backs up the configuration / restores configuration?
Being docker, it is probably the docker-compose along with a DB or sorts that would be needed right? Quick google didn’t turn anything up.
There is a ziti cli command to backup the database. ziti edge db snapshot
Database management operations for the Ziti Edge Controller
Usage:
ziti edge db [flags]
ziti edge db [command]
Available Commands:
check-integrity-status shows current results from background operation checking integrity of database references and constraints
snapshot creates a database snapshot
start-check-integrity starts background operation checking integrity of database references and constraints
It will put a copy of the database next to the online db which you can then backup by simple docker copy/paste. You’d probably would want to grab the whole PKI too.
Sorry to necropost, what do you mean by this? Is there a doc that includes what directories need to be backed up, regardless of the install method?
You’d probably would want to grab the whole PKI too.
You can look in the identity section of your controller and router YAML files to see the location.
That’s true. The identity configuration section of the controller’s and routers’ config YAMLs will cover most PKI-related files you need to back up. The identity section may appear in several places in those configs, depending on how many certificates you use.
In the controller’s config, there’s also the edge enrollment signer. This is the certificate and private key of the CA that issues leaf certificates during edge enrollment of routers and identities. These are configured in the edge.enrollment.signingCert property (link to reference doc).
Is there a common root path I can backup? We are talking about text files, so I don’t mind backing up MORE than I need for the sake of making backup simpler.
If you are starting from one of the quickstarts, all of the PKI will be in a directory called pki (though it may be different for the Kubernetes quickstart - I’ll defer to @qrkourier). A check of the YAML files is your best place to check and verify.
On my local machine, they are all stored in ~/.ziti/quickstarts/${NET_NAME}/pki (also available as env variable ZITI_PKI if running the express install per the docs).
Yes, sound off if you need specific K8s ops guidance. It’s not solved in any one size way yet. The same two resources are applicable to K8s: controller database (a file in a PVC) and PKI (owned by cert-manager).
Ok, will do. I am probably not going to run it in k8s any time soon. So it will be either locally installed or in compose.
Thanks.
Hi @qrkourier
I'm right now on creating a migration script in nodeJS for migration all OpenZiti data within K8S - in other words its' DB and the certs - to another cluster.
-> And yes I still have the AWS PCA integration in mind.. just no time yet, as I'm dealing with AWS general architecture 
My plan is to use the Edge Management API and call POST /database/snapshot. Then I would fetch the snapshot using a kubectl library for nodeJS and also get all the related certs from cert-manager.
But the question is how to restore that DB file to the "new" OpenZiti deployment, when deploying OpenZiti via helm.
Also I would like to know if it this a good idea to do migration like that or better do it via exporting identities one by one (via script) - as you could have changed the db keys ?
(But then I would loose client and router enrollments I guess)
BR
Jan
You'll need the controller's BoltDB file and at least the Ziti Edge Signer CA certs from cert-manager. Your existing identities and routers are configured to trust the web API's server certificates too, so it's probably necessary to transplant the web identity PKI as well.
EDIT: You'll also need the controler plane PKI if you are transplanting routers because they're configured to trust that server certificate.
EDIT2: The more I think about it, I can't think of any PKI you wouldn't need in at least some situations, so it's probably simpler to transplant all the resources managed by cert-manager on behalf of the Ziti controller. No need to transplant trust-manager's Bundle because it will be re-created by the new trust-manager.
As for the BoltDB file, the controller's Helm chart creates the file if it doesn't exist, so you must create the persistent volume in the new cluster, place the copied DB file in the expected filename ctrl.db, and then install the controller chart in the new cluster with value existingClaim having the name of the pre-created PVC.
Thanks for the fast response.
So basically dumping the database & PKI is the only way as I see ?
In case we also want to update the version of the controller:
-> How can we avoid issues with changes of BoltDB key-names ?
-> Could there be a database migration necessary when updating ?
BR
Yes, you will need to copy the DB file ctrl.db from the old controller's volume to the new. If you wish to obtain a copy while the controller is still running you may use the ziti agent controller snapshot-db command, which is available inside the controller container. This will produce a snapshot in the same directory as the BoltDB file.
You'll also need to recreate the PKI that are managed by cert-manager and Helm. I don't have any ready tooling for this task, but it can certainly be done with the Kube API and kubectl. One idea that came to mind is to perform a helm install on the new cluster, especially if you're planning to have different namespaces or labels, then patch the cert-manager resources with the data from the old cluster.
As long as you're upgrading and not downgrading the controller version, or begin using a new controller configuration that triggers a DB migration, I expect any database schema changes to be handled gracefully and automatically.
You will need a separate copy of the BoltDB file if you wish to roll back a controller version or controller configuration change that triggers a database migration.
Thanks for the explanation!
I managed to perform a manual migration from one K8s cluster to another - basically changing the infrastructure environment stage.
This is the process I came up with:
This workflow only works as long as the naming of all resources - certs, helm release names and namespaces - are equal to the "old" installation !
In old controller Pod run:
ziti agent controller snapshot-db
Copy db snapshot form pod to local:
kubectl cp openziti/openziti-base-controller-c46c64b69-vrm4p:/persistent/ctrl.db-20231017-142051 ctrl.db-20231017-142051
Backup openziti-base-controller-admin-secret:
kubectl get secret/openziti-base-controller-admin-secret -o yaml > openziti-base-controller-admin-secret.yaml
Backup pvc config openziti-base-controller:
kubectl get pvc/openziti-base-controller -o yaml > openziti-base-controller.yaml
Backup certs and issuers:
kubectl get -n openziti -o yaml issuer,cert > backup.yaml
Backup secrets for certs and issuers:
Backup secrets for router enrollments and respective identities
Be sure to delete annotations, labels, and creationTimeStamp, ownerReferences, resourceVersion, uid from secrets and certs/issuers before running kubectl apply on the new K8s cluster!
Deploy openziti-base-controller-admin-secret.yaml to new cluster
kubectl apply -f openziti-base-controller-admin-secret.yamlDeploy pvc to new cluster, mount it to busybox and transfer db backup to it!
kind: Pod
apiVersion: v1
metadata:
name: volume-debugger
spec:
volumes:
- name: volume-to-debug
persistentVolumeClaim:
claimName: openziti-base-controller
containers:
- name: debugger
image: busybox
command: ['sleep', '3600']
volumeMounts:
- mountPath: "/persistent"
name: volume-to-debug
kubectl apply -f busybox-pvc-controller.yamlkubectl cp ctrl.db-20231017-142051 openziti/volume-debugger:/persistent/ctrl.dbkubectl exec -it pod/volume-debugger -- /bin/shkubectl delete -f busybox-pvc-controller.yamlAdd existing pvc claim to openziti controller chart!
Run openziti-base helm chart
Override certs, issuers and secret resources with backups
kubectl apply -f certs_issuers/kubectl apply -f controller-secrets/Should the domain-names for the controller APIs have changed, some certs need to be renewed in the cert-manager:
Restart OpenZiti controller - delete and it will be recreated
This is for sure not complete, but it is a starting point 