Things to keep in mind while managing ziti controller

Ziti Overlay
1

Would be great , if by your production experience of ziti you could list the things that we should manage in self-hosted ziti network .

We will be making use ziti-controller , routers , tunnels with same PKI ziti provides and manages without importing any external CA

What are the things to keep in mind to manage and configure them via best practises to not see any surprises later .

Basically we will have a single controller in eks , routers too in k8s clusters for exposing internal services and tunnels as both linux service or tunnel as node agents when necessary .

I was able to test them according to my needs and get them working , just wanted to make sure before going to production are there are additional things to keep in mind and follow such as do any certs expire with time and break the system .. etc
because reading through Certificate Management | OpenZiti , does it mean the identities we are creating to talk to routers via tunnels will expire ? We would not allow that to happen so if thats the case

2

How would you as maintainers, do the setup and point down management strategies if you had to setup for yourself

3

Really the number one thing I can point to is to use DNS, not IP addresses. Make sure your controller and routers are all DNS-based and not reliant an IP addresses. That, and make sure you backup your database, controller config file and your overlay PKI. Now-a-days we also support ALPN, so I would use the same port for the control plane as well as the api plane. The quickstarts don't do this (yet) -- but the 'deployment' guides do.

Yes they will expire, but by default, OpenZiti doesn't enforce client certificate expiration by default. So unless you enable that feature, in practice expired client identities don't matter. I do think there is work afoot to allow clients to extend their client certificates (I don't think it's complete yet).

4

ok @TheLumberjack . Actually we install tunnels in our clusters as well to talk to each other , please tell us a way to enable the feature of automatic cert renewal from client side .

I think for routers its by default automatic

5

We dont want interuptions in middle . Things will be messed up if some manual thing in middle is required . because we handle alot of clusters

1 Like
6

@TheLumberjack @qrkourier . Following up on that . I think we are waiting for this sort of conformations before going into production . Please do help thanks

7

I'll repeat some @TheLumberjack mentioned and add a few more. More advice like this is in the deployment guides and reference docs.

  1. use DNS for security and flexibility
    1. use a DNS name that you control for the controller
    2. use DNS names that you control for services so you can obtain a trusted cert if needed
    3. Do not use any DNS names you don't control
  2. use role attributes for security and flexibility
    1. in general, write all your policies with # role attributes, not @ identifiers
    2. grant multiple routers for each router policy with # role attributes in case one fails
  3. for performance, place routers in each geographic region where you have identities
  4. for simplicity, start with a blanket #all/#all service edge router policy (SERP)
  5. backup - everything else is replaceable
    1. root certificate authorities (CA)
    2. controller database
    3. controller config
    4. router configs
  6. ensure routers are online often enough to renew their certs - they do this automatically on a timer if the filesystem is writeable (it is a writeable volume in K8s) - you do not need to do this for identity enrollments, but will in a future version of Ziti
  7. monitor controller server cert expiration (it's auto-renewed in K8s, but you may need to manage them on Linux and Docker)
  8. pin software versions everywhere, e.g., Helm chart major version, ziti major version, etc., to lower the risk of accepting an incompatible upgrade
  9. maintain a test environment or incremental deployment strategy for new software versions
  10. collect and visualize metrics so you can investigate unusual activity
  11. use excellent security hygiene with cloud accounts, including DNS and email providers
  12. consider using cert auth for normal admin operations where practical, securing the default admin password as a recovery credential
8

Thanks , For making it comprehensive .

Few follow ups:

6 - Have doubt here in this paragraphs Certificate Management | OpenZiti does it mean client identities needs manual intervention to extend their cert expiry durations

12 - I'm aware that we have such options as said here Authentication | OpenZiti . But not straight forward from docs to how to enable and use it

9

Client identities are permanent with typical controller configuration. A future feature is to allow extension with the API you mentioned.

Client identities and routers use cert auth by default. The ziti CLI is typically demonstrated with password auth and supports cert auth.

Cert auth here refers to a standard TLS client certificate request. You may use standard tools that work with TLS, like HTTP user agents and HTTP libraries.

Obtain the certificate by creating an identity with the ziti edge create identity CLI flag --admin and enrolling the token. The JSON portably contains cert and key. There is a ziti ops unwrap command to place the cert and key in separate files if wanted, e.g., for login with CLI.