Openziti installation in production like environment

I am trying to install openziti on AWS in production like environment. Which means I would like to have a controller and atleast 1 public router in one AWS account and most likely 2 edge routers in 2 different AWS accounts ( say one for client and one for server ).

Q1: Does this sound like a good idea for production like environment?

Q2: If I were to bring my own PKI for everything ( for controller rest api, for public router, for edge routers, for enrolling identities ) how can I go about doing that?

These are the steps in my mind -

  1. Install controller
  2. configure controller ( including adding my own CA for auto enrollment )
  3. install public router
  4. configure public router
  5. register public router with controller
  6. install edge router(s)
  7. configure edge router(s)
  8. register edge router(s) with controller
  9. Install tunneler(s) on endpoints
  10. Configure tunneler(s) for auto enrollment

Appreciate any feedback / recommendations / corrections.

TIA. :slight_smile:

I would make at least 2 public routers, probably in different availability zones just for safety’s sake. Overall though, yes that seems like a reasonable setup. I would also add on that you should script the backup of the controller database and ship it somewhere ‘safe’. If you have few network changes you can do this infrequently, like “daily” which is probably fine. If it happens far more frequently, then you need to determine what level of backup is best for you.

How are you going to make your PKI? This topic is “kinda big” but we can try to cover it. Realistically, by running your own OpenZiti install, you’re already “bringing your own”. If you want to use a PKI which is verified by some other public/trusted CA, you need a few things. I’d probably say you should start with “the minimalist” setup too. Only bring what you need. OpenZiti supports up to three different CA roots. The ‘network’ CA/trust root - used for the mesh network. The ‘edge’ CA/trust root, used for “edge” API. The signing CA/trust root - used for identifications. One trust chain can service all three use cases. The quickstart separates them on purpopse to demonstrate that they are different. I’ll see if we have any instructions on ‘exactly how’ to do this, but I don’t think we do yet.

Your set of instructions seem just right to me fwiw. I’d tweak the wording of ‘register’ to ‘enroll’ the routers because that’s the command you run (ziti-router enroll) but other than that, the flow looks right.

I’ll see if anyone has any pointers on exactly how one should setup the PKI, but to help out my research, what kind of PKI are you looking to create? LetsEncrypt? Digicert? Something else? Your own PKI? etc. I’ve personally only ever used openssl to mint my certs - or the ziti CLI. If it comes up, it might be good to know.

Cheers!

Thank you for the quick reply.

We run an internal private CA which I definitely want to use for auto enrollment of endpoints. So may be I will start from there. ( 3rd trust root for identification )

But in general, all AWS instances launched, come up with an identity in form of an x.509 certificate from the internal CA so I was hoping to use those respective identities for edge and network too.

Great point regarding DB and backup. I am thinking about using managed RDS for PostgreSQL which should help with DB backup etc.

Gotcha. You’ll need at least one private key to start with on the machine. Assuming that’s part of your plan you’ll need a certificate that is able to be used for ‘server’ connections.

That means it should have these properties:

        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Non Repudiation, Key Encipherment
            X509v3 Basic Constraints: critical
                CA:FALSE

You’ll need a “signing” cert/key. Which means it should have these properties set:

        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Certificate Sign, CRL Sign
            X509v3 Basic Constraints: critical
                CA:TRUE, pathlen:1

I think that’s all you need. You could try to smash those into one key/cert but imo - if you can… I would just make at least two different cert/key pairs… That enough information to get you going?

Our database is file-based and resides at wherever you tell it via the controller config file. Often you’ll see it in something like my quickstart:

db:                     "/home/ubuntu/.ziti/quickstart/ip-172-31-42-64/db/ctrl.db"

You can run ziti edge db help for help on how to safely back that db up for backup.

hope this helps!

1 Like

You’ll need a “signing” cert/key. Which means it should have these properties set:

If I register our internal CA for auto-enrollement and auto-approval, do I still need a signing CA?

Not in the controller, no, but in order to complete the 3rd party CA enrollment process and verify the 3rd party CA, you will need to be able to produce a certificate with the given verification token as the CN of the certificate, issued by your CA. Here’s a video that covers that topic and is short Using 3rd Party CAs with Ziti CLI, and ZAC - YouTube. I just ran the commands from that video (found in the description) and when you use openssl to look at the cert it should look something like:

openssl x509 -in "$ZITI_PKI/${ca_name}/certs/${ca_name}.cert" -text
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            4d:d6:af:ce:9b:2b:1a:33:75:7d:ec:31:4d:ea:93:c0
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: C = US, L = Charlotte, O = NetFoundry, OU = ADV-DEV, CN = new_ca_233932
        Validity
            Not Before: Aug 17 23:38:38 2022 GMT
            Not After : Aug 14 23:39:32 2032 GMT
        Subject: C = US, L = Charlotte, O = NetFoundry, OU = ADV-DEV, CN = KBm3LeaTr

Here you can see the CN was set to the ‘verification token’ and the ‘Issuer’ is my 3rd party CA I just created now.

The endpoints will still need a private key and public cert from the CA. Those are often in the form of pkcs12, often a .p12 or .pfx file extension. Top google result for pkcs12. I expect that distribution is already handled/accounted for?

Yes, I will get the verification cert issued from our internal CA.

Yes, all of the endpoints will have at least one identity in form of x.509 certificate ( and corresponding key ) on the host as part of the bootstrap process.
We use pem files which are placed at a well-known location.

One more thing occurred to me this morning. Make sure you monitor your cloud vm limits. We found in production usage settings, using “T” instances in AWS, you might experience strange, unexplainable timeouts. Services won’t look like they are available when they were moments ago etc. You’ll start to think that OpenZiti is not reliable but that’s not what is usually happening… Usually, it’s because the instances are running out of some sort of limit… On the NetFoundry support page for sizing you’ll see them reference:

Cloud instances should have compute optimized these instances need dedicated compute, not burst compute for AWS that means the C-Series and for Azure equivalent would be the F-Series.

So, beware of that too :slight_smile:

Note that the ziti controller uses an embedded bolt database. There are API and CLI options for creating a snapshot database file, which can be backed up/restored like a regular file.

1 Like

So just so that I understand correctly, there is Postgres DB for something and then there is a separate boltdb?

I believe there’s no postgres db, the controller stores everything on a key value db named bbolt

1 Like

I got confused by looking at the sample config file - ziti-doc/sample-controller-config.yaml at main · openziti/ziti-doc · GitHub

Yikes - time to delete that yaml file!!! It’s woefully out of date. I’ll do that now. We used postgres around 1 or 2 years ago and ditched it because setting up postgres isn’t as easy as “not having to”. :slight_smile:

1 Like

I filed an issue to track fixing/updating those files. update/remove example configs · Issue #152 · openziti/ziti-doc · GitHub

thank you very much for bringing that out. We’re in the midst right now of going back through all the doc, setting up a new site etc so it’ll get fixed soon

1 Like

ziti-router enroll command does not give me an option to use pre-existing 3rd party CA identity. Do I need to use a diff way to enroll?

$ ziti-router enroll -h
Enroll a router as an edge router

Usage:
  ziti-router enroll <config> [flags]

Flags:
  -e, --engine string   An engine
  -h, --help            help for enroll
  -j, --jwt string      The path to a JWT file
  -a, --keyAlg RSA|EC   Crypto algorithm to use when generating private key (default RSA)

Global Flags:
      --cli-agent               Enable/disable CLI Agent (enabled by default) (default true)
      --cli-agent-addr string   Specify where CLI Agent should list (ex: unix:/tmp/myfile.sock or tcp:127.0.0.1:10001)
      --debug-ops               Enable/disable debug agent operations (disabled by default)
      --log-formatter string    Specify log formatter [json|pfxlog|text]
  -v, --verbose                 Enable verbose logging

Or do I need to this in 2 steps -

  1. “ziti edge enroll” to get the identity from the controller
  2. use “ziti edge create edge-router” with that identity

and skip the ziti-router enroll step?

Hmmmm. I don’t know if that’s something we’ve ever considered supporting yet. I believe the enrollment process is designed to prevent the transferring of any sensitive material like the private key at all, which means it’s generated on that machine. Then, as part of the enrollment process, the certs are written to the locations specified in the config file.

You can’t skip the enroll step, and if you change the certs I think they’ll be overwritten whenever they need to be renewed. At this time - I don’t think there’s a way to accommodate what you’re looking for.

@andrew.martinez - is there any process to allow for an operator to maintain the crypto material on their own?

Is router enrollment any different from an endpoint enrollment?

Also while setting up the controller, I specified my internal CA values for controller and edge-controller env variables. I did generate the signing CA but not the other 2 CAs from express install. Would that be a problem?

Just to make sure we are on the same page this is what I was thinking -

  1. ziti edge enroll --cert <INTERNAL_CA_PROVIDED_CERT> --key <INTERNAL_CA_PROVIDED_KEY> --jwt <INTERNAL_CA_REUSABLE_JWT> --out <CUSTOM_IDENTITY>

  2. ziti edge create edge-router -i <CUSTOM_IDENTITY>

wouldnt this enroll the router like any other endpoint?

( This is after registering internal ca with controller for auto-enrollment and auth )