I am trying to install openziti on AWS in production like environment. Which means I would like to have a controller and atleast 1 public router in one AWS account and most likely 2 edge routers in 2 different AWS accounts ( say one for client and one for server ).
Q1: Does this sound like a good idea for production like environment?
Q2: If I were to bring my own PKI for everything ( for controller rest api, for public router, for edge routers, for enrolling identities ) how can I go about doing that?
These are the steps in my mind -
Install controller
configure controller ( including adding my own CA for auto enrollment )
install public router
configure public router
register public router with controller
install edge router(s)
configure edge router(s)
register edge router(s) with controller
Install tunneler(s) on endpoints
Configure tunneler(s) for auto enrollment
Appreciate any feedback / recommendations / corrections.
I would make at least 2 public routers, probably in different availability zones just for safety's sake. Overall though, yes that seems like a reasonable setup. I would also add on that you should script the backup of the controller database and ship it somewhere 'safe'. If you have few network changes you can do this infrequently, like "daily" which is probably fine. If it happens far more frequently, then you need to determine what level of backup is best for you.
How are you going to make your PKI? This topic is "kinda big" but we can try to cover it. Realistically, by running your own OpenZiti install, you're already "bringing your own". If you want to use a PKI which is verified by some other public/trusted CA, you need a few things. I'd probably say you should start with "the minimalist" setup too. Only bring what you need. OpenZiti supports up to three different CA roots. The 'network' CA/trust root - used for the mesh network. The 'edge' CA/trust root, used for "edge" API. The signing CA/trust root - used for identifications. One trust chain can service all three use cases. The quickstart separates them on purpopse to demonstrate that they are different. I'll see if we have any instructions on 'exactly how' to do this, but I don't think we do yet.
Your set of instructions seem just right to me fwiw. I'd tweak the wording of 'register' to 'enroll' the routers because that's the command you run (ziti-router enroll) but other than that, the flow looks right.
I'll see if anyone has any pointers on exactly how one should setup the PKI, but to help out my research, what kind of PKI are you looking to create? LetsEncrypt? Digicert? Something else? Your own PKI? etc. I've personally only ever used openssl to mint my certs - or the ziti CLI. If it comes up, it might be good to know.
We run an internal private CA which I definitely want to use for auto enrollment of endpoints. So may be I will start from there. ( 3rd trust root for identification )
But in general, all AWS instances launched, come up with an identity in form of an x.509 certificate from the internal CA so I was hoping to use those respective identities for edge and network too.
Gotcha. Youâll need at least one private key to start with on the machine. Assuming thatâs part of your plan youâll need a certificate that is able to be used for âserverâ connections.
That means it should have these properties:
X509v3 extensions:
X509v3 Key Usage: critical
Digital Signature, Non Repudiation, Key Encipherment
X509v3 Basic Constraints: critical
CA:FALSE
Youâll need a âsigningâ cert/key. Which means it should have these properties set:
I think thatâs all you need. You could try to smash those into one key/cert but imo - if you can⌠I would just make at least two different cert/key pairs⌠That enough information to get you going?
Our database is file-based and resides at wherever you tell it via the controller config file. Often youâll see it in something like my quickstart:
Not in the controller, no, but in order to complete the 3rd party CA enrollment process and verify the 3rd party CA, you will need to be able to produce a certificate with the given verification token as the CN of the certificate, issued by your CA. Hereâs a video that covers that topic and is short Using 3rd Party CAs with Ziti CLI, and ZAC - YouTube. I just ran the commands from that video (found in the description) and when you use openssl to look at the cert it should look something like:
openssl x509 -in "$ZITI_PKI/${ca_name}/certs/${ca_name}.cert" -text
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
4d:d6:af:ce:9b:2b:1a:33:75:7d:ec:31:4d:ea:93:c0
Signature Algorithm: sha256WithRSAEncryption
Issuer: C = US, L = Charlotte, O = NetFoundry, OU = ADV-DEV, CN = new_ca_233932
Validity
Not Before: Aug 17 23:38:38 2022 GMT
Not After : Aug 14 23:39:32 2032 GMT
Subject: C = US, L = Charlotte, O = NetFoundry, OU = ADV-DEV, CN = KBm3LeaTr
Here you can see the CN was set to the âverification tokenâ and the âIssuerâ is my 3rd party CA I just created now.
The endpoints will still need a private key and public cert from the CA. Those are often in the form of pkcs12, often a .p12 or .pfx file extension. Top google result for pkcs12. I expect that distribution is already handled/accounted for?
Yes, I will get the verification cert issued from our internal CA.
Yes, all of the endpoints will have at least one identity in form of x.509 certificate ( and corresponding key ) on the host as part of the bootstrap process.
We use pem files which are placed at a well-known location.
One more thing occurred to me this morning. Make sure you monitor your cloud vm limits. We found in production usage settings, using "T" instances in AWS, you might experience strange, unexplainable timeouts. Services won't look like they are available when they were moments ago etc. You'll start to think that OpenZiti is not reliable but that's not what is usually happening... Usually, it's because the instances are running out of some sort of limit... On the NetFoundry support page for sizing you'll see them reference:
Cloud instances should have compute optimized these instances need dedicated compute, not burst compute for AWS that means the C-Series and for Azure equivalent would be the F-Series.
Note that the ziti controller uses an embedded bolt database. There are API and CLI options for creating a snapshot database file, which can be backed up/restored like a regular file.
Yikes - time to delete that yaml file!!! Itâs woefully out of date. Iâll do that now. We used postgres around 1 or 2 years ago and ditched it because setting up postgres isnât as easy as ânot having toâ.
thank you very much for bringing that out. Weâre in the midst right now of going back through all the doc, setting up a new site etc so itâll get fixed soon
ziti-router enroll command does not give me an option to use pre-existing 3rd party CA identity. Do I need to use a diff way to enroll?
$ ziti-router enroll -h
Enroll a router as an edge router
Usage:
ziti-router enroll <config> [flags]
Flags:
-e, --engine string An engine
-h, --help help for enroll
-j, --jwt string The path to a JWT file
-a, --keyAlg RSA|EC Crypto algorithm to use when generating private key (default RSA)
Global Flags:
--cli-agent Enable/disable CLI Agent (enabled by default) (default true)
--cli-agent-addr string Specify where CLI Agent should list (ex: unix:/tmp/myfile.sock or tcp:127.0.0.1:10001)
--debug-ops Enable/disable debug agent operations (disabled by default)
--log-formatter string Specify log formatter [json|pfxlog|text]
-v, --verbose Enable verbose logging
Hmmmm. I donât know if thatâs something weâve ever considered supporting yet. I believe the enrollment process is designed to prevent the transferring of any sensitive material like the private key at all, which means itâs generated on that machine. Then, as part of the enrollment process, the certs are written to the locations specified in the config file.
You canât skip the enroll step, and if you change the certs I think theyâll be overwritten whenever they need to be renewed. At this time - I donât think thereâs a way to accommodate what youâre looking for.
@andrew.martinez - is there any process to allow for an operator to maintain the crypto material on their own?
Is router enrollment any different from an endpoint enrollment?
Also while setting up the controller, I specified my internal CA values for controller and edge-controller env variables. I did generate the signing CA but not the other 2 CAs from express install. Would that be a problem?
