# Openziti installation in production like environment **URL:** \ **Category:** Support\ **Created:** [August 17, 2022, 10:30pm UTC](https://openziti.discourse.group/t/openziti-installation-in-production-like-environment/707 "2022-08-17T22:30:20Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![av-dev](https://avatars.discourse-cdn.com/v4/letter/a/ba8739/32.png) [@av-dev](https://openziti.discourse.group/u/av-dev)\ **Post date:** [August 17, 2022, 10:30pm UTC](https://openziti.discourse.group/t/openziti-installation-in-production-like-environment/707/1 "2022-08-17T22:30:20Z")
I am trying to install openziti on AWS in production like environment. Which means I would like to have a controller and atleast 1 public router in one AWS account and most likely 2 edge routers in 2 different AWS accounts ( say one for client and one for server ). Q1: Does this sound like a good idea for production like environment? Q2: If I were to bring my own PKI for everything ( for controller rest api, for public router, for edge routers, for enrolling identities ) how can I go about doing that? These are the steps in my mind - 1. Install controller 2. configure controller ( including adding my own CA for auto enrollment ) 3. install public router 4. configure public router 5. register public router with controller 6. install edge router(s) 7. configure edge router(s) 8. register edge router(s) with controller 9. Install tunneler(s) on endpoints 10. Configure tunneler(s) for auto enrollment Appreciate any feedback / recommendations / corrections. TIA. 🙂 ---
**Author:** ![TheLumberjack](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/thelumberjack/32/113_2.png) [@TheLumberjack](https://openziti.discourse.group/u/TheLumberjack)\ **Post date:** [August 17, 2022, 10:41pm UTC](https://openziti.discourse.group/t/openziti-installation-in-production-like-environment/707/2 "2022-08-17T22:41:11Z")
> [@av-dev](#): > > Q1: Does this sound like a good idea for production like environment? I would make at least 2 public routers, probably in different availability zones just for safety's sake. Overall though, yes that seems like a reasonable setup. I would also add on that you should script the backup of the controller database and ship it somewhere 'safe'. If you have few network changes you can do this infrequently, like "daily" which is probably fine. If it happens far more frequently, then you need to determine what level of backup is best for you. > [@av-dev](#): > > Q2: If I were to bring my own PKI for everything ( for controller rest api, for public router, for edge routers, for enrolling identities ) how can I go about doing that? How are you going to make your PKI? This topic is "kinda big" but we can try to cover it. Realistically, by running your own OpenZiti install, you're already "bringing your own". If you want to use a PKI which is verified by some other public/trusted CA, you need a few things. I'd probably say you should start with "the minimalist" setup too. Only bring what you need. OpenZiti supports up to three different CA roots. The 'network' CA/trust root - used for the mesh network. The 'edge' CA/trust root, used for "edge" API. The signing CA/trust root - used for identifications. One trust chain can service all three use cases. The quickstart separates them on purpopse to demonstrate that they are different. I'll see if we have any instructions on 'exactly how' to do this, but I don't think we do yet. Your set of instructions seem just right to me fwiw. I'd tweak the wording of 'register' to 'enroll' the routers because that's the command you run (`ziti-router enroll`) but other than that, the flow looks right. I'll see if anyone has any pointers on exactly how one should setup the PKI, but to help out my research, what kind of PKI are you looking to create? LetsEncrypt? Digicert? Something else? Your own PKI? etc. I've personally only ever used openssl to mint my certs - or the `ziti` CLI. If it comes up, it might be good to know. Cheers! ---
**Author:** ![av-dev](https://avatars.discourse-cdn.com/v4/letter/a/ba8739/32.png) [@av-dev](https://openziti.discourse.group/u/av-dev)\ **Post date:** [August 17, 2022, 10:47pm UTC](https://openziti.discourse.group/t/openziti-installation-in-production-like-environment/707/3 "2022-08-17T22:47:47Z")
Thank you for the quick reply. We run an internal private CA which I definitely want to use for auto enrollment of endpoints. So may be I will start from there. ( 3rd trust root for identification ) But in general, all AWS instances launched, come up with an identity in form of an x.509 certificate from the internal CA so I was hoping to use those respective identities for edge and network too. ---
**Author:** ![av-dev](https://avatars.discourse-cdn.com/v4/letter/a/ba8739/32.png) [@av-dev](https://openziti.discourse.group/u/av-dev)\ **Post date:** [August 17, 2022, 10:51pm UTC](https://openziti.discourse.group/t/openziti-installation-in-production-like-environment/707/4 "2022-08-17T22:51:43Z")
Great point regarding DB and backup. I am thinking about using managed RDS for PostgreSQL which should help with DB backup etc. ---
**Author:** ![TheLumberjack](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/thelumberjack/32/113_2.png) [@TheLumberjack](https://openziti.discourse.group/u/TheLumberjack)\ **Post date:** [August 17, 2022, 10:56pm UTC](https://openziti.discourse.group/t/openziti-installation-in-production-like-environment/707/5 "2022-08-17T22:56:58Z")
Gotcha. You’ll need at least one private key to start with on the machine. Assuming that’s part of your plan you’ll need a certificate that is able to be used for ‘server’ connections. That means it should have these properties: ```auto X509v3 extensions: X509v3 Key Usage: critical Digital Signature, Non Repudiation, Key Encipherment X509v3 Basic Constraints: critical CA:FALSE ``` You’ll need a “signing” cert/key. Which means it should have these properties set: ```auto X509v3 extensions: X509v3 Key Usage: critical Digital Signature, Certificate Sign, CRL Sign X509v3 Basic Constraints: critical CA:TRUE, pathlen:1 ``` I think that’s all you need. You could try to smash those into one key/cert but imo - if you can… I would just make at least two different cert/key pairs… That enough information to get you going? Our database is file-based and resides at wherever you tell it via the controller config file. Often you’ll see it in something like my quickstart: ```auto db: "/home/ubuntu/.ziti/quickstart/ip-172-31-42-64/db/ctrl.db" ``` You can run `ziti edge db help` for help on how to safely back that db up for backup. hope this helps! ---
**Author:** ![av-dev](https://avatars.discourse-cdn.com/v4/letter/a/ba8739/32.png) [@av-dev](https://openziti.discourse.group/u/av-dev)\ **Post date:** [August 17, 2022, 11:10pm UTC](https://openziti.discourse.group/t/openziti-installation-in-production-like-environment/707/6 "2022-08-17T23:10:07Z")
> You’ll need a “signing” cert/key. Which means it should have these properties set: If I register our internal CA for auto-enrollement and auto-approval, do I still need a signing CA? ---
**Author:** ![TheLumberjack](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/thelumberjack/32/113_2.png) [@TheLumberjack](https://openziti.discourse.group/u/TheLumberjack)\ **Post date:** [August 17, 2022, 11:43pm UTC](https://openziti.discourse.group/t/openziti-installation-in-production-like-environment/707/7 "2022-08-17T23:43:49Z")
Not in the controller, no, but in order to complete the 3rd party CA enrollment process and verify the 3rd party CA, you will need to be able to produce a certificate with the given verification token as the CN of the certificate, issued by your CA. Here’s a video that covers that topic and is short [Using 3rd Party CAs with Ziti CLI, and ZAC - YouTube](https://www.youtube.com/watch?v=USMim65c4ic). I just ran the commands from that video (found in the description) and when you use openssl to look at the cert it should look something like: ```auto openssl x509 -in "$ZITI_PKI/${ca_name}/certs/${ca_name}.cert" -text Certificate: Data: Version: 3 (0x2) Serial Number: 4d:d6:af:ce:9b:2b:1a:33:75:7d:ec:31:4d:ea:93:c0 Signature Algorithm: sha256WithRSAEncryption Issuer: C = US, L = Charlotte, O = NetFoundry, OU = ADV-DEV, CN = new_ca_233932 Validity Not Before: Aug 17 23:38:38 2022 GMT Not After : Aug 14 23:39:32 2032 GMT Subject: C = US, L = Charlotte, O = NetFoundry, OU = ADV-DEV, CN = KBm3LeaTr ``` Here you can see the CN was set to the ‘verification token’ and the ‘Issuer’ is my 3rd party CA I just created now. The endpoints will still need a private key and public cert from the CA. Those are often in the form of pkcs12, often a `.p12` or `.pfx` file extension. [Top google result for pkcs12](https://www.ssl.com/how-to/export-certificates-private-key-from-pkcs12-file-with-openssl/). I expect that distribution is already handled/accounted for? ---
**Author:** ![av-dev](https://avatars.discourse-cdn.com/v4/letter/a/ba8739/32.png) [@av-dev](https://openziti.discourse.group/u/av-dev)\ **Post date:** [August 17, 2022, 11:48pm UTC](https://openziti.discourse.group/t/openziti-installation-in-production-like-environment/707/8 "2022-08-17T23:48:51Z")
Yes, I will get the verification cert issued from our internal CA. Yes, all of the endpoints will have at least one identity in form of x.509 certificate ( and corresponding key ) on the host as part of the bootstrap process. We use pem files which are placed at a well-known location. ---
**Author:** ![TheLumberjack](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/thelumberjack/32/113_2.png) [@TheLumberjack](https://openziti.discourse.group/u/TheLumberjack)\ **Post date:** [August 18, 2022, 1:28pm UTC](https://openziti.discourse.group/t/openziti-installation-in-production-like-environment/707/9 "2022-08-18T13:28:09Z")
One more thing occurred to me this morning. Make sure you monitor your cloud vm limits. We found in production usage settings, using "T" instances in AWS, you might experience strange, unexplainable timeouts. Services won't look like they are available when they were moments ago etc. You'll start to think that OpenZiti is not reliable but that's not what is **usually** happening... Usually, it's because the instances are running out of some sort of limit... On the NetFoundry support page for sizing you'll see them reference: > Cloud instances should have compute optimized these instances need dedicated compute, not burst compute for AWS that means the C-Series and for Azure equivalent would be the F-Series. So, beware of that too 🙂 ---
**Author:** ![plorenz](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/plorenz/32/54_2.png) [@plorenz](https://openziti.discourse.group/u/plorenz)\ **Post date:** [August 18, 2022, 2:03pm UTC](https://openziti.discourse.group/t/openziti-installation-in-production-like-environment/707/10 "2022-08-18T14:03:20Z")
> [@av-dev](#): > > Great point regarding DB and backup. I am thinking about using managed RDS for PostgreSQL which should help with DB backup etc. Note that the ziti controller uses an embedded bolt database. There are API and CLI options for creating a snapshot database file, which can be backed up/restored like a regular file. ---
**Author:** ![av-dev](https://avatars.discourse-cdn.com/v4/letter/a/ba8739/32.png) [@av-dev](https://openziti.discourse.group/u/av-dev)\ **Post date:** [August 18, 2022, 11:00pm UTC](https://openziti.discourse.group/t/openziti-installation-in-production-like-environment/707/11 "2022-08-18T23:00:08Z")
So just so that I understand correctly, there is Postgres DB for something and then there is a separate boltdb? ---
**Author:** ![arslane](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/arslane/32/408_2.png) [@arslane](https://openziti.discourse.group/u/arslane)\ **Post date:** [August 18, 2022, 11:24pm UTC](https://openziti.discourse.group/t/openziti-installation-in-production-like-environment/707/12 "2022-08-18T23:24:45Z")
I believe there’s no postgres db, the controller stores everything on a key value db named bbolt ---
**Author:** ![av-dev](https://avatars.discourse-cdn.com/v4/letter/a/ba8739/32.png) [@av-dev](https://openziti.discourse.group/u/av-dev)\ **Post date:** [August 18, 2022, 11:49pm UTC](https://openziti.discourse.group/t/openziti-installation-in-production-like-environment/707/13 "2022-08-18T23:49:11Z")
I got confused by looking at the sample config file - [ziti-doc/sample-controller-config.yaml at main ¡ openziti/ziti-doc ¡ GitHub](https://github.com/openziti/ziti-doc/blob/main/docfx_project/ziti/manage/sample-controller-config.yaml#L60-L67) ---
**Author:** ![TheLumberjack](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/thelumberjack/32/113_2.png) [@TheLumberjack](https://openziti.discourse.group/u/TheLumberjack)\ **Post date:** [August 19, 2022, 12:22am UTC](https://openziti.discourse.group/t/openziti-installation-in-production-like-environment/707/14 "2022-08-19T00:22:39Z")
Yikes - time to delete that yaml file!!! It’s woefully out of date. I’ll do that now. We used postgres around 1 or 2 years ago and ditched it because setting up postgres isn’t as easy as “not having to”. 🙂 ---
**Author:** ![TheLumberjack](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/thelumberjack/32/113_2.png) [@TheLumberjack](https://openziti.discourse.group/u/TheLumberjack)\ **Post date:** [August 19, 2022, 12:33am UTC](https://openziti.discourse.group/t/openziti-installation-in-production-like-environment/707/15 "2022-08-19T00:33:25Z")
I filed an issue to track fixing/updating those files. [update/remove example configs · Issue #152 · openziti/ziti-doc · GitHub](https://github.com/openziti/ziti-doc/issues/152) thank you very much for bringing that out. We’re in the midst right now of going back through all the doc, setting up a new site etc so it’ll get fixed soon ---
**Author:** ![av-dev](https://avatars.discourse-cdn.com/v4/letter/a/ba8739/32.png) [@av-dev](https://openziti.discourse.group/u/av-dev)\ **Post date:** [August 23, 2022, 11:42pm UTC](https://openziti.discourse.group/t/openziti-installation-in-production-like-environment/707/16 "2022-08-23T23:42:32Z")
ziti-router enroll command does not give me an option to use pre-existing 3rd party CA identity. Do I need to use a diff way to enroll? ```auto $ ziti-router enroll -h Enroll a router as an edge router Usage: ziti-router enroll [flags] Flags: -e, --engine string An engine -h, --help help for enroll -j, --jwt string The path to a JWT file -a, --keyAlg RSA|EC Crypto algorithm to use when generating private key (default RSA) Global Flags: --cli-agent Enable/disable CLI Agent (enabled by default) (default true) --cli-agent-addr string Specify where CLI Agent should list (ex: unix:/tmp/myfile.sock or tcp:127.0.0.1:10001) --debug-ops Enable/disable debug agent operations (disabled by default) --log-formatter string Specify log formatter [json|pfxlog|text] -v, --verbose Enable verbose logging ``` ---
**Author:** ![av-dev](https://avatars.discourse-cdn.com/v4/letter/a/ba8739/32.png) [@av-dev](https://openziti.discourse.group/u/av-dev)\ **Post date:** [August 23, 2022, 11:47pm UTC](https://openziti.discourse.group/t/openziti-installation-in-production-like-environment/707/17 "2022-08-23T23:47:53Z")
Or do I need to this in 2 steps - 1. “ziti edge enroll” to get the identity from the controller 2. use “ziti edge create edge-router” with that identity and skip the ziti-router enroll step? ---
**Author:** ![TheLumberjack](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/thelumberjack/32/113_2.png) [@TheLumberjack](https://openziti.discourse.group/u/TheLumberjack)\ **Post date:** [August 23, 2022, 11:50pm UTC](https://openziti.discourse.group/t/openziti-installation-in-production-like-environment/707/18 "2022-08-23T23:50:59Z")
Hmmmm. I don’t know if that’s something we’ve ever considered supporting yet. I believe the enrollment process is designed to prevent the transferring of any sensitive material like the private key at all, which means it’s generated on that machine. Then, as part of the enrollment process, the certs are written to the locations specified in the config file. You can’t skip the enroll step, and if you change the certs I think they’ll be overwritten whenever they need to be renewed. At this time - I don’t think there’s a way to accommodate what you’re looking for. @andrew.martinez - is there any process to allow for an operator to maintain the crypto material on their own? ---
**Author:** ![av-dev](https://avatars.discourse-cdn.com/v4/letter/a/ba8739/32.png) [@av-dev](https://openziti.discourse.group/u/av-dev)\ **Post date:** [August 23, 2022, 11:57pm UTC](https://openziti.discourse.group/t/openziti-installation-in-production-like-environment/707/19 "2022-08-23T23:57:25Z")
Is router enrollment any different from an endpoint enrollment? Also while setting up the controller, I specified my internal CA values for controller and edge-controller env variables. I did generate the signing CA but not the other 2 CAs from express install. Would that be a problem? ---
**Author:** ![av-dev](https://avatars.discourse-cdn.com/v4/letter/a/ba8739/32.png) [@av-dev](https://openziti.discourse.group/u/av-dev)\ **Post date:** [August 24, 2022, 12:00am UTC](https://openziti.discourse.group/t/openziti-installation-in-production-like-environment/707/20 "2022-08-24T00:00:50Z")
Just to make sure we are on the same page this is what I was thinking - 1. ziti edge enroll --cert \ --key \ --jwt \ --out \ 2. ziti edge create edge-router -i \ wouldnt this enroll the router like any other endpoint? ( This is after registering internal ca with controller for auto-enrollment and auth ) [Next page](https://openziti.discourse.group/t/openziti-installation-in-production-like-environment/707.md?page=2)