Theres lots of information out there but i couldn't find too much on PKI rotations and infra.
My goal is to use Vault OSS as the root of trust, (assume Vault is on the same metal and the root is kept in Vault for simplicity rn) I would like to sign an intermediate and give this to Ziti to create its own PKI, just the regular bootstrap process but I want control over the Ziti root (aka the vault int). Would this be possible to make Ziti treat the PKI as "Native" instead of 3rd party? (sounds dumb I know.)
My second challenge would be understanding how I might be able to rotate Ziti's entire PKI stack from a new Vault intermediate while preserving everything else about the Ziti network.
I'd like Vault to automate all of my IntCA Rotations and for Ziti to use the Ziti-IntCA to preform its own rotations in the Ziti network. (If there is any support for this)
Question 1 ) Is this possible to achieve?
Question 2 ) How might you go about creating this?
Question 3 ) What are the key scripts / configs I'd need to modify?
Question 4 ) Are there any useful resources you recommend?
I'll be looking around for more info but I thought I'd ask now, to see if someone already has the answer or thoughts on how to achieve this. Id really appreciate the help! thanks
If you give ziti the private key and the proper PKI and configure ziti to use that key/cert it’s "native". But if you do that, you really need to know what you’re doing and how to troubleshoot when things go wrong is all, which it sounds like you probably would understand. You’ll just have to make the ca bundle file (which shouldn’t be hard for you, it’d just contain your one root CA from Vault OSS).
Yah, that’s not something we'd probably be able to help with, but as long as your key isn’t changing, I would think you can rotate the intermediate/server cert for the controller. I don’t know of any "gotchas" off the top of my head doing that. I would just suggest you give it a try and figure out the process that works for you.
You’d only want to modify the controller config. To test this I would:
configure the controller and make sure it starts
make sure you can auth with the ziti cli tool
create a single router and make sure it starts
use the ziti cli tool with the ziti ops verify traffic command and make sure there are no errors or warnings
There’s no need in going any further than that to test it all out. Once that works, then I would install some tunneler, make an identity and services and test it with a tunneler as well.
Sorry for the long delay in the reply, hopefully that helps?
Hey thanks for getting back! I’m still testing but I’ll do a writeup with any findings when I’m done.
I just think this is a good balance between security standards and ziti’s internal functionality, Id love for this to be come a feature down the line. Anyway thanks for your help, keep up the great work
If you are only interested in using a new intermediate CA for OpenZiti to use, it is possible to do so, but it has some caveats.
the most recent versions of OpenZiti all issue certificates + chains to routers and identities
some older versions did this inconsistently or inadequately and used controller side configuration to fill the gap. So for older networks there is a set of flags that can be looked at for identities to see if they have proper chains or not. If this is a newer network/brand new, it won’t have this issue.
when routers/identities connect, it is required that they provide their cert + chain - up to but does not need to include the root. If the root is included, it is ignored.
replacing the controller’s signing cert (controller configuration file: edge.enrollment.signingCert.[cert|key] ) will only take effect after restarting the controller. The file is not watched.
this will cause all newly created identities and routes to descend from the new intermediate. Already existing identities/routers will descend from the new intermediate if they extend their certificates/rotate keys.
It is possible for admin to instruct identities to extend/rotate keys remotely
It is not currently possible for an admin to instruct routers to extend/rotate keys remotely (feature incomming as it was already internally requested). Routers can be instructed to extend by restart them with the –extend flag.
So I was actually able to modify the controller bootstrap.bash/env to inject my own intCA with a surprising amount of success but ultimately I figured the struggle wasn’t exactly worth it.
The real point was to improve ziti’s root of trust but I realize that changing the cert doesn't change the structure, ultimately the root ca is the one on the system. I reached my limit of ability to change that part. To my knowledge the only way to achieve anything else is to use the actual 3rd party CA structure and trade the versatility and features for heightened security.
Fun experiment, but I think I’ll just find a way of hardening the key storage and maybe after that I’ll see the 3rd party ca options. Only reason I’m not attracted to the 3rd party option is because I believe it will impact mobile users and some internal ziti functions, no?
Thanks for the effort guys, I appreciate it. I just wasn’t going to about it the right way.
I think this was the last version, if anyone more skilled was interested in trying to take the idea father and anchor the external chain properly. I’m tossing in the towel.
/opt/openziti/etc/controller/bootstrap.bash
(CUT)
# Check for a user-provided CA and inject it as the new Root for Ziti.
if [[ -n "${ZITI_SIGNING_CERT_PATH:-}" && -s "${ZITI_SIGNING_CERT_PATH}" && \
-n "${ZITI_SIGNING_KEY_PATH:-}" && -s "${ZITI_SIGNING_KEY_PATH}" && \
-n "${ZITI_ROOT_CA_CERT_PATH:-}" && -s "${ZITI_ROOT_CA_CERT_PATH}" ]]; then
echo "INFO: Injecting custom CA to be Ziti's Root Authority" >&3
local root_ca_dir="${ZITI_PKI_ROOT}/${ZITI_CA_FILE}"
local root_key_path="${root_ca_dir}/keys/${ZITI_CA_FILE}.key"
local root_cert_path="${root_ca_dir}/certs/${ZITI_CA_FILE}.cert"
local root_chain_path="${root_ca_dir}/certs/${ZITI_CA_FILE}.chain.pem"
mkdir -p "${root_ca_dir}/keys"
mkdir -p "${root_ca_dir}/certs"
mkdir -p "${root_ca_dir}/crls"
echo "INFO: Using '${ZITI_SIGNING_CERT_PATH}' as the Root Cert" >&3
cp "${ZITI_SIGNING_CERT_PATH}" "${root_cert_path}"
echo "INFO: Using '${ZITI_SIGNING_KEY_PATH}' as the Root Key" >&3
cp "${ZITI_SIGNING_KEY_PATH}" "${root_key_path}"
echo "INFO: Building full trust chain for the new Root Authority" >&3
cat "${ZITI_SIGNING_CERT_PATH}" "${ZITI_ROOT_CA_CERT_PATH}" > "${root_chain_path}"
# Initialize the CA database files required by 'ziti pki'
if [ ! -f "${root_ca_dir}/index.txt" ]; then
touch "${root_ca_dir}/index.txt"
fi
if [ ! -f "${root_ca_dir}/index.txt.attr" ]; then
touch "${root_ca_dir}/index.txt.attr"
fi
if [ ! -f "${root_ca_dir}/serial" ]; then
echo '01' > "${root_ca_dir}/serial"
fi
if [ ! -f "${root_ca_dir}/crlnumber" ]; then
echo '01' > "${root_ca_dir}/crlnumber"
fi
fi
(CUT)
/opt/openziti/etc/controller/bootstrap.env
# To use an existing Intermediate CA as the new Root for Ziti
ZITI_SIGNING_CERT_PATH=''
ZITI_SIGNING_KEY_PATH=''
# Path to the Root CA that signed the intermediate above, to build the trust chain
ZITI_ROOT_CA_CERT_PATH=''
At this point i believe you either drop the root cert, and you have the same structure as regular ziti, or you find a way to anchor the trust correctly.
