Hi I'm trying OpenZiti with a more realistic FQDN setup. My domain (*.zt.my-domain.dev) enforces HSTS therefore I cannot access ZAC without a trusted certificate. I have my certificates from Let's Encrypt ready for both the root domain (zt.my-domain.dev) and wildcard domain (*.zt.my-domain.dev) but don't know how to use it with OpenZiti.
I found this guide (which is then removed) and cannot make it work.
I would like my admin console (and potentially any service configured to be accessible under *.zt.my-domain.dev and tcp/443) use the certificate from Let's Encrypt. What's the proper way to set it up?
For context, my controller are deployed on the same machine with the following advertised addresses:
- controller:
controller.zt.my-domain.dev, pointed to my machine
- router:
router.zt.my-domain.dev, also pointed to my machine
- example service:
example-service.zt.my-domain.dev
Hi @nickchen120235, you just need the "alternate" certs for the controller-hosted zac, right? If I understand that properly, I think all you need is to use "alt server certs".
If you have your certs in place, you just need a block on the edge management api. For example, on my install my web section looks like:
web:
- name: client-management
bindPoints:
- interface: 0.0.0.0:8441
address: ec2-3-142-245-63.us-east-2.compute.amazonaws.com:8441
identity:
ca: "/home/ubuntu/.ziti/quickstart/ip-172-31-11-231/pki/ip-172-31-11-231-edge-controller-root-ca/certs/ip-172-31-11-231-edge-controller-root-ca.cert"
key: "/home/ubuntu/.ziti/quickstart/ip-172-31-11-231/pki/ip-172-31-11-231-edge-controller-intermediate/keys/ec2-3-142-245-63.us-east-2.compute.amazonaws.com-server.key"
server_cert: "/home/ubuntu/.ziti/quickstart/ip-172-31-11-231/pki/ip-172-31-11-231-edge-controller-intermediate/certs/ec2-3-142-245-63.us-east-2.compute.amazonaws.com-server.chain.pem"
cert: "/home/ubuntu/.ziti/quickstart/ip-172-31-11-231/pki/ip-172-31-11-231-edge-controller-intermediate/certs/ec2-3-142-245-63.us-east-2.compute.amazonaws.com-client.cert"
alt_server_certs:
- server_cert: "/data/docker/letsencrypt/live/clint.demo.openziti.org/fullchain.pem"
server_key: "/data/docker/letsencrypt/live/clint.demo.openziti.org/privkey.pem"
After that, and restarting the controller, you should be able to access the url safely. For example, my test zac url: https://ctrl.clint.demo.openziti.org:8441/zac
1 Like
Hey there good afternoon!
Which deployment model are you following?
Self-hosted, Docker , docker-compose
I can help you with the docker-compose based one 
1 Like
Hey Clint thanks for the reply. I kind of recognized the issue I'm having here when seeing your example. It is that I'm accessing ZAC (or rather the API endpoint) using the domain name the same as the advertised address (controller.zt-example.my-domain.dev), which ziti will use the self-signed certs. When I use the base domain (zt-example.my-domain.dev, which is also pointed to the machine) to access the endpoint it works.
Are there any suggestions regarding picking a domain for the controller (and router) or any that resolves works?
Also, I wonder if I can use the wildcard certificate for a service exposed by ziti (for example, a http server with the following intercept.v1 config and I want to access it via https://demo.zt-example.my-domain.dev)?
Hey thanks for the reply. I'm hosting my instance directly inside my machine. Thank you anyway for the help though.
What you absolutely CANNOT do, is have the self-signed certificates and alternate certificates use the same FQDN. OpenZiti uses SNI to locate the proper certificate to return to the connecting client. If you use the same FQDN for both self-signed (aka the PKI generated during the installation) as well as for the LE certs you will get non-deterministic behavior as the controller will select the certs however it decides and it's not guaranteed which will be selected. For example, you cannot use controller.zt-example.my-domain.dev as your self-signed FQDN and then make a wildcard for *.zt-example.my-domain.dev. If I understand you -- it sounds like that's the situation?
It really doesn't matter as long as they don't overlap. You can call it whatever you want. I expect the problem is that your wildcard is for the 'root' domain and it's overlapping 
Heck yeah you could! That would be just fine. As long as you make an intercept for "whatever.your.domain" (like you show in the cap) and are running a tunneler, that'd work fantastically.
1 Like
Yeah I think that's the problem I'm having.
So I guess it is the underlying service that should be https-capable (i.e. maybe I should put my LE certs inside the service rather than somewhere in the ziti overlay)? I've tried the setup as the screenshot and apparently it didn't work.
And now that I can set alt certs for the web endpoint, is it worth setting the global alt cert for the controller? Like for example my controller FQDN is set to ct.zt-example.my-domain.dev, and I have one LE cert for controller.zt-example.my-domain.dev, which is also pointed to the controller, set as the global alt cert. What would happen if I try to connect an edge router to the controller via controller.zt-example.my-domain.dev when bootstrapping? Will it work or will something break?
realistically, OpenZiti maintains its own PKI and there's generally no "good reason" to assign legitmate certificates to the OpenZiti overlay network -- unless you start using BrowZer or zrok. those will require legit server certs but they are also slightly different use cases... If you use a tunneler, or if you are writing your own apps using an OpenZiti sdk, you don't need legit certs on any of the OpenZiti underlay servers...
That said, yes - assigning your legit LE certs to services that are protected by OpenZiti makes a TON of sense because the browser will not give you/your users the scary "this site is insecure" warning...
Answered above I think. In general, I find it of low value personally but if you publish the Ziti Admin Console on the management API then maybe it's useful? I dunno, not many people are likely to be using that ZAC so.. Doens't feel all that necessary to me...
The routers will not use the same endpoint that you might go to the controller on (the management or client APIs). They'll never know there's an alternate cert configured, it's only for HTTP-based endpoints where it might come into play. That's why with routers and with BrowZer and with the "web socket" transport, it's important because with BrowZer the web browser will try to make a connection to ws-enabled routers and those connectoins must be verifiable in some way... using a legit third party server cert from LE or other is the easiest way to solve that. But, you could also choose to install YOUR self-signed cert in your browser and that'll work fine too...
Hope all that makes sense?
Yep that makes sense to me. So I'm thinking that since the overlay network is transparent to the end users, it is more than fine to just let ziti manage its own PKI. Only what the end users will interact with (e.g. zac, underlying services exposed on 443) needs some sort of legitimate certs to avoid the insecure cert warning.
I'll try out some setups later today and see what will happen. Thanks for the explanation!
Now I think my real problem was that I was trying to use the same domain as the advertised address (aka self-signed certs) rather than something else to access zac. I now use ct.zt-example.my-domain.dev with the wildcard cert and it works? I'm not sure if I'm just being lucky or I'm doing the right thing. I think that's because ct.zt-example.my-domain.dev does not match controller.zt-example.my-domain.dev so alt certs are used instead? Does that mean I can use the wildcard cert as long as the FQDN rule is satisfied?
Yeah, easy mistake to make for sure
I'm not exactly sure what you mean, but i think "yes"? I would expect (although I haven't been able to confirm it yet by reading a spec that proves it) that a more-specific cert will be preferred over a lesser-specific match.
What I'm trying to prove is is that if you have a wildcard cert and an A record for *.zt-example.my-domain.dev and then install OpenZiti and use an advertised address as: overlay.zt-example.my-domain.dev, when OpenZiti installs itself, it will make a private, self-signed PKI for overlay.zt-example.my-domain.dev and it will configure this PKI as the "main" PKI. Then, if you use the wildcard for the alt server cert, your controller will have two certificates to choose from, either:
*.zt-example.my-domain.devoverlay.zt-example.my-domain.dev
I would expect the SNI algorithm to favor the most-specific certificate in this case -- i just can't prove it yet. I tried it out, and it SEEMS to be reliable.
Is that making sense? I'm gonna try to find a spec somewhere or figure out exactly how the golang TLS engine works to feel good about this.
You can see what I mean by probing my network.
If you run this you'll see a certificate that's valid for overlay.clint.demo.openziti.org and is self-signed
openssl \
s_client \
-connect overlay.clint.demo.openziti.org:8441 < /dev/null 2>&1 | \
openssl x509 -text | \
grep "DNS:overlay.clint.demo.openziti.org" -B1
And if you run this command, you'll get the wildcard cert back:
openssl \
s_client \
-connect zac.clint.demo.openziti.org:8441 < /dev/null 2>&1 | \
openssl x509 -text | \
grep "DNS:\*.clint.demo.openziti.org" -B1
Make sense? I'll see if i can PROVE this -- but in general it seems reasonable and in practice it seems to work as I'd expect.
Ok. I have found something that is authoritative that can help.
https://www.rfc-editor.org/rfc/rfc6125#ref-TLS-EXT
states the following:
The '*' (ASCII 42) wildcard character is allowed in subjectAltName
values of type dNSName, and then only as the left-most (least
significant) DNS label in that value. This wildcard matches any
left-most DNS label in the server name. That is, the subject
*.example.com matches the server names a.example.com and
b.example.com, but does not match example.com or a.b.example.com.
So for a precise answer that should be spec-perfect, it would be best to put your overlay network onto a subdomain so that the wildcard will not overlap. You could do something like:
ziti.controller.zt-example.my-domain.dev*.zt-example.my-domain.dev
or go the other way around:
ziti-controller.zt-example.my-domain.dev*.apps.zt-example.my-domain.dev
using a subdomain would appear to prevent any problems, assuming I've read that spec properly and the implementation is inline...
1 Like
