Integrating OpenZiti with a Golang TLS server

Building/Development
1

Situation

I have the http golang server demo working.

Next.. I wanted to add one more layer by making it TLS enabled.

So.. I found some source code to start with.

unzitified code

However.. when I started to read through the setup.. I realised that there is something that I don't understand.

Where does the certificate come from?

This sounds a bit stupid.. as like in all of the examples.. you just create it on the fly..

Though how does it relate with the certificates required for the controller and router?

I was thinking that maybe its best to create a certificate based upon the certificates from the controller and router.... so that it has the same hierarchy.

This is hinted at in the boot-strapping article.. but not specifically mentioned.

https://openziti.github.io/articles/bootstrapping-trust/part-01.encryption-everywhere.html

Hence.. as I am fumbling around... I thought to post this as a question.

If you do use the "intermediate" certificates.. from the Ziti Quick Install.. what commands do you need to use to create your certificate?

Let me know if you have any tips.

2

To further investigate, I did a search on pem files.

find /home/opc/.ziti/quickstart/instance-20220416-1603 -depth -name "*.pem"

When I did this, I noticed that a public key was created for the controller.

However.. there is a difference between ZAC and an application server.

  • ZAC is a standalone application to manage the Ziti controller

Whereas.. my situation is to create a handshake from the router to a terminating endpoint (is that how it is described?)..

  • My understanding is that the TLS handshake needs to be trusted via CSR.. so it will need to be linked back to the "self-signed" certificate authority that was created during the Quick Start installation process.

The next step that I have is to make this golang server a reverse proxy server

PS.. I saw this command in another google search to create a certificate

$ openssl x509 -req -days 365 -in localhost.csr -signkey localhost.key -out localhost.crt

which raises more question.. where is the csr file in the Quick Install folder.. as I could not find it using a search command

sudo find /home/opc/.ziti/quickstart/instance-20220416-1603 -depth -name "*.csr"

3

Now I feel a bit stupid.

  • .crt ā€” Alternate synonymous most common among *nix systems .pem (pubkey).

I have already found them using the find command above :slight_smile:

I also now realise the crt files are in the following directory

/etc/pki/tls/certs/ca-bundle.crt

Still not sure though what to put into the command like to create the SSL certificate though.

4

I had a chat with a friend this morning to piece together a few more bits of the puzzle.

  1. you need to create a key pair for the serverā€¦ this is how it is identifiedā€¦

  2. you then need to sign the key pair using the self signed certificate.

I am going to watch a few more videos to work through all of this and will post my learnings as I go

5

I was working through the "ziti-cli-functions.sh" file and stumbled across the following command

ziti pki

I was going to use something like openssl.. but it probably makes more sense to use the tools integrated with Open Ziti :slight_smile:

6

As I was working through this, I initially thought that all I needed was a server certificate.

However.. as I was walking through the function that generates the certificates.. you also need to have a client certificate..

This twigged something back in the boostrapping trust documentation... then I realised that you need both... server and client certificate.. as both are used by OpenZiti

the steps are covered in the link below

https://openziti.github.io/articles/bootstrapping-trust/part-05.bootstrapping-trust.html

unction pki_client_server {
name_local=${1-}
ZITI_CA_NAME_local=$2
ip_local=$3

if [[ "${ip_local}" == "" ]]; then
ip_local="127.0.0.1"
fi

if ! test -f "${ZITI_PKI}/${ZITI_CA_NAME_local}/keys/${name_local}-server.key"; then
echo "Creating server cert from ca: ${ZITI_CA_NAME_local} for ${name_local}"
"${ZITI_BIN_DIR-}/ziti" pki create server --pki-root="${ZITI_PKI_OS_SPECIFIC}" --ca-name "${ZITI_CA_NAME_local}"
--server-file "${name_local}-server"
--dns "${name_local},localhost" --ip "${ip_local}"
--server-name "${name_local} server certificate"
else
echo "Creating server cert from ca: ${ZITI_CA_NAME_local} for ${name_local}"
echo "key exists"
fi

if ! test -f "${ZITI_PKI}/${ZITI_CA_NAME_local}/keys/${name_local}-client.key"; then
echo "Creating client cert from ca: ${ZITI_CA_NAME_local} for ${name_local}"
"${ZITI_BIN_DIR-}/ziti" pki create client --pki-root="${ZITI_PKI_OS_SPECIFIC}" --ca-name "${ZITI_CA_NAME_local}"
--client-file "${name_local}-client"
--key-file "${name_local}-server"
--client-name "${name_local}"
else
echo "Creating client cert from ca: ${ZITI_CA_NAME_local} for ${name_local}"
echo "key exists"
fi
echo " "
}

7

I found this to be a useful video

8

This was also useful to fill in a few more gaps.

9

I have reached the conclusion that I first need to work through the following example before I try to get the certificate signed by the controller.

The reason is because the zitified http server needs some extra code to enable the TLS service.. of which... I dont have much idea what it is.. other than.. its not a simple one liner.

Does anyone have any samples of how to create a zitified tls enabled golang server?

10

Made progress with the tls golang serverā€¦ I found this to be the most useful

11

Quick update.. I think I have worked this out.. but still yet to test it

I thought to put my hypothesis out there in case others see something wrong.. as I am really the blind leading the blind

what helped is to search through the SDK folders.. I only just realised that you can do this..

As I could not find any TLS implementation.. I realised that it needs to be setup separately to the zitified listener..

well.. at least I think so.. from what I can work out

next breakthrough moment

Going through the steps to implement a Golang TLS server helped me understand what all of the config requirements are re certificates.. and the extra code required to implement.

The main line is the following which connects the http service to TLS.. where you need to pass in the certificate and private key

http.HandleFunc("/", handler)
err := http.ListenAndServeTLS(":443", "cert.pem", "key.pem", nil)

But.. now what..

Well.. here is a sample from the zitified http server.. where the http.Serve activates the ziti listener.

http.HandleFunc("/hello", hello)
http.HandleFunc("/add", add)
if err := http.Serve(createZitiListener(), nil); err != nil {
panic(err)
}

So.. if you combine these together.. this is what I believe is needed from a coding perspective.

http.HandleFunc("/hello", hello)
http.HandleFunc("/add", add)
if err := http.ListenAndServeTLS(":443", "cert.pem", "key.pem", nil); err != nil {
panic(err)
}
if err := http.Serve(createZitiListener(), nil); err != nil {
panic(err)
}

Now for the unknown piece

How do you create the certificate and private keys?

In the above example where I implemented a SSL golang server.. they used the following command

go run $(go env GOROOT)/src/crypto/tls/generate_cert.go --host=localhost

This creates two files in the directory that it is run in

  • cert.pem
  • key.pem

However.. this is not linked to the keys from the controller or router.. so if I use this.. I think its going to fail.. with an error like bad certificate.

I do know that there is a ziti command for pkis.. but are not 100% sure on the variables

ziti pki create

This is now at the edge of my knowns.. and now everything else is 100% unknown.

Let me know if you have any tips are how to finish this off :slight_smile:

12

I think I need to do something like this

ziti pki create server --??

Would this be the key.pem above?

ziti pki create csr --??

Would this be the cert.pem above?

Once created.. pass them into the server as extra variables..

Does this make sense?

13

After more research... I think I have to go back to the drawing board.. because my thinking is wrong.

I picked it up watching this video from 15:30.

https://www.youtube.com/watch?v=ZKlwKg-f__0&list=PLSak_q1UXfPp2VwUQ4ZdUVJdMO6pfi5v_&index=41

In short.. ListenAndServe is replaced with ListenAndServeTLS

You will notice this when you look at the http server example before it is zitified

if err := http.ListenAndServe(":8090", nil); err != nil {
panic(err)
}

after it is zitified.. it becomes

if err := http.Serve(createZitiListener(), nil); err != nil {
panic(err)
}

This means.. the certificate and key need to be passed into this code somewhere

func createZitiListener() net.Listener {
cfg, err := config.NewFromFile(os.Args[1])
if err != nil {
panic(err)
}
options := ziti.ListenOptions{
ConnectTimeout: 5 * time.Minute,
}
listener, err := ziti.NewContextWithConfig(cfg).ListenWithOptions(os.Args[2], &options)
if err != nil {
fmt.Printf("Error binding service %+v\n", err)
panic(err)
}
return listener
}

So I will keep doing walkabout for a while longer

14

Think I get it now..

When you enrol an identity.. it creates a json file.. which contains all of the bits and pieces.. pem.. cert.. and the cert chain etc

So.. you dont need to create a server / client key manually is the case of the golang ssl example above.

all you need to do is to create an identity.. and enroll it on the server.. which will take care of everything for you.

Well... that is what I believe for now.. I will give you an update as more unfolds.

ps.. I am starting to learn golang at the same time..

15

Yes. Sort of. OpenZiti will make a mutual TLS connection from any piece of the overlay to another. If you are using a zitified client with a zitified server - then yes thatā€™s 100% perfectly secure and fine.

Itā€™s when you bring the tunneler into the mix that itā€™ll get more complex. Accessing that service via TLS using a tunneler would require a THIRD layer of encryption, provided by ā€œthe http serverā€. Conceptually you might think of that like three pipesā€¦ sorta like thisā€¦ the inner pipe is the third layer of encryption that is NOT provided by openziti. This would be done from ā€œthe browserā€ to ā€œthe https serverā€. In my pic below that would be the yellow pipeā€¦

image
image838Ɨ360 22.6 KB

Wrapping your yellow pipe, the https protocol negotiated by your browser and the https server, is the orange pipe. This represents the end to end encryption (e2ee) that OpenZiti provides by default (you can disable it but why would you??? :slight_smile: ). Thatā€™s negotiated on the ā€œclient sideā€ (the tunneler, or inside your golang app itself, wherever) and the far side.

Finally there is the ā€œmutual TLSā€ (mTLS) encryption that you get also provided by OpenZiti which you cannot disable. This layer protects the links from being inspected by tools like Wireshark/tcpdump.

You should start by making a web server WITHOUT ziti and providing https. Thereā€™s a bunch of tutorials on the web how to do that. Once you understand that - then I think you could move over to ā€˜zitifyingā€™ that app and using OpenZiti on the server tooā€¦ Thatā€™d be my reccomendation.

1 Like
16

Thanks @TheLumberjack that's some nice Selleys work to fill in a few more gaps :slight_smile:

When I worked through the SSL golang example.. this was the main line of code required to get the TLS to work.. i.e. adding TLS to the ListenAndServe method

err := http.ListenAndServeTLS(":443", "cert.pem", "key.pem", nil)

When I was looking through the zitified http server.. I noticed that this was all embedded in the ziti library

So.. what I have done.. maybe its wrong.. is to replace the line above with the following

if err := http.Serve(createZitiListener(), nil); err != nil {
panic(err)
}

Then.. when I pass in the json and service.. it should run and be encrypted ... though I am unsure about the ssl part.

go run main-ssl.go "$HOME/golang.http.server.json" golanghttp

Now.. I did find some references to tls in the ziti code.. but that was on the client end.. so maybe there is more that I need to do.

Am I heading in the right direction?

PS> my next challenge is to make this a reverse proxy.. if that is possible

17

It'll be encrypted yes. BUT - if you use a tunneling app (a tunneller) and try to access the server you will find that you must use "http" (as in "http://your.intercept.name.here"). You WILL NOT be able to use https.

Keep playing with it and you'll probably come to discover what I mean :slight_smile: good luck!

1 Like
18

Now that is a brainteaser.

soā€¦ like its encryptedā€¦ but its over httpā€¦ thought its okā€¦ because the encryption is end to endā€¦

19

I think this is where some of my gaps start to fade awayā€¦ because it makes sense why I keep stumbling on this ssl thingā€¦ and why the json file has all of that information in it.

20

the ā€œouterā€ (yellow pipe) transport will be unencrypted http. that INSECURE protocol is only insecure until the tunneling client intercepts the packets and puts the packets onto the ziti overlay network. from there - itā€™s ā€œtunneledā€ unencrypted to the other side. since the other side is an http server itself - the payload wonā€™t ever be unencrypted anywhere except for that first local hop from browser ā†’ local ziti tunneller.

If you had a tunneling app on the server side (which you donā€™t in this example) it would leave the far side as ā€˜httpā€™ and be able to be sniffed by tcpdump/wireshark. By using ziti - thatā€™s not possible on the server side (since the e2ee goes INTO the server and never hits the local os network stack unencrypted). it WOULD be able to be sniffed by wireshark unecrypted on the ā€˜client sideā€™ if you used a tunneling app with HTTP though because between the browser and the tunneller it hits the OS network stack unencrypted (http)ā€¦ If you arenā€™t using the browser, and if you use the golang http client - at that point itā€™ll be e2ee the full wayā€¦ and encrypted but not ā€œhttpsā€.

Itā€™s complex I know - have a walkabout and I bet youā€™ll get it. cheers

1 Like