Making ZAC dark

I have used a golang reverse proxy to make ZAC accessible over a private DNS.

However.. when I close the port on the VCN... I lose access to the console.

I believe this is happening because I have the reverse proxy located in the wrong location.. rather than being hosted on the same server..

I believe it needs to be hosted within the Docker instance.

However.. I am not really sure how to access this instance.. so that I can add my reverse proxy app into it.. and retest.. to confirm that this solved the problem

Let me know if you have any tips

@markamind - what is the configuration that you have? Are you dockering everything within the docker-compose environment, and what do you have?

I will speak with regards to the simplified-docker-compose as that is what I am currently using. In the docker-compose I have 3 services running: controller (ziti-controller), edge router (ziti-edge-router) and console: ziti-console (ziti-console).

These are all attached to the zitiblue and zitired networks. This means that they can all talk too each other without having to go to the host network (ie stays within docker layer).
I have created a tunnel to feed this through without breaking to the host, by these commands:

ziti edge create config zac.intercept.v1 intercept.v1 '{"protocols":["tcp"],"addresses":["zac.ziti"], "portRanges":[{"low":443, "high":443}]}'
ziti edge create config zac.host.v1 host.v1 '{"protocol":"tcp", "address":"ziti-console", "port":8443}'
ziti edge create service zac.svc --configs zac.intercept.v1,zac.host.v1
ziti edge create service-policy zac.policy.dial Dial --service-roles "@zac.svc" --identity-roles '#zac-clients'
ziti edge create service-policy zac.policy.bind Bind --service-roles '@zac.svc' --identity-roles "@ziti-edge-router"

What I like about this, is that I am doing port translation transparently on the way through, that is, I can use normal HTTPS request, ie https://zac.ziti and it translates it through to port 8443 at the other end. I use the address of ziti-console in the host.v1 declaration as this is resolvable (by docker) to the containers on that network. Hence, when I restart docker and it changes the IP address around, it will always find it.

So, now, I can get the zitified ZAC console by going to : https://zac.ziti and I get the ZAC console. The address of the controller (when connecting) is: https://ziti-controller:1280

Since I have this going, I can now make the 8443 port invisible to the private network, but commenting out the port command in docker compose:

  ziti-console:
    image: openziti/zac
    environment:
      - ZAC_SERVER_CERT_CHAIN=/openziti/pki/ziti-controller-intermediate/certs/ziti-controller-server.cert
      - ZAC_SERVER_KEY=/openziti/pki/ziti-controller-intermediate/keys/ziti-controller-server.key
    depends_on:
      - ziti-controller
    ports:
      - "1408:1408"
#      - "18443:8443"

and restarting the console (I needed to use 18443 as it was colliding with another service on 8443). So, now ZAC is dark on host network that is running docker. Caveat: While it is dark to the network, it is not dark to the host (where docker is running), because the local host can route to all docker networks. You can confirm this, by typing docker inspect docker_ziti-console-1 and it will show you the IP address that the container is running on. If you open a browser on host you can connect to it on 8443.

Having ZAC dark is a chicken and edge condition. You cannot make ZAC dark until you get some clients connected to it. You cannot get clients connected to it while it is dark. So the process would be to keep it attached to a local network, then when get the ZAC console going through ziti, then make it dark.

2 Likes

@gooseleggs - nicely done! One thing documented in a readme you might be interested in (which wonā€™t be included in the docker stuff yet) would be to split off the management API from the ā€œedge clientā€ API. In release 0.20.0, the APIs were made separately hostable. That means you could reduce the exposure to management/configuration API as well. You could opt to listen to /management calls on 127.0.0.1, for example, and not listen on 0.0.0.0 - or listen on a different port entirely from the ā€œclientā€ API. Itā€™s documented decently in the changelog, but there are no examples or ā€˜how-tosā€™ at this time. You can read about it in the relevant changelog entry.

@markamind, Could you diagram how youā€™re deploying ZAC and using ziti here? What port are you closing in the cloud? 8443? I donā€™t quite have enough vision to understand what went wrong for you.

Can you open developer tools and see ā€˜whatā€™ might be timing out? I need a bit more to go on.

Thanks!

1 Like

This is awesome feedbackā€¦ I fumble along with an aspirational goal and an open mind. and then learn something every now and then that makes a few things stick.

For instance, I was thinkingā€¦ why do I need a reverse proxyā€¦ I should be able to set up this intercept via a routerā€¦ and then @gooseleggs provided the steps to do just that. Thanks :slight_smile:

I dont really know much about Dockerā€¦ other than it virtualises an OSā€¦ and I get lost when it comes to configuring the networking layer.

Soā€¦ I will check this out in more detail.

In relation to the configuration, you will find a diagram below.

Rather than configure the edge router, I have a custom golang reverse proxy that acts as a Ziti hostā€¦ this is what creates the terminator for the private DNS.

One advantage of doing this over an approach using a router is that you can customise headers, add extra logging, automate authentication and do all sorts of extra data manipulationā€¦ howeverā€¦ it does come at a cost

in relation to my situation, ZAC is hosted in Docker which is how its installed as a part of the standard implementation.

https://openziti.github.io/ziti/quickstarts/zac/installation.html

In this configuration, ZAC runs on port 8443. When I leave port 8443 open in the VCNā€¦ I can access ZAC over a private DNS.

The specific problem that I am working through is that it stops working when port 8443 on the VCN is closed.

  1. all of the components are all located on the same compute within the same virtual cloud network VCN

  2. ZAC operates in Docker containerā€¦ because of this, I believe the traffic interfaces the VCN firstā€¦ where the traffic is then re-routed to the private DNS via the reverse proxy.

  3. Hence, when I close port 8443 on the VCNā€¦ the private DNS no longer worksā€¦ because the reverse proxy can no longer access the ZAC console.

To resolve this, I wan to install my reverse proxy server in the docker containerā€¦ so that its on the same dockerised subbetā€¦ if that makes sense. My understandingā€¦ is that it should then just work.

In relation to splitting the management API from the client API.. would this require two different ZAC installations on two different computes

where each compute has a separate configuration?

actually.. I just realised that this is for the controller.. not ZAC..

I wonder if you're closing this port not only to inbound from the internet but also inbound from the private network as well? From the machine running your proxy, with port 8443 closed can the proxy get to ZAC? If yes - then I would try what @gooseleggs wrote up and see if you still have issues. If you do, look at the logs from the router that's trying to access ZAC. It might provide the missing information.

No, not at all. Every computer that's connected to the internet will have two different interfaces available to you. You'll have the "loopback" or "localhost" or "127.0.0.1" and you'll have 'whatever' the IP address is, let's call it 1.2.3.4...

For example, if you any quickstart for your setup, the controller config will have a section called "web.name.bindPoints" and in there will be an 'interface'. By default, it'll listen on ALL interface by using 0.0.0.0. However you could choose to change this (just beware, if you "do it wrong", you'll know because suddenly things won't respond any more lol).

If you change this to 1.2.3.4 - well then suddenly you won't be able to access your controller API using 127.0.0.1 because you have bound to the IP address specifically, which won't be on the "loopback" interface. Or conversely, you change this to 127.0.0.1 - now "nothing" will work from anywhere other than localhost because you're binding specifically to the loopback, the one that has IP 127.0.0.1 assigned to it...

That's what you can do for the management API... You can leave the "client" API bound to 0.0.0.0 but keep the "management" API bound to 127.0.0.1. That would prevent changes to your network from anywhere except for when the traffic is on the local machine.

I didn't test this (shame on me) but that means you would change the config from looking like this:

web:
  - name: client-management
    bindPoints:
      - interface: 0.0.0.0:8441
        address: ec2-18-188-201-183.us-east-2.compute.amazonaws.com:8441
    identity: (removed section for brevity)
    options: (removed section for brevity)
    apis:
      - binding: fabric
        options: { }
      - binding: health-checks
        options: { }
      - binding: edge-management
        options: { }
      - binding: edge-client
        options: {

to look like this:

web:
  - name: management-apis
    bindPoints:
      - interface: 127.0.0.1:5309
        address: "127.0.0.1:5309"
    identity: (removed section for brevity)
    options: (removed section for brevity)
    apis:
      - binding: fabric
        options: { }
      - binding: health-checks
        options: { }
      - binding: edge-management
        options: { }
#      - binding: edge-client ----- notice this is commented out
#        options: { } ----- notice this is commented out
  - name: client-apis
    bindPoints:
      - interface: 0.0.0.0:8441
        address: ec2-18-188-201-183.us-east-2.compute.amazonaws.com:8441
    options:
    apis:
      - binding: edge-client
        options: { }

Now your management API is only accessible from "localhost" but your "edge" API is still accessible from anywhere. Obviously if you do this - it affects ZAC since it makes heavy use of the management api.

1 Like

Nice.. I can see how this could work.. will play around with this a bit more now.

I was looking through the ZAC installation... and I think it does not use Docker.. as I could not find Docker installed on the compute.. I think its a NodeJS installation.. but somehow.. it still uses the Dockerfile for configurations.

When I close port 8443.. I do it in the VCN... as opposed to closing the firewall on the compute..

So.. somehow.. something is happening not as I expected.. I will try a few more options to see what is happening.. as if its all local on the machine.. it should not make any difference what is changed in the VCN.. or at least that is what I understand

It uses whatever you want it to. The "host it anywhere" quickstart doesn't use docker, I think you followed that one. You won't be using docker if you didn't go out of your way to run it. :slight_smile:

Good luck on the port closing of the VCN. I don't think I can help you much there, unfortunately.

1 Like

worked it outā€¦ wowā€¦ so happy now.

you will never guessā€¦ all I had to do was to make an entry in the hosts file.

what was happening is that I was accessing ZAC from an external IP addressā€¦ so when I closed the portā€¦ I could no longer access itā€¦ and I had no record in the hosts file to attach it to the local host.

As soon as I did thisā€¦ bingoā€¦ dark ZAC

PSā€¦ with what I have now setupā€¦ I see no chicken an egg problem with ZACā€¦ and I see no reason why it cannot be defaulted to a dark config from step 0.

Soā€¦ I thought to ask for more details as to whyā€¦ ZAC cannot be made dark before the admin user is setup

For instanceā€¦ before you setup ZACā€¦ all you need to do is to use the ziti command line to create an identity, enroll it on the local machine and the configure it as a host for a private DNS

With this configuredā€¦ you can install ZACā€¦ without opening the portā€¦ and access it over a private DNSā€¦

either through a proxy routerā€¦ or a reverse proxyā€¦ or other alternative you prefer.

Does this make sense?

If soā€¦ how can I help you to have this configured as the default config for ZACā€¦ as I now see no reason why this config should no longer be the default.

Think back to your first interactions with OpenZiti. Keeping ZAC visible, until you, the learner, are ready to turn it dark is very useful for exploring the offering. Turning ZAC dark is one of the articles/learning guides I have planned, along with splitting the APIs for the controller. It'll happen one day but probably not terribly soon.

1 Like

@jeremy.tellier - is there a way to "forget" ziti controllers? For @markamind here, it'd make sense for him to "forget" his old controller and "add" a new one that uses "localhost:8441" (in this case). Then he wouldn't need the hosts entry. I know under the dropdown there's a "connect to a new Edge Controller" option but I don't know how to forget one.

image

1 Like

Connect a new one, then use the ā€œSettings > Server Settingsā€ screen to remove the unwanted controllers.

2 Likes

Are you open to us (non ziti people) to submit howtos/kb articles or such somehow that you can then get and then publish. Would this help?

1 Like

This would be a huge help and greatly appreciated :face_holding_back_tears:! We always encourage outside help. @TheLumberjack is more familiar with the doc so Iā€™ll let him reply with the specifics of where to place the docs and how to format them to his liking.

The docs are written using docfx but they are mostly just markdown. You can check out how the existing docs are written by perusing the GitHub - openziti/ziti-doc: Documentation describing the usage of the Ziti platform. repository. The ziti-doc repo is the content that generates the www.openziti.github.io website.

1 Like

Would it help?!?!?! Like @gberl002 said it'd be awesome and greatly appreciated!!! The GitHub - openziti/ziti-doc: Documentation describing the usage of the Ziti platform. readme "should" (I hope) give you enough to get started. It's written with docfx currently which is "on me" for sure. I had high hopes for it - it was getting a lot of love from Microsoft for a while but that 'love' has dried up... :frowning: That's what we have for now though.

If you're a linux person you'll need the legit version of mono. The docfx install steps clearly tell you to get the full version, not the one from the OS distro.

If you're a windows person, I'd really recommend you install WSL2, install the full version of mono and then docfx to get moving. If you do that the gendoc.sh shell script at the root of the checkout should work for you just fine.

If you run into any bumps along the way - I will 100% help you out over any hurdle! Even if you just 'wrote the markdown' and plunked it somewhere, put a PR up, that'd be amazing.

I'm very much looking forward to anything you want to contribute! Is it "how to make ZAC dark???" :slight_smile: :slight_smile:

Cheers

I suppose there is the simple question. Can we eat our own dog food? By this, I mean, should there be an option within the console (and config variables in Docker) to make it dark? There have been a couple of options posted in this thread that are all externally done to achieve this. How I would think it could go:

  1. Button in the console
  2. Enter the URL that you want the console to listen on.
  3. Enter the group name that clients need to be a part of the access console
  4. hit the go button
  5. Background process enrolls into controller
  6. Background process creates access group if not created
  7. Background process puts current user into that group if no one in that group
  8. User then restarts the ZAC and then it listens as an application endpoint and does not ā€˜fireā€™ up other listening endpoints.
  9. Have a backdoor configuration variable to break it out of this mode as a breakglass recovery

Just a random thought

Neat idea. @jeremy.tellier is starting to play with "recipes". Seems like a natural recipe to have
image

@jeremy.tellier would this be a something that fits recipes? It's sort of a new idea he's been playing with, I don't actually know if it would fit but maybe.

the bit to ACTUALLY turn off the ACL in the firewall etc should be manual in my opinion. That's too hard to script "right" and probably a lot of effort to do it 'everywhere'.

This is a really good point.. and I appreciate your position.

To make it simpler.. one option is to come up with a series of challenges for newbies.. that progressively get harder .. this could be integrated into the recipes section.

  1. install the controller and ZAC
  2. setup ziti ssh, scp, ping, cat examples, reflex example
  3. setup a private DNS using a router
  4. make ZAC dark
  5. integrate a Ziti SDK
  6. create a Ziti host
  7. create a Ziti dialer
  8. build a multi cloud fabric
  9. implement HA configuration
  10. implement network monitoring
  11. monitor and troubleshoot network performance
  12. implement posture checks
  13. integrate external Certificate Authority

etc

I think this would help a lot of people that are new.. as it will help build a foundation in the basics

Just a thought.

1 Like