Best practices of securing controller (and ZAC)

Ziti Overlay
1

Hi there,

are there best practices to protect the controller and, if in use, ZAC?
Currently, we have ZAC running on the same instance as the controller, ZAC being accessible only after the port was opened - my next step would be to zitify it and make it accessible by the authorized identities.

I also noticed that the management API is running on the same port as the client API, I assume that’s something that should be changed to prevent additional exposure, because as I understood, someone currently can access the controller management if he knows username+pw?

{"data":{"apiVersions":{"edge":{"v1":{"apiBaseUrls":["https://my-controller.com:8441/edge/client/v1"],"path":"/edge/client/v1"}},"edge-client":{"v1":{"apiBaseUrls":["https://my-controller.com:8441/edge/client/v1"],"path":"/edge/client/v1"}},"edge-management":{"v1":{"apiBaseUrls":["https://my-controller.com:8441/edge/management/v1"],"path":"/edge/management/v1"}}},"buildDate":"2022-10-13T15:31:04Z","revision":"72978b5aa932","runtimeVersion":"go1.19.1","version":"v0.26.10"},"meta":{}}

How would I go about changing that? Any extra recommendations?

1 Like
2

Exact my thoughts - this is why I used to separate ports in the helm chart for client and mgmt api :rofl:

You can define two separate web endpoints for client and management api in the controller config:
client api:
https://github.com/openziti/helm-charts/blob/main/charts/ziti-controller/templates/configmap.yaml#L172

management api:
https://github.com/openziti/helm-charts/blob/main/charts/ziti-controller/templates/configmap.yaml#L240

just set .Values.managementApi.service.enabled to false in your mind when reading the template :wink:

Cheers

Ps: I had to escape the links as i’m not allowed to post links…

1 Like
3

Thanks @marvkis - that worked!
Now I also have two different ports defined with the one for the management API only being accessible to ZAC, which will be zitified.
You can still see where the management interface is running but it’s not accessible from outside - I guess that’s fine:

{"data":{"apiVersions":{"edge":{"v1":{"apiBaseUrls":["https://my-controller.com:8441/edge/client/v1"],"path":"/edge/client/v1"}},"edge-client":{"v1":{"apiBaseUrls":["https://my-controller.com:8441/edge/client/v1"],"path":"/edge/client/v1"}},"edge-management":{"v1":{"apiBaseUrls":["https://my-controller.com:8440/edge/management/v1"],"path":"/edge/management/v1"}}},"buildDate":"2022-10-13T15:31:04Z","revision":"72978b5aa932","runtimeVersion":"go1.19.1","version":"v0.26.10"},"meta":{}}

Here’s my very simple config with both ports separated:

v: 3
db:                     "/myuser/.ziti/quickstart/zt/db/ctrl.db"
identity:
  cert:                 "/myuser/.ziti/quickstart/zt/pki/zt-intermediate/certs/zt-client.cert"
  server_cert:          "/myuser/.ziti/quickstart/zt/pki/zt-intermediate/certs/zt-server.chain.pem"
  key:                  "/myuser/.ziti/quickstart/zt/pki/zt-intermediate/keys/zt-server.key"
  ca:                   "/myuser/.ziti/quickstart/zt/pki/cas.pem"
ctrl:
  listener:             tls:0.0.0.0:6262
mgmt:
  listener:             tls:0.0.0.0:10000
healthChecks:
  boltCheck:
    interval: 30s
    timeout: 20s
    initialDelay: 30s
edge:
  api:
    sessionTimeout: 30m
    address: my-controller.com:8441
  enrollment:
    signingCert:
      cert: /myuser/.ziti/quickstart/zt/pki/zt-signing-intermediate/certs/zt-signing-intermediate.cert
      key:  /myuser/.ziti/quickstart/zt/pki/zt-signing-intermediate/keys/zt-signing-intermediate.key
    edgeIdentity:
      duration: 180m
    edgeRouter:
      duration: 180m
web:
  - name: client-management
    bindPoints:
      - interface: 0.0.0.0:8441
        address: my-controller.com:8441
    identity:
      ca:          "/myuser/.ziti/quickstart/zt/pki/my-controller.com-intermediate/certs/my-controller.com-intermediate.cert"
      key:         "/myuser/.ziti/quickstart/zt/pki/my-controller.com-intermediate/keys/my-controller.com-server.key"
      server_cert: "/myuser/.ziti/quickstart/zt/pki/my-controller.com-intermediate/certs/my-controller.com-server.chain.pem"
      cert:        "/myuser/.ziti/quickstart/zt/pki/my-controller.com-intermediate/certs/my-controller.com-client.cert"
    options:
      idleTimeout: 5000ms  #http timeouts, new
      readTimeout: 5000ms
      writeTimeout: 100000ms
      minTLSVersion: TLS1.2
      maxTLSVersion: TLS1.3
    apis:
      - binding: edge-client
        options: { }
      - binding: fabric
        options: { }
  - name: management
    bindPoints:
      - interface: 0.0.0.0:8440
        address: my-controller.com:8440
    options:
      idleTimeout: 5000ms  #http timeouts, new
      readTimeout: 5000ms
      writeTimeout: 100000ms
      minTLSVersion: TLS1.3
      maxTLSVersion: TLS1.3
    apis:
      - binding: edge-management
        options: { }
4

You can turn ZAC dark by creating your own private end point. I believe there was a demo about this a while back.

While this makes some progress… it still allows the controller API to be accessed from another machine…

To further lock it down… you can close the ZAC port… which stops ZAC from being used… and you need to use the CLI instead.

After a while… you tend to prefer the CLI… as you can do a lot more automation that way.

A few thoughts anyway… I am sure others may also have other ideas.

PS>. I believe there is a way to make ZAC use the Ziti SDK… which would probably be the best solution… but I am not sure of the specific changes to activate it…

5

PS… I believe best practice is to have the controller and public edge router on separate computes… which are configured to use ports 80 and 443… to obfuscate traffic. I know it can be done… you just need to play around with it to get it to work how you want.

6

sorry about that. Discourse is "extra aggressive" sometimes when it comes to things it thinks are spam. I keep trying to tune down that spam-filtering-ness of it... I tuned it lower again after seeing this and seeing a bunch of your posts got flagged...

@jeremy.tellier where are we at with re-enabling this feature from ZAC? I know Curt enabled it at some point, it'd be nice to know how to turn the ZAC fully dark.

7

In my meeting with @dmuensterer yesterday, I discussed how I had integrated the Ziti SDK for NodeJS into ZAC, and how this SDK has the ability to easily allow NodeJS/Express apps (like ZAC) to host Ziti services, thus listening only for incoming Ziti connections, not on open TCP ports.

To enable this feature in ZAC you just need to set two env vars before starting up ZAC:

ZITI_SERVICE_NAME should be set to the name of the Service ZAC is hosting (it will default to ‘zac’)

ZITI_IDENTITY_FILE should be set to the full file-system path to the identity.json file for the ZAC’s Identity

2 Likes
8

@curt How do I enabled both simultaneously?

When expressing the run configuration with Helm it’s simpler to have an additive logic like listen normal; if ziti; then listen ziti; endif than branching logic like if ziti; then listen ziti; else listen normal; endif.

Either can be expressed, but in Kubernetes it’s not the fact of a server listener that effects exposure, but rather an additional concept of load balancer or ingress resource. In other words, even if the console has an underlay listener, it’s not necessarily exposed to anything outside the cluster.

9

A bit more context: apps running inside a “smart” context like an orchestrator in general or k8s specifically, often benefit from having a liveness probe available on underlay for the orchestrator to know whether the app is healthy or needs to be restarted. An insecure underlay listener is useful for that cluster-internal health check even if the only exposure outside the cluster is via Ziti.

10

If you are asking about how to enable both a TCP port listener simultaneously with a Ziti Connection listener, then ZAC (or more specifically, the Ziti NodeJS SDK) does not currently support that without additional code changes to the app.

That is, if a Node app wanted to be dark and listen for Ziti connections for “normal” request processing, but it also wanted to expose a public TCP port listener and a dedicated health-check request router, then additional code would need to be there to facilitate the “public” health-check processor.

11

Not too familiar with zitifying apps so far as I’ve only used the tunnelers before:
Which host configuration do I need to set up to let out the traffic on the correct port - HTTP, HTTPS or another port?
Looks like neither a host config for 8443 not 1408 seems to work. What am I missing?
I’ve set both env variables in zt.env…

12

Doesn’t seem to be working for me. @curt or @jeremy.tellier can you fellas try?

I tried to enable this in my own environment right now. First issue I hit was I had to run npm install openziti/ziti-sdk-nodejs. I was getting:

Mar 08 01:20:47 ip-172-31-11-231 systemd[1]: Started Ziti-Console.
Mar 08 01:20:48 ip-172-31-11-231 node[173427]: Error: Unable to import module @openziti/ziti-sdk-nodejs
Mar 08 01:20:48 ip-172-31-11-231 node[173427]:     at loadModule (file:///root/.ziti/quickstart/ip-172-31-11-231/ziti-console/server.js:29:10)

After that I hit another error:

Mar 08 01:22:35 ip-172-31-11-231 node[174052]: Loading Settings File From: /root/.ziti/quickstart/ip-172-31-11-231/ziti-console/../ziti/settings.json
Mar 08 01:22:35 ip-172-31-11-231 node[174052]: file:///root/.ziti/quickstart/ip-172-31-11-231/ziti-console/server.js:1511
Mar 08 01:22:35 ip-172-31-11-231 node[174052]:         }).on('error', function(err) {
Mar 08 01:22:35 ip-172-31-11-231 node[174052]:           ^
Mar 08 01:22:35 ip-172-31-11-231 node[174052]: TypeError: Cannot read properties of undefined (reading 'on')
Mar 08 01:22:35 ip-172-31-11-231 node[174052]:     at StartServer (file:///root/.ziti/quickstart/ip-172-31-11-231/ziti-console/server.js:1511:4)
Mar 08 01:22:35 ip-172-31-11-231 node[174052]:     at file:///root/.ziti/quickstart/ip-172-31-11-231/ziti-console/server.js:1496:1
Mar 08 01:22:35 ip-172-31-11-231 node[174052]: Node.js v18.14.0
Mar 08 01:22:35 ip-172-31-11-231 systemd[1]: ziti-console.service: Main process exited, code=exited, status=1/FAILURE
Mar 08 01:22:35 ip-172-31-11-231 systemd[1]: ziti-console.service: Failed with result 'exit-code'.
13

assuming your ZAC started properly (unlike mine) once you have a 'zitified ZAC' you'll need to define an intercept and grant access to ZAC via ziti itself. For example, here was what I did using the ziti cli. You can see I made a service and planned to intercept it (using ziti) on http://zac.ziti:80.

service_name=zac
the_port=80

ziti edge create identity device ${service_name}.user -o ${service_name}.user.jwt -a "${service_name}.binders"
ziti edge enroll ${service_name}.user.jwt

ziti edge create config "${service_name}.host.v1" host.v1 \
  '{"protocol":"tcp", "address":"localhost","port":'"${the_port}"'}'
ziti edge create config "${service_name}.intercept.v1" intercept.v1 \
  '{"protocols":["tcp"],"addresses":["'"${service_name}.ziti"'"], "portRanges":[{"low":'"${the_port}"', "high":'"${the_port}"'}]}'
ziti edge create service "${service_name}" \
  --configs "${service_name}.intercept.v1","${service_name}.host.v1"
ziti edge create service-policy "${service_name}-binding" Bind \
  --service-roles "@${service_name}" \
  --identity-roles "#${service_name}.binders"
ziti edge create service-policy "${service_name}-dialing" Dial \
  --service-roles "@${service_name}" \
  --identity-roles "#${service_name}.dialers"

Did your ZAC start cleanly? I also edited my /etc/systemd/system/ziti-console.service file and added these two lines to it to start it:

Environment="ZITI_SERVICE_NAME=zac"
Environment="ZITI_IDENTITY_FILE=/root/zac.user.json"
1 Like
14

Ahaa. Interesing - I wasn’t sure how ZAC and the rest of Ziti communicates, if so.
I didn’t set the environment variables for the service, but in zt.env.

Let me try real quick if I get the same error as you when using systemd env variables.

15

Hmm, still getting the same import error even after successfully executing npm install openziti/ziti-sdk-nodejs.
Not sure, what the reason for that is - not too familiar with npm but I assume it’s looking to import the dependency from the wrong path…

16

were you in your ziti-console directory when you ran that? if you use the quickstart and have the environment variables sourced, it’s usually at $ZITI_HOME/ziti-console. It’s probably not worth it anyway until me/@curt/@jeremy.tellier can power through “the problem”. :slight_smile:

We’ll let you know when we do.

17

Jup, same folder and I can also see it being installed under node_module.
Thanks for checking the error

18

@dmuensterer we just pushed some updates to the ZAC repo. The latest is now v2.6.7.

This build corrects some things that prevented you from running ZAC to host a Ziti service and listen only for incoming Zit connections, as I described above.

Please pull the latest from GitHub, then try again and let us know how it goes.

1 Like
19

btw, after you pull the latest ZAC from GitHub, then just run npm i in your local ZAC repo. That will pull down the necessary Ziti SDK for NodeJS dependency you will need to run a Zitified ZAC.

1 Like
20

I’m on 2.6.6 now, not sure what the problem is:

Mar 08 20:40:11 zt node[1209311]: Error: Unable to import module @openziti/ziti-sdk-nodejs
Mar 08 20:40:11 zt node[1209311]:     at loadModule (file:///home/ziti/.ziti/quickstart/zt/ziti-console/server.js:29:10)
Mar 08 20:40:11 zt node[1209311]:     at async file:///home/ziti/.ziti/quickstart/zt/ziti-console/server.js:38:9
Mar 08 20:40:11 zt systemd[1]: ziti-console.service: Succeeded.

ls shows that the package is installed:

ls /home/ziti/.ziti/quickstart/zt/ziti-console/node_modules/@openziti/ziti-sdk-nodejs/
API_REFERENCE.hbs  CODE_OF_CONDUCT.md  LICENSE	   README.md	appveyor.yml-obsolete  build	   lib		 scripts  tests    ziti.png
API_REFERENCE.md   CONTRIBUTING.md     README.hbs  SECURITY.md	binding.gyp	       jsdoc.json  package.json  src	  ziti.js