Browzer with keycloack and self-signed certs

BrowZer
1

Hello,

Watching this video: https://www.youtube.com/watch?v=ti1w7dQ3gSY&ab_channel=OpenZiti

Would it work to combine the the docker compose simplified quickstart with browzer and keycloack using self-signed certs only?

Thanks!

2

Hi @CarlosHleb, welcome to the community and to OpenZiti and Browzer!

You could get it working with self-signed certs only, sure, but it's going to force you to add the self-signed CA into any and all clients that use Browzer...

The reason the WSS-enabled edge router needs an independently verifiable certificate is so that "any" browser would work. If you want to have your users install the self-signed CA, then then things should work fine.

Hopefully that makes sense and makes it clear why it's needed?

1 Like
3

Hello,

I am having issues with adding the self signed controllers certificats to browzer.
Here's my docker-compose:

services:
  ziti-controller:
    image: "${ZITI_IMAGE}:${ZITI_VERSION}"
    healthcheck:
      test: curl -m 1 -s -k -f https://${ZITI_CTRL_EDGE_ADVERTISED_ADDRESS:-ziti-edge-controller}:${ZITI_CTRL_EDGE_ADVERTISED_PORT:-1280}/edge/client/v1/version
      interval: 1s
      timeout: 3s
      retries: 30
    env_file:
      - ./.env
    ports:
      - ${ZITI_INTERFACE:-0.0.0.0}:${ZITI_CTRL_EDGE_ADVERTISED_PORT:-1280}:${ZITI_CTRL_EDGE_ADVERTISED_PORT:-1280}
      - ${ZITI_INTERFACE:-0.0.0.0}:${ZITI_CTRL_ADVERTISED_PORT:-6262}:${ZITI_CTRL_ADVERTISED_PORT:-6262}
    environment:
      - ZITI_CTRL_NAME=${ZITI_CTRL_NAME:-ziti-edge-controller}
      - ZITI_CTRL_EDGE_ADVERTISED_ADDRESS=${ZITI_CTRL_EDGE_ADVERTISED_ADDRESS:-ziti-edge-controller}
      - ZITI_CTRL_EDGE_ADVERTISED_PORT=${ZITI_CTRL_EDGE_ADVERTISED_PORT:-1280}
      - ZITI_CTRL_EDGE_IP_OVERRIDE=${ZITI_CTRL_EDGE_IP_OVERRIDE:-127.0.0.1}
      - ZITI_CTRL_ADVERTISED_PORT=${ZITI_CTRL_ADVERTISED_PORT:-6262}
      - ZITI_EDGE_IDENTITY_ENROLLMENT_DURATION=${ZITI_EDGE_IDENTITY_ENROLLMENT_DURATION}
      - ZITI_ROUTER_ENROLLMENT_DURATION=${ZITI_ROUTER_ENROLLMENT_DURATION}
      - ZITI_USER=${ZITI_USER:-admin}
      - ZITI_PWD=${ZITI_PWD}
    networks:
      ziti:
        aliases:
          - ziti-edge-controller
    volumes:
      - ziti-fs:/persistent
    entrypoint:
      - "/var/openziti/scripts/run-controller.sh"

  ziti-controller-init-container:
    image: "${ZITI_IMAGE}:${ZITI_VERSION}"
    depends_on:
      ziti-controller:
        condition: service_healthy
    environment:
      - ZITI_CTRL_EDGE_ADVERTISED_ADDRESS=${ZITI_CTRL_EDGE_ADVERTISED_ADDRESS:-ziti-edge-controller}
      - ZITI_CTRL_EDGE_ADVERTISED_PORT=${ZITI_CTRL_EDGE_ADVERTISED_PORT:-1280}
    env_file:
      - ./.env
    networks:
      ziti:
    volumes:
      - ziti-fs:/persistent
    entrypoint:
      - "/var/openziti/scripts/run-with-ziti-cli.sh"
    command:
      - "/var/openziti/scripts/access-control.sh"

  ziti-edge-router:
    image: "${ZITI_IMAGE}:${ZITI_VERSION}"
    env_file:
      - ./.env
    depends_on:
      ziti-controller:
        condition: service_healthy
    ports:
      - ${ZITI_INTERFACE:-0.0.0.0}:${ZITI_ROUTER_PORT:-3022}:${ZITI_ROUTER_PORT:-3022}
      - ${ZITI_INTERFACE:-0.0.0.0}:${ZITI_ROUTER_LISTENER_BIND_PORT:-10080}:${ZITI_ROUTER_LISTENER_BIND_PORT:-10080}
    environment:
      - ZITI_CTRL_ADVERTISED_ADDRESS=${ZITI_CTRL_ADVERTISED_ADDRESS:-ziti-controller}
      - ZITI_CTRL_ADVERTISED_PORT=${ZITI_CTRL_ADVERTISED_PORT:-6262}
      - ZITI_CTRL_EDGE_ADVERTISED_ADDRESS=${ZITI_CTRL_EDGE_ADVERTISED_ADDRESS:-ziti-edge-controller}
      - ZITI_CTRL_EDGE_ADVERTISED_PORT=${ZITI_CTRL_EDGE_ADVERTISED_PORT:-1280}
      - ZITI_ROUTER_NAME=${ZITI_ROUTER_NAME:-ziti-edge-router}
      - ZITI_ROUTER_ADVERTISED_ADDRESS=${ZITI_ROUTER_ADVERTISED_ADDRESS:-ziti-edge-router}
      - ZITI_ROUTER_PORT=${ZITI_ROUTER_PORT:-3022}
      - ZITI_ROUTER_LISTENER_BIND_PORT=${ZITI_ROUTER_LISTENER_BIND_PORT:-10080}
      - ZITI_ROUTER_ROLES=public
    networks:
      - ziti
    volumes:
      - ziti-fs:/persistent
    entrypoint: /bin/bash
    command: "/var/openziti/scripts/run-router.sh edge"

  ziti-console:
    image: openziti/zac
    working_dir: /usr/src/app
    environment:
      - ZAC_SERVER_CERT_CHAIN=/persistent/pki/${ZITI_CTRL_EDGE_ADVERTISED_ADDRESS:-ziti-edge-controller}-intermediate/certs/${ZITI_CTRL_EDGE_ADVERTISED_ADDRESS:-ziti-edge-controller}-server.cert
      - ZAC_SERVER_KEY=/persistent/pki/${ZITI_CTRL_EDGE_ADVERTISED_ADDRESS:-ziti-edge-controller}-intermediate/keys/${ZITI_CTRL_EDGE_ADVERTISED_ADDRESS:-ziti-edge-controller}-server.key
      - ZITI_CTRL_EDGE_ADVERTISED_ADDRESS=${ZITI_CTRL_EDGE_ADVERTISED_ADDRESS:-ziti-edge-controller}
      - ZITI_CTRL_EDGE_ADVERTISED_PORT=${ZITI_CTRL_EDGE_ADVERTISED_PORT:-1280}
      - ZITI_CTRL_NAME=${ZITI_CTRL_NAME:-ziti-edge-controller}
      - PORTTLS=8443
    depends_on:
      ziti-controller:
        condition: service_healthy
    ports:
      - ${ZITI_INTERFACE:-0.0.0.0}:8443:8443
    volumes:
      - ziti-fs:/persistent
    networks:
      - ziti

  keycloack:
      image: quay.io/keycloak/keycloak:25.0.0
      command: 
      - "start-dev"
      - "--https-certificate-file=/docker-data/example.com.crt"
      - "--https-certificate-key-file=/docker-data/example.com.key"
      - "--https-port=8081"
      ports:
        - 8081:8081
      environment:
        - KEYCLOAK_ADMIN=admin
        - KEYCLOAK_ADMIN_PASSWORD=rootroot
      networks:
        - ziti
      volumes:
        - ./docker-data:/docker-data

  browzer:
    image: ghcr.io/openziti/ziti-browzer-bootstrapper:latest
    restart: always
    depends_on:
      ziti-controller:
        condition: service_healthy
    networks:
      - ziti

    volumes:
      - ./docker-data:/docker-data
      - ./ziti-fs:/persistent

    ports:
      - "443:443"

    environment:
      NODE_EXTRA_CA_CERTS: /persistent/pki/ziti-edge-controller-intermediate/certs/ziti-edge-controller-server.chain.pem
      # NODE_TLS_REJECT_UNAUTHORIZED: 0
      NODE_ENV: production
      ZITI_BROWZER_BOOTSTRAPPER_LOGLEVEL: debug
      ZITI_BROWZER_RUNTIME_LOGLEVEL: debug
      ZITI_CONTROLLER_HOST: ziti-edge-controller.com
      ZITI_CONTROLLER_PORT: 1280
      ZITI_BROWZER_BOOTSTRAPPER_HOST: example.com
      ZITI_BROWZER_BOOTSTRAPPER_LISTEN_PORT: 443
      ZITI_BROWZER_BOOTSTRAPPER_CERTIFICATE_PATH: /docker-data/example.com.crt
      ZITI_BROWZER_BOOTSTRAPPER_KEY_PATH: /docker-data/example.com.key
      ZITI_BROWZER_BOOTSTRAPPER_SCHEME: https
      ZITI_BROWZER_BOOTSTRAPPER_TARGETS: >
          {
            "targetArray": [
            {
                      "vhost": "example.com",
                      "service": "docker-whale",
                      "path": "/",
                      "scheme": "http",
                      "idp_issuer_base_url": "example.com:8081/realms/master",
                      "idp_client_id": "${ZITI_BROWZER_CLIENT_ID}",
                      "idp_type": "keycloak",
                      "idp_realm": "${KEYCLOAK_REALM}"
            }
            ]
          }

  docker-whale:
    image: crccheck/hello-world
    ports:
      - "2000:8000"

networks:
  ziti:
    driver: bridge
    ipam:
      config:
        - subnet: 192.168.0.0/16
          aux_addresses:
            ziti-controller: 192.168.144.3

volumes:
  ziti-fs:
    driver: local
    driver_opts:
      type: none
      o: bind
      device: ${PWD}/ziti-fs

The error i get:

browzer-1                         | {"level":"info","message":"ziti-browzer-bootstrapper initializing","timestamp":"2024-06-11T09:53:21.506Z","version":"0.61.0"}
browzer-1                         | {"field":"idp_type","level":"warn","message":"obsolete config field encountered - ignored","timestamp":"2024-06-11T09:53:21.509Z","version":"0.61.0"}
browzer-1                         | {"host":"ziti-edge-controller.com","level":"info","message":"contacting specified controller","port":"1280","timestamp":"2024-06-11T09:53:21.512Z","version":"0.61.0"}
browzer-1                         | {"level":"debug","message":"configured target service(s)","targets":{"targetArray":[{"idp_client_id":"","idp_issuer_base_url":"example.com:8081/realms/master","idp_realm":"","idp_type":"keycloak","path":"/","scheme":"http","service":"docker-whale","vhost":"example.com"}]},"timestamp":"2024-06-11T09:53:21.806Z","version":"0.61.0"}
browzer-1                         | {"certificate_path":"/docker-data/example.com.crt","key_path":"/docker-data/example.com.key","level":"info","message":"new tlsContext created","timestamp":"2024-06-11T09:53:21.810Z","version":"0.61.0"}
browzer-1                         | {"level":"info","message":"listening","port":"443","scheme":"https","timestamp":"2024-06-11T09:53:21.815Z","version":"0.61.0"}
ziti-controller-1                 | [   8.902]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:1280]: {remote=[192.168.144.5:57438] error=[EOF]} handshake failed
browzer-1                         | {"code":"SELF_SIGNED_CERT_IN_CHAIN","level":"error","message":"self signed certificate in certificate chain","stack":"Error: self signed certificate in certificate chain\n    at TLSSocket.onConnectSecure (node:_tls_wrap:1539:34)\n    at TLSSocket.emit (node:events:513:28)\n    at TLSSocket._finishInit (node:_tls_wrap:953:8)\n    at TLSWrap.ssl.onhandshakedone (node:_tls_wrap:734:12)","timestamp":"2024-06-11T09:53:21.817Z","version":"0.61.0"}
4

I think we'll want to have @curt confirm this, but that looks to me like the the browzer bootstrapper will need to have the self-signed CA as part of it's trust store. It might be necessary to add it to the os trust but I would start by making a CA bundle of trusted certs and setting this in your compose file:

NODE_EXTRA_CA_CERTS=/some/path/to/extra/certs

If that doesn't work, maybe there's a browzer mechanism to add it to the bootstrapper I'm not aware of, but I think this would work for you.

5

Doesn't my docker compose already have that?

      NODE_EXTRA_CA_CERTS: /persistent/pki/ziti-edge-controller-intermediate/certs/ziti-edge-controller-server.chain.pem

Maybe i am confusing stuff. I also tested by making bundle.pem and adding contents of ziti-edge-controller-server.chain.pem and example.com.cert to it with no luck.

6

This is how i made my example.com.cert:

openssl req -x509 -newkey rsa:4096 -sha256 -days 3650   -nodes -keyout example.com.key -out example.com.crt -subj "/CN=example.com"   -addext "subjectAltName=DNS:example.com,DNS:*.example.com,IP:0.0.0.0"
7

I completely missed that your compose had it! Sorry about that... that chain is unlikely to contain the root CA, it should have everything BUT the root CA in it, I believe. Let me poke at it for a bit and see if I can find the issue.

8

Just to give you an update, I tried to get a workable docker compose file working today and got closer but ended up hitting a myriad of different issues.

With version 0.59.0 we introduced a bug into using NODE_EXTRA_CA_CERTS. If you use 0.58.0 you can get it to work, but you need to override the entrypoint as the current entrypoint script it's not compatible with setting NODE_EXTRA_CA_CERTS via environment variable. I just doubt anyone has tried to make a purely self-signed cert work, they must all be using legit CAs like LetsEncrypt...

Anyway, I've been trying to work through this but didn't end up getting something workable yet. I can share my compose if you want but I'm not sure if it's really useful yet?

9

As a workaround i used NODE_TLS_REJECT_UNAUTHORIZED: 0, but ...

I got another failed to verify certificate: x509: certificate signed by unknown authority, this time from the controller.

Here's my full setup and logs thus far:
docker-compose.yaml:

name: ${PROJ_NAME}
services:
  ziti-controller:
    image: "${ZITI_IMAGE}:${ZITI_VERSION}"
    healthcheck:
      test: curl -m 1 -s -k -f https://${ZITI_CTRL_EDGE_ADVERTISED_ADDRESS:-ziti-edge-controller}:${ZITI_CTRL_EDGE_ADVERTISED_PORT:-1280}/edge/client/v1/version
      interval: 1s
      timeout: 3s
      retries: 30
    env_file:
      - ./.env
    ports:
      - ${ZITI_INTERFACE:-0.0.0.0}:${ZITI_CTRL_EDGE_ADVERTISED_PORT:-1280}:${ZITI_CTRL_EDGE_ADVERTISED_PORT:-1280}
      - ${ZITI_INTERFACE:-0.0.0.0}:${ZITI_CTRL_ADVERTISED_PORT:-6262}:${ZITI_CTRL_ADVERTISED_PORT:-6262}
    environment:
      - ZITI_CTRL_NAME=${ZITI_CTRL_NAME:-ziti-edge-controller}
      - ZITI_CTRL_EDGE_ADVERTISED_ADDRESS=${ZITI_CTRL_EDGE_ADVERTISED_ADDRESS:-ziti-edge-controller}
      - ZITI_CTRL_EDGE_ADVERTISED_PORT=${ZITI_CTRL_EDGE_ADVERTISED_PORT:-1280}
      - ZITI_CTRL_EDGE_IP_OVERRIDE=${ZITI_CTRL_EDGE_IP_OVERRIDE:-127.0.0.1}
      - ZITI_CTRL_ADVERTISED_PORT=${ZITI_CTRL_ADVERTISED_PORT:-6262}
      - ZITI_EDGE_IDENTITY_ENROLLMENT_DURATION=${ZITI_EDGE_IDENTITY_ENROLLMENT_DURATION}
      - ZITI_ROUTER_ENROLLMENT_DURATION=${ZITI_ROUTER_ENROLLMENT_DURATION}
      - ZITI_USER=${ZITI_USER:-admin}
      - ZITI_PWD=${ZITI_PWD}
    networks:
      ziti:
        aliases:
          - ziti-edge-controller
        ipv4_address: 192.168.144.3
    volumes:
      - ziti-fs:/persistent
    entrypoint:
      - "/var/openziti/scripts/run-controller.sh"

  ziti-controller-init-container:
    image: "${ZITI_IMAGE}:${ZITI_VERSION}"
    depends_on:
      ziti-controller:
        condition: service_healthy
    environment:
      - ZITI_CTRL_EDGE_ADVERTISED_ADDRESS=${ZITI_CTRL_EDGE_ADVERTISED_ADDRESS:-ziti-edge-controller}
      - ZITI_CTRL_EDGE_ADVERTISED_PORT=${ZITI_CTRL_EDGE_ADVERTISED_PORT:-1280}
    env_file:
      - ./.env
    networks:
      ziti:
    volumes:
      - ziti-fs:/persistent
    entrypoint:
      - "/var/openziti/scripts/run-with-ziti-cli.sh"
    command:
      - "/var/openziti/scripts/access-control.sh"

  ziti-edge-router:
    image: "${ZITI_IMAGE}:${ZITI_VERSION}"
    env_file:
      - ./.env
    depends_on:
      ziti-controller:
        condition: service_healthy
    ports:
      - ${ZITI_INTERFACE:-0.0.0.0}:${ZITI_ROUTER_PORT:-3022}:${ZITI_ROUTER_PORT:-3022}
      - ${ZITI_INTERFACE:-0.0.0.0}:${ZITI_ROUTER_LISTENER_BIND_PORT:-10080}:${ZITI_ROUTER_LISTENER_BIND_PORT:-10080}
      - 3023:3023
    environment:
      - ZITI_CTRL_ADVERTISED_ADDRESS=${ZITI_CTRL_ADVERTISED_ADDRESS:-ziti-controller}
      - ZITI_CTRL_ADVERTISED_PORT=${ZITI_CTRL_ADVERTISED_PORT:-6262}
      - ZITI_CTRL_EDGE_ADVERTISED_ADDRESS=${ZITI_CTRL_EDGE_ADVERTISED_ADDRESS:-ziti-edge-controller}
      - ZITI_CTRL_EDGE_ADVERTISED_PORT=${ZITI_CTRL_EDGE_ADVERTISED_PORT:-1280}
      - ZITI_ROUTER_NAME=${ZITI_ROUTER_NAME:-ziti-edge-router}
      - ZITI_ROUTER_ADVERTISED_ADDRESS=${ZITI_ROUTER_ADVERTISED_ADDRESS:-ziti-edge-router}
      - ZITI_ROUTER_PORT=${ZITI_ROUTER_PORT:-3022}
      - ZITI_ROUTER_LISTENER_BIND_PORT=${ZITI_ROUTER_LISTENER_BIND_PORT:-10080}
      - ZITI_ROUTER_ROLES=public
    networks:
      - ziti
    volumes:
      - ziti-fs:/persistent
    entrypoint: /bin/bash
    command: "/var/openziti/scripts/run-router.sh edge"

  ziti-console:
    image: openziti/zac
    working_dir: /usr/src/app
    environment:
      - ZAC_SERVER_CERT_CHAIN=/persistent/pki/${ZITI_CTRL_EDGE_ADVERTISED_ADDRESS:-ziti-edge-controller}-intermediate/certs/${ZITI_CTRL_EDGE_ADVERTISED_ADDRESS:-ziti-edge-controller}-server.cert
      - ZAC_SERVER_KEY=/persistent/pki/${ZITI_CTRL_EDGE_ADVERTISED_ADDRESS:-ziti-edge-controller}-intermediate/keys/${ZITI_CTRL_EDGE_ADVERTISED_ADDRESS:-ziti-edge-controller}-server.key
      - ZITI_CTRL_EDGE_ADVERTISED_ADDRESS=${ZITI_CTRL_EDGE_ADVERTISED_ADDRESS:-ziti-edge-controller}
      - ZITI_CTRL_EDGE_ADVERTISED_PORT=${ZITI_CTRL_EDGE_ADVERTISED_PORT:-1280}
      - ZITI_CTRL_NAME=${ZITI_CTRL_NAME:-ziti-edge-controller}
      - PORTTLS=8443
    depends_on:
      ziti-controller:
        condition: service_healthy
    ports:
      - ${ZITI_INTERFACE:-0.0.0.0}:8443:8443
    volumes:
      - ziti-fs:/persistent
    networks:
      - ziti

  keycloak:
      image: quay.io/keycloak/keycloak:25.0.0
      command: 
      - "start-dev"
      - "--https-certificate-file=${KEYCLOAK_CERT}"
      - "--https-certificate-key-file=${KEYCLOAK_KEY}"
      - "--https-port=${KEYCLOAK_PORT}"
      ports:
        - ${KEYCLOAK_PORT}:${KEYCLOAK_PORT}
      environment:
        - KEYCLOAK_ADMIN=${KEYCLOAK_ADMIN_USER}
        - KEYCLOAK_ADMIN_PASSWORD=${KEYCLOAK_ADMIN_PWD}
      networks:
        ziti:
          ipv4_address: 192.168.144.2

      volumes:
        - ./docker-data:/docker-data

  browzer:
    image: ghcr.io/openziti/ziti-browzer-bootstrapper:latest
    restart: always
    depends_on:
      ziti-controller:
        condition: service_healthy
    networks:
      - ziti

    volumes:
      - ./docker-data:/docker-data
      - ziti-fs:/persistent

    ports:
      - "${ZITI_BROWZER_PORT}:${ZITI_BROWZER_PORT}"

    # entrypoint: ['/docker-data/start.sh', "/home/node/ziti-browzer-bootstrapper/zha-docker-entrypoint"]

    environment:
      # NODE_EXTRA_CA_CERTS: /persistent/pki/ziti-signing-root-ca/certs/ziti-signing-intermediate_grandparent_intermediate.cert
      NODE_TLS_REJECT_UNAUTHORIZED: 0
      NODE_ENV: production
      ZITI_BROWZER_RUNTIME_ORIGIN_TRIAL_TOKEN: ${BROWZER_ORIGINAL_TRIAL_TOKEN}
      ZITI_BROWZER_BOOTSTRAPPER_LOGLEVEL: debug
      ZITI_BROWZER_RUNTIME_LOGLEVEL: debug
      ZITI_CONTROLLER_HOST: ${ZITI_BROWZER_CONTROLLER_HOST}
      ZITI_CONTROLLER_PORT: 1280
      ZITI_BROWZER_BOOTSTRAPPER_HOST: example.com
      ZITI_BROWZER_BOOTSTRAPPER_LISTEN_PORT: ${ZITI_BROWZER_PORT}
      ZITI_BROWZER_BOOTSTRAPPER_CERTIFICATE_PATH: /docker-data/example.com.crt
      ZITI_BROWZER_BOOTSTRAPPER_KEY_PATH: /docker-data/example.com.key
      ZITI_BROWZER_BOOTSTRAPPER_SCHEME: https
      ZITI_BROWZER_BOOTSTRAPPER_TARGETS: >
          {
            "targetArray": [
            {
              "vhost": "${ZITI_BROWZER_VHOST}",
              "service": "${ZITI_BROWZER_SERVICE}",
              "path": "/",
              "scheme": "http",
              "idp_issuer_base_url": "${KEYCLOAK_HOST_AND_PORT}realms/zitirealm",
              "idp_client_id": "${ZITI_BROWZER_CLIENT_ID}",
              "idp_type": "keycloak",
              "idp_realm": "${KEYCLOAK_REALM}"
            }
            ]
          }

  docker-whale:
    image: crccheck/hello-world
    ports:
      - "2000:8000"

networks:
  ziti:
    driver: bridge
    ipam:
      config:
        - subnet: 192.168.0.0/16

volumes:
  ziti-fs:
    driver: local
    driver_opts:
      type: none
      o: bind
      device: ${PWD}/${ZITI_VOLUME_DIR}

.env:

export PROJ_NAME="openziti"
export WILDCARD_DNS="example.com"
export COMPOSE_FILE="./docker-compose.yaml"

# OpenZiti Variables
export ZITI_IMAGE=openziti/quickstart
export ZITI_VERSION=latest
export ZITI_VOLUME_DIR="./ziti-fs"

# the user and password to use
# Leave password blank to have a unique value generated or set the password explicitly
export ZITI_USER=admin
export ZITI_PWD=rootroot
 
export ZITI_INTERFACE=0.0.0.0

# controller name, address/port information
export ZITI_CTRL_NAME=ziti-controller
export ZITI_CTRL_EDGE_ADVERTISED_ADDRESS=ziti-edge-controller
export ZITI_CTRL_ADVERTISED_ADDRESS=ziti-controller
#ZITI_CTRL_EDGE_IP_OVERRIDE=10.10.10.10
#ZITI_CTRL_EDGE_ADVERTISED_PORT=8441
#ZITI_CTRL_ADVERTISED_PORT=8440

# The duration of the enrollment period (in minutes), default if not set. shown - 7days
export ZITI_EDGE_IDENTITY_ENROLLMENT_DURATION=10080
export ZITI_ROUTER_ENROLLMENT_DURATION=10080


# router address/port information
#ZITI_ROUTER_NAME=ziti-edge-router
#ZITI_ROUTER_ADVERTISED_ADDRESS=ziti-edge-router
#ZITI_ROUTER_PORT=8442
#ZITI_ROUTER_IP_OVERRIDE=10.10.10.10
#ZITI_ROUTER_LISTENER_BIND_PORT=8444
#ZITI_ROUTER_ROLES=public


# keycloak
export KEYCLOAK_ADMIN_USER=admin
export KEYCLOAK_ADMIN_PWD=rootroot
export KEYCLOAK_REALM=zitirealm
export KEYCLOAK_PORT=8081
export KEYCLOAK_BASE="keycloak.example.com"
export KEYCLOAK_CERT=/docker-data/example.com.crt
export KEYCLOAK_KEY=/docker-data/example.com.key
export KEYCLOAK_NOSSL_URL="http://0.0.0.0:8080"

# browzer
export ZITI_BROWZER_CONTROLLER_HOST=ziti-edge-controller.com
export ZITI_BROWZER_PORT=443
export ZITI_BROWZER_WSS_ER_PORT=8505
export ZITI_BROWZER_HTTP_AGENT_URL="browzer.${WILDCARD_DNS}"
export KEYCLOAK_HOST_AND_PORT="https://${KEYCLOAK_BASE}:${KEYCLOAK_PORT}/"
export ZITI_BROWZER_OIDC_URL="${KEYCLOAK_HOST_AND_PORT}realms/${KEYCLOAK_REALM}"
export ZITI_BROWZER_CLIENT_ID="browzerBootstrapClient"
export ZITI_BROWZER_VHOST="docker-whale.example.com"
export ZITI_BROWZER_SERVICE="docker.whale"
export ZITI_BROWZER_WSS_ER_HOST="wss.${WILDCARD_DNS}"
export ZITI_BROWZER_IDENTITIES="carlosdomhleba@gmail.com"
export ZITI_BROWZER_DOCKER_PROJECT="browzer-compose"
export BROWZER_ORIGINAL_TRIAL_TOKEN="my_token_here"

After i do docker compose up i use install.sh file which i run.
Here's install.sh:

#!/bin/bash

source .env

./keycloak-configure.sh

./browzer-configure.sh

keycloak-configure.sh:

function kcadm {
  docker compose \
    -f ${COMPOSE_FILE}.yml \
    --project-name ${PROJ_NAME} \
    exec -it keycloak \
    /opt/keycloak/bin/kcadm.sh $@;
}

kcadm config credentials \
  --server ${KEYCLOAK_NOSSL_URL} \
  --realm master \
  --user ${KEYCLOAK_ADMIN_USER} \
  --password ${KEYCLOAK_ADMIN_PWD}

kcadm create realms \
  -s realm=${KEYCLOAK_REALM} \
  -s enabled=true

kcadm create clients \
  -r ${KEYCLOAK_REALM} \
  -s clientId=${ZITI_BROWZER_CLIENT_ID} \
  -s protocol=openid-connect \
  -s 'redirectUris=["https://'${ZITI_BROWZER_VHOST}'/*","https://brozac.'${WILDCARD_DNS}'/*"]' \
  -s 'webOrigins=["https://'${ZITI_BROWZER_VHOST}'","https://brozac.'${WILDCARD_DNS}'"]' \
  -s 'directAccessGrantsEnabled=true'

CLIENT_SCOPE_ID=$(kcadm get clients -r ${KEYCLOAK_REALM} | jq -r '.[] | select(.clientId == "'${ZITI_BROWZER_CLIENT_ID}'") | .id')
kcadm update realms/${KEYCLOAK_REALM}/clients/${CLIENT_SCOPE_ID} --set fullScopeAllowed=false

kcadm create clients/${CLIENT_SCOPE_ID}/protocol-mappers/models \
  -r ${KEYCLOAK_REALM} \
  -s name=audience-mapping \
  -s protocol=openid-connect \
  -s protocolMapper=oidc-audience-mapper \
  -s config.\"included.custom.audience\"="${ZITI_BROWZER_CLIENT_ID}" \
  -s config.\"access.token.claim\"=\"true\" \
  -s config.\"id.token.claim\"=\"false\"

browzer-configure.sh:

if grep -q "wss:" $ZITI_VOLUME_DIR/ziti-edge-router.yaml; then 
  echo "router config file appears to have web socket listener already: $ZITI_VOLUME_DIR/ziti-edge-router.yaml"
else
  echo "adding/replacing settings in the quickstart router. adding web socket listener, configuring ws block"
  sed -i 's#tproxy|host#tproxy|host\n  - binding: edge\n    address: wss:0.0.0.0:'${ZITI_BROWZER_WSS_ER_PORT}'\n    options:\n      advertise: '${ZITI_BROWZER_WSS_ER_HOST}':'${ZITI_BROWZER_WSS_ER_PORT}'\n      connectTimeoutMs: 5000\n      getSessionTimeout: 60#g' $ZITI_VOLUME_DIR/ziti-edge-router.yaml
  sed -i 's`#transport\:`transport\:`g' $ZITI_VOLUME_DIR/ziti-edge-router.yaml
  sed -i 's`#  ws\:`  ws\:`g' $ZITI_VOLUME_DIR/ziti-edge-router.yaml
  sed -i 's`#    `    `g' $ZITI_VOLUME_DIR/ziti-edge-router.yaml

  echo "Required to restart edge router"
fi

function zitiEx {
  docker compose \
    -f ${COMPOSE_FILE}.yml \
    --project-name ${PROJ_NAME} \
    exec -it ziti-controller \
    /var/openziti/ziti-bin/ziti "$@";
}

zitiEx edge login -u $ZITI_USER -p $ZITI_PWD

ziti_object_prefix=browzer-keycloak
issuer=$(curl -k ${ZITI_BROWZER_OIDC_URL}/.well-known/openid-configuration | jq -r .issuer)
jwks=$(curl -k ${ZITI_BROWZER_OIDC_URL}/.well-known/openid-configuration | jq -r .jwks_uri)

echo "OIDC issuer   : $issuer"
echo "OIDC jwks url : $jwks"

ext_jwt_signer=$(zitiEx edge create ext-jwt-signer "${ziti_object_prefix}-ext-jwt-signer" "${issuer}" --jwks-endpoint "${jwks}" --audience "${ZITI_BROWZER_CLIENT_ID}" --claims-property email)
echo "ext jwt signer id: $ext_jwt_signer"

auth_policy=$(zitiEx edge create auth-policy ${ziti_object_prefix}-auth-policy --primary-ext-jwt-allowed --primary-ext-jwt-allowed-signers ${ext_jwt_signer} --secondary-req-ext-jwt-signer ${ext_jwt_signer})
echo "auth policy id: $auth_policy"

echo "creating users specified by ZITI_BROWZER_IDENTITIES: ${ZITI_BROWZER_IDENTITIES}"
for id in ${ZITI_BROWZER_IDENTITIES}; do
  zitiEx edge create identity user "${id}" --auth-policy ${auth_policy} --external-id "${id}" -a docker.whale.dialers,brozac.dialers
done


echo "adding router ziti-edge-router as docker.whale.binders"
zitiEx edge update identity "ziti-edge-router" -a docker.whale.binders,brozac.binders,brozac.binders

source ./docker.whale
createService

# source $SCRIPT_DIR/brozac
# createService

Heres controllers logs:

ziti-controller-1                 | [  65.266]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:1280]: {error=[EOF] remote=[192.168.0.1:57314]} handshake failed
ziti-controller-1                 | [  65.299]    INFO ziti/controller/env.(*AppEnv).GetControllerPublicKey: looking for signer: tj28pVRBNLkyZcDjzxf29ZCSkXIc4gRzdIrrjCfxsdo
ziti-controller-1                 | [  65.299]   ERROR ziti/controller/env.(*AppEnv).getJwtTokenFromRequest: {error=[token is unverifiable: error while executing keyfunc: key for kid tj28pVRBNLkyZcDjzxf29ZCSkXIc4gRzdIrrjCfxsdo, not found]} error during JWT parsing during API request
ziti-controller-1                 | [  65.613]    INFO ziti/controller/env.(*AppEnv).GetControllerPublicKey: looking for signer: tj28pVRBNLkyZcDjzxf29ZCSkXIc4gRzdIrrjCfxsdo
ziti-controller-1                 | [  65.613]   ERROR ziti/controller/env.(*AppEnv).getJwtTokenFromRequest: {error=[token is unverifiable: error while executing keyfunc: key for kid tj28pVRBNLkyZcDjzxf29ZCSkXIc4gRzdIrrjCfxsdo, not found]} error during JWT parsing during API request
ziti-controller-1                 | [  65.637]   ERROR ziti/controller/model.(*AuthModuleExtJwt).pubKeyLookup: {extJwtSignerName=[browzer-keycloak-ext-jwt-signer] error=[could not resolve jwks endpoint: Get "https://keycloak.example.com:8081/realms/zitirealm/protocol/openid-connect/certs": tls: failed to verify certificate: x509: certificate signed by unknown authority] kid=[tj28pVRBNLkyZcDjzxf29ZCSkXIc4gRzdIrrjCfxsdo] issuer=[https://keycloak.example.com:8081/realms/zitirealm] extJwtSignerId=[2Nqw7v8iAUgRQlnx1SfQxp] method=[ext-jwt]} error attempting to resolve extJwtSigner certificate used for signing
ziti-controller-1                 | [  65.637]   ERROR ziti/controller/model.(*AuthModuleExtJwt).process: {authMethod=[ext-jwt]} authorization failed, jwt did not verify
ziti-controller-1                 | [  65.643]    INFO ziti/controller/env.(*AppEnv).GetControllerPublicKey: looking for signer: tj28pVRBNLkyZcDjzxf29ZCSkXIc4gRzdIrrjCfxsdo
ziti-controller-1                 | [  65.643]   ERROR ziti/controller/env.(*AppEnv).getJwtTokenFromRequest: {error=[token is unverifiable: error while executing keyfunc: key for kid tj28pVRBNLkyZcDjzxf29ZCSkXIc4gRzdIrrjCfxsdo, not found]} error during JWT parsing during API request
browzer-1                         | {"error":"User [carlosdomhleba@gmail.com] cannot be authenticated onto Ziti Network","error_code":1001,"level":"error","message":"Check that this 'externalId' exists and has case-sensitive match","timestamp":"2024-06-12T11:18:16.432Z","version":"0.61.0"}
ziti-controller-1                 | [  65.709]    INFO ziti/controller/env.(*AppEnv).GetControllerPublicKey: looking for signer: tj28pVRBNLkyZcDjzxf29ZCSkXIc4gRzdIrrjCfxsdo
ziti-controller-1                 | [  65.709]   ERROR ziti/controller/env.(*AppEnv).getJwtTokenFromRequest: {error=[token is unverifiable: error while executing keyfunc: key for kid tj28pVRBNLkyZcDjzxf29ZCSkXIc4gRzdIrrjCfxsdo, not found]} error during JWT parsing during API request
ziti-controller-1                 | [  66.210]    INFO ziti/controller/env.(*AppEnv).GetControllerPublicKey: looking for signer: tj28pVRBNLkyZcDjzxf29ZCSkXIc4gRzdIrrjCfxsdo
ziti-controller-1                 | [  66.210]   ERROR ziti/controller/env.(*AppEnv).getJwtTokenFromRequest: {error=[token is unverifiable: error while executing keyfunc: key for kid tj28pVRBNLkyZcDjzxf29ZCSkXIc4gRzdIrrjCfxsdo, not found]} error during JWT parsing during API request
ziti-controller-1                 | [  66.210]   ERROR ziti/controller/model.(*AuthModuleExtJwt).process: {authMethod=[ext-jwt]} authorization failed, jwt did not verify
ziti-controller-1                 | [  66.215]    INFO ziti/controller/env.(*AppEnv).GetControllerPublicKey: looking for signer: tj28pVRBNLkyZcDjzxf29ZCSkXIc4gRzdIrrjCfxsdo
ziti-controller-1                 | [  66.215]   ERROR ziti/controller/env.(*AppEnv).getJwtTokenFromRequest: {error=[token is unverifiable: error while executing keyfunc: key for kid tj28pVRBNLkyZcDjzxf29ZCSkXIc4gRzdIrrjCfxsdo, not found]} error during JWT parsing during API request
ziti-controller-1                 | [  66.216]   ERROR ziti/controller/model.(*AuthModuleExtJwt).process: {authMethod=[ext-jwt]} authorization failed, jwt did not verify
ziti-controller-1                 | [  66.219]    INFO ziti/controller/env.(*AppEnv).GetControllerPublicKey: looking for signer: tj28pVRBNLkyZcDjzxf29ZCSkXIc4gRzdIrrjCfxsdo
ziti-controller-1                 | [  66.219]   ERROR ziti/controller/env.(*AppEnv).getJwtTokenFromRequest: {error=[token is unverifiable: error while executing keyfunc: key for kid tj28pVRBNLkyZcDjzxf29ZCSkXIc4gRzdIrrjCfxsdo, not found]} error during JWT parsing during API request
ziti-controller-1                 | [  66.729]    INFO ziti/controller/env.(*AppEnv).GetControllerPublicKey: looking for signer: tj28pVRBNLkyZcDjzxf29ZCSkXIc4gRzdIrrjCfxsdo
ziti-controller-1                 | [  66.729]   ERROR ziti/controller/env.(*AppEnv).getJwtTokenFromRequest: {error=[token is unverifiable: error while executing keyfunc: key for kid tj28pVRBNLkyZcDjzxf29ZCSkXIc4gRzdIrrjCfxsdo, not found]} error during JWT parsing during API request
ziti-controller-1                 | [  66.730]   ERROR ziti/controller/model.(*AuthModuleExtJwt).process: {authMethod=[ext-jwt]} authorization failed, jwt did not verify
ziti-controller-1                 | [  66.734]    INFO ziti/controller/env.(*AppEnv).GetControllerPublicKey: looking for signer: tj28pVRBNLkyZcDjzxf29ZCSkXIc4gRzdIrrjCfxsdo
ziti-controller-1                 | [  66.734]   ERROR ziti/controller/env.(*AppEnv).getJwtTokenFromRequest: {error=[token is unverifiable: error while executing keyfunc: key for kid tj28pVRBNLkyZcDjzxf29ZCSkXIc4gRzdIrrjCfxsdo, not found]} error during JWT parsing during API request
ziti-controller-1                 | [  66.734]   ERROR ziti/controller/model.(*AuthModuleExtJwt).process: {authMethod=[ext-jwt]} authorization failed, jwt did not verify
ziti-controller-1                 | [  66.738]    INFO ziti/controller/env.(*AppEnv).GetControllerPublicKey: looking for signer: tj28pVRBNLkyZcDjzxf29ZCSkXIc4gRzdIrrjCfxsdo
ziti-controller-1                 | [  66.738]   ERROR ziti/controller/env.(*AppEnv).getJwtTokenFromRequest: {error=[token is unverifiable: error while executing keyfunc: key for kid tj28pVRBNLkyZcDjzxf29ZCSkXIc4gRzdIrrjCfxsdo, not found]} error during JWT parsing during API request
browzer-1                         | {"error":"User [carlosdomhleba@gmail.com] cannot be authenticated onto Ziti Network","error_code":1001,"level":"error","message":"Check that this 'externalId' exists and has case-sensitive match","timestamp":"2024-06-12T11:18:21.932Z","version":"0.61.0"}
ziti-controller-1                 | [  71.663]    INFO ziti/controller/env.(*AppEnv).GetControllerPublicKey: looking for signer: tj28pVRBNLkyZcDjzxf29ZCSkXIc4gRzdIrrjCfxsdo
ziti-controller-1                 | [  71.663]   ERROR ziti/controller/env.(*AppEnv).getJwtTokenFromRequest: {error=[token is unverifiable: error while executing keyfunc: key for kid tj28pVRBNLkyZcDjzxf29ZCSkXIc4gRzdIrrjCfxsdo, not found]} error during JWT parsing during API request
ziti-controller-1                 | [  71.691]   ERROR ziti/controller/model.(*AuthModuleExtJwt).pubKeyLookup: {extJwtSignerId=[2Nqw7v8iAUgRQlnx1SfQxp] extJwtSignerName=[browzer-keycloak-ext-jwt-signer] method=[ext-jwt] error=[could not resolve jwks endpoint: Get "https://keycloak.example.com:8081/realms/zitirealm/protocol/openid-connect/certs": tls: failed to verify certificate: x509: certificate signed by unknown authority] kid=[tj28pVRBNLkyZcDjzxf29ZCSkXIc4gRzdIrrjCfxsdo] issuer=[https://keycloak.example.com:8081/realms/zitirealm]} error attempting to resolve extJwtSigner certificate used for signing
ziti-controller-1                 | [  71.691]   ERROR ziti/controller/model.(*AuthModuleExtJwt).process: {authMethod=[ext-jwt]} authorization failed, jwt did not verify
ziti-controller-1                 | [  71.695]    INFO ziti/controller/env.(*AppEnv).GetControllerPublicKey: looking for signer: tj28pVRBNLkyZcDjzxf29ZCSkXIc4gRzdIrrjCfxsdo
ziti-controller-1                 | [  71.695]   ERROR ziti/controller/env.(*AppEnv).getJwtTokenFromRequest: {error=[token is unverifiable: error while executing keyfunc: key for kid tj28pVRBNLkyZcDjzxf29ZCSkXIc4gRzdIrrjCfxsdo, not found]} error during JWT parsing during API request
ziti-controller-1                 | [  71.696]   ERROR ziti/controller/model.(*AuthModuleExtJwt).process: {authMethod=[ext-jwt]} authorization failed, jwt did not verify
ziti-controller-1                 | [  71.700]    INFO ziti/controller/env.(*AppEnv).GetControllerPublicKey: looking for signer: tj28pVRBNLkyZcDjzxf29ZCSkXIc4gRzdIrrjCfxsdo
ziti-controller-1                 | [  71.700]   ERROR ziti/controller/env.(*AppEnv).getJwtTokenFromRequest: {error=[token is unverifiable: error while executing keyfunc: key for kid tj28pVRBNLkyZcDjzxf29ZCSkXIc4gRzdIrrjCfxsdo, not found]} error during JWT parsing during API request

Browser console:

GET https://docker-whale.example.com/browzer_error?browzer_error_data==%7B%22status%22:511,%22code%22:1001,%22title%22:%22User%20%5Bcarlosdomhleba@gmail.com%5D%20cannot%20be%20authenticated%20onto%20Ziti%20Network%22,%22message%22:%22Check%20that%20this%20%27externalId%27%20exists%20and%20has%20case-sensitive%20match%22,%22myvar%22:%7B%22type%22:%22zbr%22%7D%7D 511 (Network Authentication Required)
12

Hi @CarlosHleb. Discourse marked your posts as spam, sorry about that... I've added the one that had the most information back to this thread...

Thanks for providing your steps. They're incredibly similar to what I was doing, too. I am trying to get a workable answer/solution for you. I'll keep poking at it...

13

Ok @CarlosHleb -- here we go. I got it all to work together. I put it all up on my github repo here.

The README.md should take you through it all and I think between what you have and what's on that github you should be able to get it all running.

Here's the gist from that readme:

To bring up this environment do the following:

  • clone this repo
  • cd to browzer-docker-compose
  • mkdir 'docker-data'
  • cp env-template .env
  • open the .env file and edit all the relevant settings
  • after editing the .env file, source it: source .env
  • generate the self-signed root CA and wildcard server keypair by running ./initialize-pki.sh
  • add the root ca to the appropriate trust store
  • bring up the compose file: docker compose up (use -d if you want)
  • configure keycloak by running: ./keycloak-configure.sh
  • configure the openziti overlay by running: ./openziti-configure.sh
  • OPTIONALLY: configure GitHub/Google (if using federated auth)

I didn't record a video showing everything in action, but if you need it I can. Have a look. Two key things is using the 'alt' server certs and also using a self-signed CA to make a wildcard server cert...

1 Like
14

Oh one more thing, I use browzer 0.58.0 and overrode the entrypoint/command. That's also relevant. I'm going to try to make a new container that works with NODE_EXTRA_CA_CERTS next so we can get on a newer version of browzer (0.58.0 is not 'old', but it's not latest)

15

I have a PR up that will allow NODE_EXTRA_CA_CERTS to be overridden. I'd expect this fix to roll out in 0.62 of browzer. When it rolls out, I'll reply here and update the example compose file with :latest and ensure it all still works. Cheers

16

@CarlosHleb, version 0.61.1 of the browzer bootstrapper was released and worked for me so you can update the compose file to :latest and it should work fine.

cheers

17

I'm covering this live on Ziti TV this week. If you're interested in seeing it live or watching the replay, it'll be here: