Hello,
Watching this video: https://www.youtube.com/watch?v=ti1w7dQ3gSY&ab_channel=OpenZiti
Would it work to combine the the docker compose simplified quickstart with browzer and keycloack using self-signed certs only?
Thanks!
Hi @CarlosHleb, welcome to the community and to OpenZiti and Browzer!
You could get it working with self-signed certs only, sure, but it's going to force you to add the self-signed CA into any and all clients that use Browzer...
The reason the WSS-enabled edge router needs an independently verifiable certificate is so that "any" browser would work. If you want to have your users install the self-signed CA, then then things should work fine.
Hopefully that makes sense and makes it clear why it's needed?
1 Like
Hello,
I am having issues with adding the self signed controllers certificats to browzer.
Here's my docker-compose:
services:
ziti-controller:
image: "${ZITI_IMAGE}:${ZITI_VERSION}"
healthcheck:
test: curl -m 1 -s -k -f https://${ZITI_CTRL_EDGE_ADVERTISED_ADDRESS:-ziti-edge-controller}:${ZITI_CTRL_EDGE_ADVERTISED_PORT:-1280}/edge/client/v1/version
interval: 1s
timeout: 3s
retries: 30
env_file:
- ./.env
ports:
- ${ZITI_INTERFACE:-0.0.0.0}:${ZITI_CTRL_EDGE_ADVERTISED_PORT:-1280}:${ZITI_CTRL_EDGE_ADVERTISED_PORT:-1280}
- ${ZITI_INTERFACE:-0.0.0.0}:${ZITI_CTRL_ADVERTISED_PORT:-6262}:${ZITI_CTRL_ADVERTISED_PORT:-6262}
environment:
- ZITI_CTRL_NAME=${ZITI_CTRL_NAME:-ziti-edge-controller}
- ZITI_CTRL_EDGE_ADVERTISED_ADDRESS=${ZITI_CTRL_EDGE_ADVERTISED_ADDRESS:-ziti-edge-controller}
- ZITI_CTRL_EDGE_ADVERTISED_PORT=${ZITI_CTRL_EDGE_ADVERTISED_PORT:-1280}
- ZITI_CTRL_EDGE_IP_OVERRIDE=${ZITI_CTRL_EDGE_IP_OVERRIDE:-127.0.0.1}
- ZITI_CTRL_ADVERTISED_PORT=${ZITI_CTRL_ADVERTISED_PORT:-6262}
- ZITI_EDGE_IDENTITY_ENROLLMENT_DURATION=${ZITI_EDGE_IDENTITY_ENROLLMENT_DURATION}
- ZITI_ROUTER_ENROLLMENT_DURATION=${ZITI_ROUTER_ENROLLMENT_DURATION}
- ZITI_USER=${ZITI_USER:-admin}
- ZITI_PWD=${ZITI_PWD}
networks:
ziti:
aliases:
- ziti-edge-controller
volumes:
- ziti-fs:/persistent
entrypoint:
- "/var/openziti/scripts/run-controller.sh"
ziti-controller-init-container:
image: "${ZITI_IMAGE}:${ZITI_VERSION}"
depends_on:
ziti-controller:
condition: service_healthy
environment:
- ZITI_CTRL_EDGE_ADVERTISED_ADDRESS=${ZITI_CTRL_EDGE_ADVERTISED_ADDRESS:-ziti-edge-controller}
- ZITI_CTRL_EDGE_ADVERTISED_PORT=${ZITI_CTRL_EDGE_ADVERTISED_PORT:-1280}
env_file:
- ./.env
networks:
ziti:
volumes:
- ziti-fs:/persistent
entrypoint:
- "/var/openziti/scripts/run-with-ziti-cli.sh"
command:
- "/var/openziti/scripts/access-control.sh"
ziti-edge-router:
image: "${ZITI_IMAGE}:${ZITI_VERSION}"
env_file:
- ./.env
depends_on:
ziti-controller:
condition: service_healthy
ports:
- ${ZITI_INTERFACE:-0.0.0.0}:${ZITI_ROUTER_PORT:-3022}:${ZITI_ROUTER_PORT:-3022}
- ${ZITI_INTERFACE:-0.0.0.0}:${ZITI_ROUTER_LISTENER_BIND_PORT:-10080}:${ZITI_ROUTER_LISTENER_BIND_PORT:-10080}
environment:
- ZITI_CTRL_ADVERTISED_ADDRESS=${ZITI_CTRL_ADVERTISED_ADDRESS:-ziti-controller}
- ZITI_CTRL_ADVERTISED_PORT=${ZITI_CTRL_ADVERTISED_PORT:-6262}
- ZITI_CTRL_EDGE_ADVERTISED_ADDRESS=${ZITI_CTRL_EDGE_ADVERTISED_ADDRESS:-ziti-edge-controller}
- ZITI_CTRL_EDGE_ADVERTISED_PORT=${ZITI_CTRL_EDGE_ADVERTISED_PORT:-1280}
- ZITI_ROUTER_NAME=${ZITI_ROUTER_NAME:-ziti-edge-router}
- ZITI_ROUTER_ADVERTISED_ADDRESS=${ZITI_ROUTER_ADVERTISED_ADDRESS:-ziti-edge-router}
- ZITI_ROUTER_PORT=${ZITI_ROUTER_PORT:-3022}
- ZITI_ROUTER_LISTENER_BIND_PORT=${ZITI_ROUTER_LISTENER_BIND_PORT:-10080}
- ZITI_ROUTER_ROLES=public
networks:
- ziti
volumes:
- ziti-fs:/persistent
entrypoint: /bin/bash
command: "/var/openziti/scripts/run-router.sh edge"
ziti-console:
image: openziti/zac
working_dir: /usr/src/app
environment:
- ZAC_SERVER_CERT_CHAIN=/persistent/pki/${ZITI_CTRL_EDGE_ADVERTISED_ADDRESS:-ziti-edge-controller}-intermediate/certs/${ZITI_CTRL_EDGE_ADVERTISED_ADDRESS:-ziti-edge-controller}-server.cert
- ZAC_SERVER_KEY=/persistent/pki/${ZITI_CTRL_EDGE_ADVERTISED_ADDRESS:-ziti-edge-controller}-intermediate/keys/${ZITI_CTRL_EDGE_ADVERTISED_ADDRESS:-ziti-edge-controller}-server.key
- ZITI_CTRL_EDGE_ADVERTISED_ADDRESS=${ZITI_CTRL_EDGE_ADVERTISED_ADDRESS:-ziti-edge-controller}
- ZITI_CTRL_EDGE_ADVERTISED_PORT=${ZITI_CTRL_EDGE_ADVERTISED_PORT:-1280}
- ZITI_CTRL_NAME=${ZITI_CTRL_NAME:-ziti-edge-controller}
- PORTTLS=8443
depends_on:
ziti-controller:
condition: service_healthy
ports:
- ${ZITI_INTERFACE:-0.0.0.0}:8443:8443
volumes:
- ziti-fs:/persistent
networks:
- ziti
keycloack:
image: quay.io/keycloak/keycloak:25.0.0
command:
- "start-dev"
- "--https-certificate-file=/docker-data/example.com.crt"
- "--https-certificate-key-file=/docker-data/example.com.key"
- "--https-port=8081"
ports:
- 8081:8081
environment:
- KEYCLOAK_ADMIN=admin
- KEYCLOAK_ADMIN_PASSWORD=rootroot
networks:
- ziti
volumes:
- ./docker-data:/docker-data
browzer:
image: ghcr.io/openziti/ziti-browzer-bootstrapper:latest
restart: always
depends_on:
ziti-controller:
condition: service_healthy
networks:
- ziti
volumes:
- ./docker-data:/docker-data
- ./ziti-fs:/persistent
ports:
- "443:443"
environment:
NODE_EXTRA_CA_CERTS: /persistent/pki/ziti-edge-controller-intermediate/certs/ziti-edge-controller-server.chain.pem
# NODE_TLS_REJECT_UNAUTHORIZED: 0
NODE_ENV: production
ZITI_BROWZER_BOOTSTRAPPER_LOGLEVEL: debug
ZITI_BROWZER_RUNTIME_LOGLEVEL: debug
ZITI_CONTROLLER_HOST: ziti-edge-controller.com
ZITI_CONTROLLER_PORT: 1280
ZITI_BROWZER_BOOTSTRAPPER_HOST: example.com
ZITI_BROWZER_BOOTSTRAPPER_LISTEN_PORT: 443
ZITI_BROWZER_BOOTSTRAPPER_CERTIFICATE_PATH: /docker-data/example.com.crt
ZITI_BROWZER_BOOTSTRAPPER_KEY_PATH: /docker-data/example.com.key
ZITI_BROWZER_BOOTSTRAPPER_SCHEME: https
ZITI_BROWZER_BOOTSTRAPPER_TARGETS: >
{
"targetArray": [
{
"vhost": "example.com",
"service": "docker-whale",
"path": "/",
"scheme": "http",
"idp_issuer_base_url": "example.com:8081/realms/master",
"idp_client_id": "${ZITI_BROWZER_CLIENT_ID}",
"idp_type": "keycloak",
"idp_realm": "${KEYCLOAK_REALM}"
}
]
}
docker-whale:
image: crccheck/hello-world
ports:
- "2000:8000"
networks:
ziti:
driver: bridge
ipam:
config:
- subnet: 192.168.0.0/16
aux_addresses:
ziti-controller: 192.168.144.3
volumes:
ziti-fs:
driver: local
driver_opts:
type: none
o: bind
device: ${PWD}/ziti-fs
The error i get:
browzer-1 | {"level":"info","message":"ziti-browzer-bootstrapper initializing","timestamp":"2024-06-11T09:53:21.506Z","version":"0.61.0"}
browzer-1 | {"field":"idp_type","level":"warn","message":"obsolete config field encountered - ignored","timestamp":"2024-06-11T09:53:21.509Z","version":"0.61.0"}
browzer-1 | {"host":"ziti-edge-controller.com","level":"info","message":"contacting specified controller","port":"1280","timestamp":"2024-06-11T09:53:21.512Z","version":"0.61.0"}
browzer-1 | {"level":"debug","message":"configured target service(s)","targets":{"targetArray":[{"idp_client_id":"","idp_issuer_base_url":"example.com:8081/realms/master","idp_realm":"","idp_type":"keycloak","path":"/","scheme":"http","service":"docker-whale","vhost":"example.com"}]},"timestamp":"2024-06-11T09:53:21.806Z","version":"0.61.0"}
browzer-1 | {"certificate_path":"/docker-data/example.com.crt","key_path":"/docker-data/example.com.key","level":"info","message":"new tlsContext created","timestamp":"2024-06-11T09:53:21.810Z","version":"0.61.0"}
browzer-1 | {"level":"info","message":"listening","port":"443","scheme":"https","timestamp":"2024-06-11T09:53:21.815Z","version":"0.61.0"}
ziti-controller-1 | [ 8.902] ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:1280]: {remote=[192.168.144.5:57438] error=[EOF]} handshake failed
browzer-1 | {"code":"SELF_SIGNED_CERT_IN_CHAIN","level":"error","message":"self signed certificate in certificate chain","stack":"Error: self signed certificate in certificate chain\n at TLSSocket.onConnectSecure (node:_tls_wrap:1539:34)\n at TLSSocket.emit (node:events:513:28)\n at TLSSocket._finishInit (node:_tls_wrap:953:8)\n at TLSWrap.ssl.onhandshakedone (node:_tls_wrap:734:12)","timestamp":"2024-06-11T09:53:21.817Z","version":"0.61.0"}
I think we'll want to have @curt confirm this, but that looks to me like the the browzer bootstrapper will need to have the self-signed CA as part of it's trust store. It might be necessary to add it to the os trust but I would start by making a CA bundle of trusted certs and setting this in your compose file:
NODE_EXTRA_CA_CERTS=/some/path/to/extra/certs
If that doesn't work, maybe there's a browzer mechanism to add it to the bootstrapper I'm not aware of, but I think this would work for you.
Doesn't my docker compose already have that?
NODE_EXTRA_CA_CERTS: /persistent/pki/ziti-edge-controller-intermediate/certs/ziti-edge-controller-server.chain.pem
Maybe i am confusing stuff. I also tested by making bundle.pem and adding contents of ziti-edge-controller-server.chain.pem and example.com.cert to it with no luck.
This is how i made my example.com.cert:
openssl req -x509 -newkey rsa:4096 -sha256 -days 3650 -nodes -keyout example.com.key -out example.com.crt -subj "/CN=example.com" -addext "subjectAltName=DNS:example.com,DNS:*.example.com,IP:0.0.0.0"
I completely missed that your compose had it! Sorry about that... that chain is unlikely to contain the root CA, it should have everything BUT the root CA in it, I believe. Let me poke at it for a bit and see if I can find the issue.
Just to give you an update, I tried to get a workable docker compose file working today and got closer but ended up hitting a myriad of different issues.
With version 0.59.0 we introduced a bug into using NODE_EXTRA_CA_CERTS. If you use 0.58.0 you can get it to work, but you need to override the entrypoint as the current entrypoint script it's not compatible with setting NODE_EXTRA_CA_CERTS via environment variable. I just doubt anyone has tried to make a purely self-signed cert work, they must all be using legit CAs like LetsEncrypt...
Anyway, I've been trying to work through this but didn't end up getting something workable yet. I can share my compose if you want but I'm not sure if it's really useful yet?
As a workaround i used NODE_TLS_REJECT_UNAUTHORIZED: 0, but ...
I got another failed to verify certificate: x509: certificate signed by unknown authority, this time from the controller.
Here's my full setup and logs thus far:
docker-compose.yaml:
name: ${PROJ_NAME}
services:
ziti-controller:
image: "${ZITI_IMAGE}:${ZITI_VERSION}"
healthcheck:
test: curl -m 1 -s -k -f https://${ZITI_CTRL_EDGE_ADVERTISED_ADDRESS:-ziti-edge-controller}:${ZITI_CTRL_EDGE_ADVERTISED_PORT:-1280}/edge/client/v1/version
interval: 1s
timeout: 3s
retries: 30
env_file:
- ./.env
ports:
- ${ZITI_INTERFACE:-0.0.0.0}:${ZITI_CTRL_EDGE_ADVERTISED_PORT:-1280}:${ZITI_CTRL_EDGE_ADVERTISED_PORT:-1280}
- ${ZITI_INTERFACE:-0.0.0.0}:${ZITI_CTRL_ADVERTISED_PORT:-6262}:${ZITI_CTRL_ADVERTISED_PORT:-6262}
environment:
- ZITI_CTRL_NAME=${ZITI_CTRL_NAME:-ziti-edge-controller}
- ZITI_CTRL_EDGE_ADVERTISED_ADDRESS=${ZITI_CTRL_EDGE_ADVERTISED_ADDRESS:-ziti-edge-controller}
- ZITI_CTRL_EDGE_ADVERTISED_PORT=${ZITI_CTRL_EDGE_ADVERTISED_PORT:-1280}
- ZITI_CTRL_EDGE_IP_OVERRIDE=${ZITI_CTRL_EDGE_IP_OVERRIDE:-127.0.0.1}
- ZITI_CTRL_ADVERTISED_PORT=${ZITI_CTRL_ADVERTISED_PORT:-6262}
- ZITI_EDGE_IDENTITY_ENROLLMENT_DURATION=${ZITI_EDGE_IDENTITY_ENROLLMENT_DURATION}
- ZITI_ROUTER_ENROLLMENT_DURATION=${ZITI_ROUTER_ENROLLMENT_DURATION}
- ZITI_USER=${ZITI_USER:-admin}
- ZITI_PWD=${ZITI_PWD}
networks:
ziti:
aliases:
- ziti-edge-controller
ipv4_address: 192.168.144.3
volumes:
- ziti-fs:/persistent
entrypoint:
- "/var/openziti/scripts/run-controller.sh"
ziti-controller-init-container:
image: "${ZITI_IMAGE}:${ZITI_VERSION}"
depends_on:
ziti-controller:
condition: service_healthy
environment:
- ZITI_CTRL_EDGE_ADVERTISED_ADDRESS=${ZITI_CTRL_EDGE_ADVERTISED_ADDRESS:-ziti-edge-controller}
- ZITI_CTRL_EDGE_ADVERTISED_PORT=${ZITI_CTRL_EDGE_ADVERTISED_PORT:-1280}
env_file:
- ./.env
networks:
ziti:
volumes:
- ziti-fs:/persistent
entrypoint:
- "/var/openziti/scripts/run-with-ziti-cli.sh"
command:
- "/var/openziti/scripts/access-control.sh"
ziti-edge-router:
image: "${ZITI_IMAGE}:${ZITI_VERSION}"
env_file:
- ./.env
depends_on:
ziti-controller:
condition: service_healthy
ports:
- ${ZITI_INTERFACE:-0.0.0.0}:${ZITI_ROUTER_PORT:-3022}:${ZITI_ROUTER_PORT:-3022}
- ${ZITI_INTERFACE:-0.0.0.0}:${ZITI_ROUTER_LISTENER_BIND_PORT:-10080}:${ZITI_ROUTER_LISTENER_BIND_PORT:-10080}
- 3023:3023
environment:
- ZITI_CTRL_ADVERTISED_ADDRESS=${ZITI_CTRL_ADVERTISED_ADDRESS:-ziti-controller}
- ZITI_CTRL_ADVERTISED_PORT=${ZITI_CTRL_ADVERTISED_PORT:-6262}
- ZITI_CTRL_EDGE_ADVERTISED_ADDRESS=${ZITI_CTRL_EDGE_ADVERTISED_ADDRESS:-ziti-edge-controller}
- ZITI_CTRL_EDGE_ADVERTISED_PORT=${ZITI_CTRL_EDGE_ADVERTISED_PORT:-1280}
- ZITI_ROUTER_NAME=${ZITI_ROUTER_NAME:-ziti-edge-router}
- ZITI_ROUTER_ADVERTISED_ADDRESS=${ZITI_ROUTER_ADVERTISED_ADDRESS:-ziti-edge-router}
- ZITI_ROUTER_PORT=${ZITI_ROUTER_PORT:-3022}
- ZITI_ROUTER_LISTENER_BIND_PORT=${ZITI_ROUTER_LISTENER_BIND_PORT:-10080}
- ZITI_ROUTER_ROLES=public
networks:
- ziti
volumes:
- ziti-fs:/persistent
entrypoint: /bin/bash
command: "/var/openziti/scripts/run-router.sh edge"
ziti-console:
image: openziti/zac
working_dir: /usr/src/app
environment:
- ZAC_SERVER_CERT_CHAIN=/persistent/pki/${ZITI_CTRL_EDGE_ADVERTISED_ADDRESS:-ziti-edge-controller}-intermediate/certs/${ZITI_CTRL_EDGE_ADVERTISED_ADDRESS:-ziti-edge-controller}-server.cert
- ZAC_SERVER_KEY=/persistent/pki/${ZITI_CTRL_EDGE_ADVERTISED_ADDRESS:-ziti-edge-controller}-intermediate/keys/${ZITI_CTRL_EDGE_ADVERTISED_ADDRESS:-ziti-edge-controller}-server.key
- ZITI_CTRL_EDGE_ADVERTISED_ADDRESS=${ZITI_CTRL_EDGE_ADVERTISED_ADDRESS:-ziti-edge-controller}
- ZITI_CTRL_EDGE_ADVERTISED_PORT=${ZITI_CTRL_EDGE_ADVERTISED_PORT:-1280}
- ZITI_CTRL_NAME=${ZITI_CTRL_NAME:-ziti-edge-controller}
- PORTTLS=8443
depends_on:
ziti-controller:
condition: service_healthy
ports:
- ${ZITI_INTERFACE:-0.0.0.0}:8443:8443
volumes:
- ziti-fs:/persistent
networks:
- ziti
keycloak:
image: quay.io/keycloak/keycloak:25.0.0
command:
- "start-dev"
- "--https-certificate-file=${KEYCLOAK_CERT}"
- "--https-certificate-key-file=${KEYCLOAK_KEY}"
- "--https-port=${KEYCLOAK_PORT}"
ports:
- ${KEYCLOAK_PORT}:${KEYCLOAK_PORT}
environment:
- KEYCLOAK_ADMIN=${KEYCLOAK_ADMIN_USER}
- KEYCLOAK_ADMIN_PASSWORD=${KEYCLOAK_ADMIN_PWD}
networks:
ziti:
ipv4_address: 192.168.144.2
volumes:
- ./docker-data:/docker-data
browzer:
image: ghcr.io/openziti/ziti-browzer-bootstrapper:latest
restart: always
depends_on:
ziti-controller:
condition: service_healthy
networks:
- ziti
volumes:
- ./docker-data:/docker-data
- ziti-fs:/persistent
ports:
- "${ZITI_BROWZER_PORT}:${ZITI_BROWZER_PORT}"
# entrypoint: ['/docker-data/start.sh', "/home/node/ziti-browzer-bootstrapper/zha-docker-entrypoint"]
environment:
# NODE_EXTRA_CA_CERTS: /persistent/pki/ziti-signing-root-ca/certs/ziti-signing-intermediate_grandparent_intermediate.cert
NODE_TLS_REJECT_UNAUTHORIZED: 0
NODE_ENV: production
ZITI_BROWZER_RUNTIME_ORIGIN_TRIAL_TOKEN: ${BROWZER_ORIGINAL_TRIAL_TOKEN}
ZITI_BROWZER_BOOTSTRAPPER_LOGLEVEL: debug
ZITI_BROWZER_RUNTIME_LOGLEVEL: debug
ZITI_CONTROLLER_HOST: ${ZITI_BROWZER_CONTROLLER_HOST}
ZITI_CONTROLLER_PORT: 1280
ZITI_BROWZER_BOOTSTRAPPER_HOST: example.com
ZITI_BROWZER_BOOTSTRAPPER_LISTEN_PORT: ${ZITI_BROWZER_PORT}
ZITI_BROWZER_BOOTSTRAPPER_CERTIFICATE_PATH: /docker-data/example.com.crt
ZITI_BROWZER_BOOTSTRAPPER_KEY_PATH: /docker-data/example.com.key
ZITI_BROWZER_BOOTSTRAPPER_SCHEME: https
ZITI_BROWZER_BOOTSTRAPPER_TARGETS: >
{
"targetArray": [
{
"vhost": "${ZITI_BROWZER_VHOST}",
"service": "${ZITI_BROWZER_SERVICE}",
"path": "/",
"scheme": "http",
"idp_issuer_base_url": "${KEYCLOAK_HOST_AND_PORT}realms/zitirealm",
"idp_client_id": "${ZITI_BROWZER_CLIENT_ID}",
"idp_type": "keycloak",
"idp_realm": "${KEYCLOAK_REALM}"
}
]
}
docker-whale:
image: crccheck/hello-world
ports:
- "2000:8000"
networks:
ziti:
driver: bridge
ipam:
config:
- subnet: 192.168.0.0/16
volumes:
ziti-fs:
driver: local
driver_opts:
type: none
o: bind
device: ${PWD}/${ZITI_VOLUME_DIR}
.env:
export PROJ_NAME="openziti"
export WILDCARD_DNS="example.com"
export COMPOSE_FILE="./docker-compose.yaml"
# OpenZiti Variables
export ZITI_IMAGE=openziti/quickstart
export ZITI_VERSION=latest
export ZITI_VOLUME_DIR="./ziti-fs"
# the user and password to use
# Leave password blank to have a unique value generated or set the password explicitly
export ZITI_USER=admin
export ZITI_PWD=rootroot
export ZITI_INTERFACE=0.0.0.0
# controller name, address/port information
export ZITI_CTRL_NAME=ziti-controller
export ZITI_CTRL_EDGE_ADVERTISED_ADDRESS=ziti-edge-controller
export ZITI_CTRL_ADVERTISED_ADDRESS=ziti-controller
#ZITI_CTRL_EDGE_IP_OVERRIDE=10.10.10.10
#ZITI_CTRL_EDGE_ADVERTISED_PORT=8441
#ZITI_CTRL_ADVERTISED_PORT=8440
# The duration of the enrollment period (in minutes), default if not set. shown - 7days
export ZITI_EDGE_IDENTITY_ENROLLMENT_DURATION=10080
export ZITI_ROUTER_ENROLLMENT_DURATION=10080
# router address/port information
#ZITI_ROUTER_NAME=ziti-edge-router
#ZITI_ROUTER_ADVERTISED_ADDRESS=ziti-edge-router
#ZITI_ROUTER_PORT=8442
#ZITI_ROUTER_IP_OVERRIDE=10.10.10.10
#ZITI_ROUTER_LISTENER_BIND_PORT=8444
#ZITI_ROUTER_ROLES=public
# keycloak
export KEYCLOAK_ADMIN_USER=admin
export KEYCLOAK_ADMIN_PWD=rootroot
export KEYCLOAK_REALM=zitirealm
export KEYCLOAK_PORT=8081
export KEYCLOAK_BASE="keycloak.example.com"
export KEYCLOAK_CERT=/docker-data/example.com.crt
export KEYCLOAK_KEY=/docker-data/example.com.key
export KEYCLOAK_NOSSL_URL="http://0.0.0.0:8080"
# browzer
export ZITI_BROWZER_CONTROLLER_HOST=ziti-edge-controller.com
export ZITI_BROWZER_PORT=443
export ZITI_BROWZER_WSS_ER_PORT=8505
export ZITI_BROWZER_HTTP_AGENT_URL="browzer.${WILDCARD_DNS}"
export KEYCLOAK_HOST_AND_PORT="https://${KEYCLOAK_BASE}:${KEYCLOAK_PORT}/"
export ZITI_BROWZER_OIDC_URL="${KEYCLOAK_HOST_AND_PORT}realms/${KEYCLOAK_REALM}"
export ZITI_BROWZER_CLIENT_ID="browzerBootstrapClient"
export ZITI_BROWZER_VHOST="docker-whale.example.com"
export ZITI_BROWZER_SERVICE="docker.whale"
export ZITI_BROWZER_WSS_ER_HOST="wss.${WILDCARD_DNS}"
export ZITI_BROWZER_IDENTITIES="carlosdomhleba@gmail.com"
export ZITI_BROWZER_DOCKER_PROJECT="browzer-compose"
export BROWZER_ORIGINAL_TRIAL_TOKEN="my_token_here"
After i do docker compose up i use install.sh file which i run.
Here's install.sh:
#!/bin/bash
source .env
./keycloak-configure.sh
./browzer-configure.sh
keycloak-configure.sh:
function kcadm {
docker compose \
-f ${COMPOSE_FILE}.yml \
--project-name ${PROJ_NAME} \
exec -it keycloak \
/opt/keycloak/bin/kcadm.sh $@;
}
kcadm config credentials \
--server ${KEYCLOAK_NOSSL_URL} \
--realm master \
--user ${KEYCLOAK_ADMIN_USER} \
--password ${KEYCLOAK_ADMIN_PWD}
kcadm create realms \
-s realm=${KEYCLOAK_REALM} \
-s enabled=true
kcadm create clients \
-r ${KEYCLOAK_REALM} \
-s clientId=${ZITI_BROWZER_CLIENT_ID} \
-s protocol=openid-connect \
-s 'redirectUris=["https://'${ZITI_BROWZER_VHOST}'/*","https://brozac.'${WILDCARD_DNS}'/*"]' \
-s 'webOrigins=["https://'${ZITI_BROWZER_VHOST}'","https://brozac.'${WILDCARD_DNS}'"]' \
-s 'directAccessGrantsEnabled=true'
CLIENT_SCOPE_ID=$(kcadm get clients -r ${KEYCLOAK_REALM} | jq -r '.[] | select(.clientId == "'${ZITI_BROWZER_CLIENT_ID}'") | .id')
kcadm update realms/${KEYCLOAK_REALM}/clients/${CLIENT_SCOPE_ID} --set fullScopeAllowed=false
kcadm create clients/${CLIENT_SCOPE_ID}/protocol-mappers/models \
-r ${KEYCLOAK_REALM} \
-s name=audience-mapping \
-s protocol=openid-connect \
-s protocolMapper=oidc-audience-mapper \
-s config.\"included.custom.audience\"="${ZITI_BROWZER_CLIENT_ID}" \
-s config.\"access.token.claim\"=\"true\" \
-s config.\"id.token.claim\"=\"false\"
browzer-configure.sh:
if grep -q "wss:" $ZITI_VOLUME_DIR/ziti-edge-router.yaml; then
echo "router config file appears to have web socket listener already: $ZITI_VOLUME_DIR/ziti-edge-router.yaml"
else
echo "adding/replacing settings in the quickstart router. adding web socket listener, configuring ws block"
sed -i 's#tproxy|host#tproxy|host\n - binding: edge\n address: wss:0.0.0.0:'${ZITI_BROWZER_WSS_ER_PORT}'\n options:\n advertise: '${ZITI_BROWZER_WSS_ER_HOST}':'${ZITI_BROWZER_WSS_ER_PORT}'\n connectTimeoutMs: 5000\n getSessionTimeout: 60#g' $ZITI_VOLUME_DIR/ziti-edge-router.yaml
sed -i 's`#transport\:`transport\:`g' $ZITI_VOLUME_DIR/ziti-edge-router.yaml
sed -i 's`# ws\:` ws\:`g' $ZITI_VOLUME_DIR/ziti-edge-router.yaml
sed -i 's`# ` `g' $ZITI_VOLUME_DIR/ziti-edge-router.yaml
echo "Required to restart edge router"
fi
function zitiEx {
docker compose \
-f ${COMPOSE_FILE}.yml \
--project-name ${PROJ_NAME} \
exec -it ziti-controller \
/var/openziti/ziti-bin/ziti "$@";
}
zitiEx edge login -u $ZITI_USER -p $ZITI_PWD
ziti_object_prefix=browzer-keycloak
issuer=$(curl -k ${ZITI_BROWZER_OIDC_URL}/.well-known/openid-configuration | jq -r .issuer)
jwks=$(curl -k ${ZITI_BROWZER_OIDC_URL}/.well-known/openid-configuration | jq -r .jwks_uri)
echo "OIDC issuer : $issuer"
echo "OIDC jwks url : $jwks"
ext_jwt_signer=$(zitiEx edge create ext-jwt-signer "${ziti_object_prefix}-ext-jwt-signer" "${issuer}" --jwks-endpoint "${jwks}" --audience "${ZITI_BROWZER_CLIENT_ID}" --claims-property email)
echo "ext jwt signer id: $ext_jwt_signer"
auth_policy=$(zitiEx edge create auth-policy ${ziti_object_prefix}-auth-policy --primary-ext-jwt-allowed --primary-ext-jwt-allowed-signers ${ext_jwt_signer} --secondary-req-ext-jwt-signer ${ext_jwt_signer})
echo "auth policy id: $auth_policy"
echo "creating users specified by ZITI_BROWZER_IDENTITIES: ${ZITI_BROWZER_IDENTITIES}"
for id in ${ZITI_BROWZER_IDENTITIES}; do
zitiEx edge create identity user "${id}" --auth-policy ${auth_policy} --external-id "${id}" -a docker.whale.dialers,brozac.dialers
done
echo "adding router ziti-edge-router as docker.whale.binders"
zitiEx edge update identity "ziti-edge-router" -a docker.whale.binders,brozac.binders,brozac.binders
source ./docker.whale
createService
# source $SCRIPT_DIR/brozac
# createService
Heres controllers logs:
ziti-controller-1 | [ 65.266] ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:1280]: {error=[EOF] remote=[192.168.0.1:57314]} handshake failed
ziti-controller-1 | [ 65.299] INFO ziti/controller/env.(*AppEnv).GetControllerPublicKey: looking for signer: tj28pVRBNLkyZcDjzxf29ZCSkXIc4gRzdIrrjCfxsdo
ziti-controller-1 | [ 65.299] ERROR ziti/controller/env.(*AppEnv).getJwtTokenFromRequest: {error=[token is unverifiable: error while executing keyfunc: key for kid tj28pVRBNLkyZcDjzxf29ZCSkXIc4gRzdIrrjCfxsdo, not found]} error during JWT parsing during API request
ziti-controller-1 | [ 65.613] INFO ziti/controller/env.(*AppEnv).GetControllerPublicKey: looking for signer: tj28pVRBNLkyZcDjzxf29ZCSkXIc4gRzdIrrjCfxsdo
ziti-controller-1 | [ 65.613] ERROR ziti/controller/env.(*AppEnv).getJwtTokenFromRequest: {error=[token is unverifiable: error while executing keyfunc: key for kid tj28pVRBNLkyZcDjzxf29ZCSkXIc4gRzdIrrjCfxsdo, not found]} error during JWT parsing during API request
ziti-controller-1 | [ 65.637] ERROR ziti/controller/model.(*AuthModuleExtJwt).pubKeyLookup: {extJwtSignerName=[browzer-keycloak-ext-jwt-signer] error=[could not resolve jwks endpoint: Get "https://keycloak.example.com:8081/realms/zitirealm/protocol/openid-connect/certs": tls: failed to verify certificate: x509: certificate signed by unknown authority] kid=[tj28pVRBNLkyZcDjzxf29ZCSkXIc4gRzdIrrjCfxsdo] issuer=[https://keycloak.example.com:8081/realms/zitirealm] extJwtSignerId=[2Nqw7v8iAUgRQlnx1SfQxp] method=[ext-jwt]} error attempting to resolve extJwtSigner certificate used for signing
ziti-controller-1 | [ 65.637] ERROR ziti/controller/model.(*AuthModuleExtJwt).process: {authMethod=[ext-jwt]} authorization failed, jwt did not verify
ziti-controller-1 | [ 65.643] INFO ziti/controller/env.(*AppEnv).GetControllerPublicKey: looking for signer: tj28pVRBNLkyZcDjzxf29ZCSkXIc4gRzdIrrjCfxsdo
ziti-controller-1 | [ 65.643] ERROR ziti/controller/env.(*AppEnv).getJwtTokenFromRequest: {error=[token is unverifiable: error while executing keyfunc: key for kid tj28pVRBNLkyZcDjzxf29ZCSkXIc4gRzdIrrjCfxsdo, not found]} error during JWT parsing during API request
browzer-1 | {"error":"User [carlosdomhleba@gmail.com] cannot be authenticated onto Ziti Network","error_code":1001,"level":"error","message":"Check that this 'externalId' exists and has case-sensitive match","timestamp":"2024-06-12T11:18:16.432Z","version":"0.61.0"}
ziti-controller-1 | [ 65.709] INFO ziti/controller/env.(*AppEnv).GetControllerPublicKey: looking for signer: tj28pVRBNLkyZcDjzxf29ZCSkXIc4gRzdIrrjCfxsdo
ziti-controller-1 | [ 65.709] ERROR ziti/controller/env.(*AppEnv).getJwtTokenFromRequest: {error=[token is unverifiable: error while executing keyfunc: key for kid tj28pVRBNLkyZcDjzxf29ZCSkXIc4gRzdIrrjCfxsdo, not found]} error during JWT parsing during API request
ziti-controller-1 | [ 66.210] INFO ziti/controller/env.(*AppEnv).GetControllerPublicKey: looking for signer: tj28pVRBNLkyZcDjzxf29ZCSkXIc4gRzdIrrjCfxsdo
ziti-controller-1 | [ 66.210] ERROR ziti/controller/env.(*AppEnv).getJwtTokenFromRequest: {error=[token is unverifiable: error while executing keyfunc: key for kid tj28pVRBNLkyZcDjzxf29ZCSkXIc4gRzdIrrjCfxsdo, not found]} error during JWT parsing during API request
ziti-controller-1 | [ 66.210] ERROR ziti/controller/model.(*AuthModuleExtJwt).process: {authMethod=[ext-jwt]} authorization failed, jwt did not verify
ziti-controller-1 | [ 66.215] INFO ziti/controller/env.(*AppEnv).GetControllerPublicKey: looking for signer: tj28pVRBNLkyZcDjzxf29ZCSkXIc4gRzdIrrjCfxsdo
ziti-controller-1 | [ 66.215] ERROR ziti/controller/env.(*AppEnv).getJwtTokenFromRequest: {error=[token is unverifiable: error while executing keyfunc: key for kid tj28pVRBNLkyZcDjzxf29ZCSkXIc4gRzdIrrjCfxsdo, not found]} error during JWT parsing during API request
ziti-controller-1 | [ 66.216] ERROR ziti/controller/model.(*AuthModuleExtJwt).process: {authMethod=[ext-jwt]} authorization failed, jwt did not verify
ziti-controller-1 | [ 66.219] INFO ziti/controller/env.(*AppEnv).GetControllerPublicKey: looking for signer: tj28pVRBNLkyZcDjzxf29ZCSkXIc4gRzdIrrjCfxsdo
ziti-controller-1 | [ 66.219] ERROR ziti/controller/env.(*AppEnv).getJwtTokenFromRequest: {error=[token is unverifiable: error while executing keyfunc: key for kid tj28pVRBNLkyZcDjzxf29ZCSkXIc4gRzdIrrjCfxsdo, not found]} error during JWT parsing during API request
ziti-controller-1 | [ 66.729] INFO ziti/controller/env.(*AppEnv).GetControllerPublicKey: looking for signer: tj28pVRBNLkyZcDjzxf29ZCSkXIc4gRzdIrrjCfxsdo
ziti-controller-1 | [ 66.729] ERROR ziti/controller/env.(*AppEnv).getJwtTokenFromRequest: {error=[token is unverifiable: error while executing keyfunc: key for kid tj28pVRBNLkyZcDjzxf29ZCSkXIc4gRzdIrrjCfxsdo, not found]} error during JWT parsing during API request
ziti-controller-1 | [ 66.730] ERROR ziti/controller/model.(*AuthModuleExtJwt).process: {authMethod=[ext-jwt]} authorization failed, jwt did not verify
ziti-controller-1 | [ 66.734] INFO ziti/controller/env.(*AppEnv).GetControllerPublicKey: looking for signer: tj28pVRBNLkyZcDjzxf29ZCSkXIc4gRzdIrrjCfxsdo
ziti-controller-1 | [ 66.734] ERROR ziti/controller/env.(*AppEnv).getJwtTokenFromRequest: {error=[token is unverifiable: error while executing keyfunc: key for kid tj28pVRBNLkyZcDjzxf29ZCSkXIc4gRzdIrrjCfxsdo, not found]} error during JWT parsing during API request
ziti-controller-1 | [ 66.734] ERROR ziti/controller/model.(*AuthModuleExtJwt).process: {authMethod=[ext-jwt]} authorization failed, jwt did not verify
ziti-controller-1 | [ 66.738] INFO ziti/controller/env.(*AppEnv).GetControllerPublicKey: looking for signer: tj28pVRBNLkyZcDjzxf29ZCSkXIc4gRzdIrrjCfxsdo
ziti-controller-1 | [ 66.738] ERROR ziti/controller/env.(*AppEnv).getJwtTokenFromRequest: {error=[token is unverifiable: error while executing keyfunc: key for kid tj28pVRBNLkyZcDjzxf29ZCSkXIc4gRzdIrrjCfxsdo, not found]} error during JWT parsing during API request
browzer-1 | {"error":"User [carlosdomhleba@gmail.com] cannot be authenticated onto Ziti Network","error_code":1001,"level":"error","message":"Check that this 'externalId' exists and has case-sensitive match","timestamp":"2024-06-12T11:18:21.932Z","version":"0.61.0"}
ziti-controller-1 | [ 71.663] INFO ziti/controller/env.(*AppEnv).GetControllerPublicKey: looking for signer: tj28pVRBNLkyZcDjzxf29ZCSkXIc4gRzdIrrjCfxsdo
ziti-controller-1 | [ 71.663] ERROR ziti/controller/env.(*AppEnv).getJwtTokenFromRequest: {error=[token is unverifiable: error while executing keyfunc: key for kid tj28pVRBNLkyZcDjzxf29ZCSkXIc4gRzdIrrjCfxsdo, not found]} error during JWT parsing during API request
ziti-controller-1 | [ 71.691] ERROR ziti/controller/model.(*AuthModuleExtJwt).pubKeyLookup: {extJwtSignerId=[2Nqw7v8iAUgRQlnx1SfQxp] extJwtSignerName=[browzer-keycloak-ext-jwt-signer] method=[ext-jwt] error=[could not resolve jwks endpoint: Get "https://keycloak.example.com:8081/realms/zitirealm/protocol/openid-connect/certs": tls: failed to verify certificate: x509: certificate signed by unknown authority] kid=[tj28pVRBNLkyZcDjzxf29ZCSkXIc4gRzdIrrjCfxsdo] issuer=[https://keycloak.example.com:8081/realms/zitirealm]} error attempting to resolve extJwtSigner certificate used for signing
ziti-controller-1 | [ 71.691] ERROR ziti/controller/model.(*AuthModuleExtJwt).process: {authMethod=[ext-jwt]} authorization failed, jwt did not verify
ziti-controller-1 | [ 71.695] INFO ziti/controller/env.(*AppEnv).GetControllerPublicKey: looking for signer: tj28pVRBNLkyZcDjzxf29ZCSkXIc4gRzdIrrjCfxsdo
ziti-controller-1 | [ 71.695] ERROR ziti/controller/env.(*AppEnv).getJwtTokenFromRequest: {error=[token is unverifiable: error while executing keyfunc: key for kid tj28pVRBNLkyZcDjzxf29ZCSkXIc4gRzdIrrjCfxsdo, not found]} error during JWT parsing during API request
ziti-controller-1 | [ 71.696] ERROR ziti/controller/model.(*AuthModuleExtJwt).process: {authMethod=[ext-jwt]} authorization failed, jwt did not verify
ziti-controller-1 | [ 71.700] INFO ziti/controller/env.(*AppEnv).GetControllerPublicKey: looking for signer: tj28pVRBNLkyZcDjzxf29ZCSkXIc4gRzdIrrjCfxsdo
ziti-controller-1 | [ 71.700] ERROR ziti/controller/env.(*AppEnv).getJwtTokenFromRequest: {error=[token is unverifiable: error while executing keyfunc: key for kid tj28pVRBNLkyZcDjzxf29ZCSkXIc4gRzdIrrjCfxsdo, not found]} error during JWT parsing during API request
Browser console:
GET https://docker-whale.example.com/browzer_error?browzer_error_data==%7B%22status%22:511,%22code%22:1001,%22title%22:%22User%20%5Bcarlosdomhleba@gmail.com%5D%20cannot%20be%20authenticated%20onto%20Ziti%20Network%22,%22message%22:%22Check%20that%20this%20%27externalId%27%20exists%20and%20has%20case-sensitive%20match%22,%22myvar%22:%7B%22type%22:%22zbr%22%7D%7D 511 (Network Authentication Required)Hi @CarlosHleb. Discourse marked your posts as spam, sorry about that... I've added the one that had the most information back to this thread...
Thanks for providing your steps. They're incredibly similar to what I was doing, too. I am trying to get a workable answer/solution for you. I'll keep poking at it...
Ok @CarlosHleb -- here we go. I got it all to work together. I put it all up on my github repo here.
The README.md should take you through it all and I think between what you have and what's on that github you should be able to get it all running.
Here's the gist from that readme:
To bring up this environment do the following:
- clone this repo
- cd to
browzer-docker-compose - mkdir 'docker-data'
cp env-template .env- open the
.envfile and edit all the relevant settings - after editing the
.envfile, source it:source .env - generate the self-signed root CA and wildcard server keypair by running
./initialize-pki.sh - add the root ca to the appropriate trust store
- bring up the compose file:
docker compose up(use -d if you want) - configure keycloak by running:
./keycloak-configure.sh - configure the openziti overlay by running:
./openziti-configure.sh - OPTIONALLY: configure GitHub/Google (if using federated auth)
I didn't record a video showing everything in action, but if you need it I can. Have a look. Two key things is using the 'alt' server certs and also using a self-signed CA to make a wildcard server cert...
1 Like
Oh one more thing, I use browzer 0.58.0 and overrode the entrypoint/command. That's also relevant. I'm going to try to make a new container that works with NODE_EXTRA_CA_CERTS next so we can get on a newer version of browzer (0.58.0 is not 'old', but it's not latest)
I have a PR up that will allow NODE_EXTRA_CA_CERTS to be overridden. I'd expect this fix to roll out in 0.62 of browzer. When it rolls out, I'll reply here and update the example compose file with :latest and ensure it all still works. Cheers
@CarlosHleb, version 0.61.1 of the browzer bootstrapper was released and worked for me so you can update the compose file to :latest and it should work fine.
cheers
I'm covering this live on Ziti TV this week. If you're interested in seeing it live or watching the replay, it'll be here:
