Creating certs for a remote private router

Hi,

I’m trying to achieve what’s in the diagram below :
Diagramme sans nom.drawio

I did the Oracle Cloud part using the quickstart and now, I’m kinda stuck with the PKI certs creation part in AWS, I’m trying to follow along with the docker-compose you guys made and it’s using these two functions to create the router’s cets :

function createRouterPki {
  # Allow router name to be passed in as arg
  router_name="${1-}"
  if [[ "${router_name}" == "" ]]; then
    echo -e "  * ERROR: $(RED "routerName needs to be supplied") "
    return 1
  fi
  mkdir -p "${ZITI_PKI_OS_SPECIFIC}/routers/${router_name}"
  export ZITI_ROUTER_IDENTITY_CERT="${ZITI_PKI_OS_SPECIFIC}/routers/${router_name}/client.cert"
  export ZITI_ROUTER_IDENTITY_SERVER_CERT="${ZITI_PKI_OS_SPECIFIC}/routers/${router_name}/server.cert"
  export ZITI_ROUTER_IDENTITY_KEY="${ZITI_PKI_OS_SPECIFIC}/routers/${router_name}/server.key"
  export ZITI_ROUTER_IDENTITY_CA="${ZITI_PKI_OS_SPECIFIC}/routers/${router_name}/cas.cert"
  pki_client_server "${router_name},localhost,127.0.0.1,$(hostname)" "${ZITI_CONTROLLER_INTERMEDIATE_NAME}" "${ZITI_EDGE_ROUTER_IP_OVERRIDE-}" "${router_name}"
}

function pki_client_server {
  allow_list=${1-}
  ZITI_CA_NAME_local=$2
  ip_local=$3
  file_name=$4

  if [[ "${ip_local}" == "" ]]; then
    ip_local="127.0.0.1"
  fi

  if ! test -f "${ZITI_PKI}/${ZITI_CA_NAME_local}/keys/${file_name}-server.key"; then
    echo "Creating server cert from ca: ${ZITI_CA_NAME_local} for ${allow_list} / ${ip_local}"
    "${ZITI_BIN_DIR-}/ziti" pki create server --pki-root="${ZITI_PKI_OS_SPECIFIC}" --ca-name "${ZITI_CA_NAME_local}" \
          --server-file "${file_name}-server" \
          --dns "${allow_list}" --ip "${ip_local}" \
          --server-name "${file_name} server certificate"
  else
    echo "Creating server cert from ca: ${ZITI_CA_NAME_local} for ${allow_list}"
    echo "key exists"
  fi

  if ! test -f "${ZITI_PKI}/${ZITI_CA_NAME_local}/keys/${file_name}-client.key"; then
    echo "Creating client cert from ca: ${ZITI_CA_NAME_local} for ${allow_list}"
    "${ZITI_BIN_DIR-}/ziti" pki create client --pki-root="${ZITI_PKI_OS_SPECIFIC}" --ca-name "${ZITI_CA_NAME_local}" \
          --client-file "${file_name}-client" \
          --key-file "${file_name}-server" \
          --client-name "${file_name}"
  else
    echo "Creating client cert from ca: ${ZITI_CA_NAME_local} for ${allow_list}"
    echo "key exists"
  fi
  echo " "
}

These two functions translates to these commands :

mkdir -p "/home/ubuntu/.ziti/quickstart/instance-20220723-2134/pki/routers/ziti-test-private-router"
export ZITI_ROUTER_IDENTITY_CERT="/home/ubuntu/.ziti/quickstart/instance-20220723-2134/pki/routers/ziti-test-private-router/client.cert"
export ZITI_ROUTER_IDENTITY_SERVER_CERT="/home/ubuntu/.ziti/quickstart/instance-20220723-2134/pki/routers/ziti-test-private-router/server.cert"
export ZITI_ROUTER_IDENTITY_KEY="/home/ubuntu/.ziti/quickstart/instance-20220723-2134/pki/routers/ziti-test-private-router/server.key"
export ZITI_ROUTER_IDENTITY_CA="/home/ubuntu/.ziti/quickstart/instance-20220723-2134/pki/routers/ziti-test-private-router/cas.cert"

ziti pki create server --pki-root="/home/ubuntu/.ziti/quickstart/instance-20220723-2134/pki" --ca-name "instance-20220723-2134-intermediate" --server-file "ziti-test-private-router-server" --dns "ziti-test-private-router,localhost,127.0.0.1" --ip "127.0.0.1,35.180.190.208" --server-name "ziti-test-private-router server certificate"

ziti pki create client --pki-root="/home/ubuntu/.ziti/quickstart/instance-20220723-2134/pki" --ca-name "instance-20220723-2134-intermediate" --client-file "ziti-test-private-router-client" --key-file "ziti-test-private-router-server" --client-name "ziti-test-private-router"

N.B : 35.180.190.208 is my AWS EC2 instance public IP. Correct me if I’m wrong, but I must put the public IP in here (or public External DNS) ?

The problem I’m facing, is that when I look certs of the private routers created by the docker-compose, they’re stored in ZITI_HOME/pki/routers/private_router_name/. But, when trying with the command above, it doesn’t create this folder. Instead, it’s putting them inside the controller intermediate pki folder.

1 Like

Here’s what you will need to do.

First, make sure the public edge router has a “link listener” section and that the advertise address in there is addressable by the private routers and that the port listed in there is open in the web ACL. If not addressable or if the port is not allowed through the web ACL (and local iptables too) then your private routers won’t be able to connect to the public edge router (which is what you want).

Once you are sure the edge router is setup properly - now you can go about registering the private edge routers. For that you just need to:

  • Create the private edge routers. this line from the quickstart is what does it. It’s basically: ziti edge create edge-router my.private.er.1 -o my.private.er.1.jwt -t. I highly recommend using the -t option which tells the controller that this router can also act like a tunneler. (the quickstart and docker-compose setup always uses -t).

  • scp the my.private.er.1.jwt file to the private edge router in AWS

  • ssh to the private er in AWS. run this command to get ziti and put it on your shell’s path (or do it however you like): source /dev/stdin <<< "$(wget -qO- https://raw.githubusercontent.com/openziti/ziti/release-next/quickstart/docker/image/ziti-cli-functions.sh)"; getZiti yes. ziti-router will be on your path now

  • use ziti-router to enroll the jwt you just scp’ed. This line in the quickstart does that. It’s bascially: ziti-router enroll my.private.router.1.config.yaml --jwt my.private.router.1.jwt

  • Now your router should start and run. ziti-router run my.private.router.1.config.yaml. I always run it interactively one time myself just to make sure i have the config correct etc. Then I’ll copy the systemd file for the router from the initial quickstart and set it up with systemd.

I don’t know why the quickstart still references the “createRouterPki”… I think that’s just leftover from days long ago. I probably should look into that and clean it up. I’ll make a mental note of that too.

I think I got all these steps right. It’s on the list of things to document, how to setup more nodes but we just haven’t done it yet. If these instructions aren’t good enough or you get stuck, just post back and let me know where you’re at.

1 Like

I ended up making a video too where I narrate the changes I did, why, etc. Hopefully that helps too

2 Likes

and one last thing not shown in the video. After the private router comes online you can issue ziti fabric list links. It should show you the “dialer” and “acceptor” and one link (from private router <–> public router)

ziti fabric list links
╭────────────────────────┬────────────┬─────────────────────────────┬─────────────┬─────────────┬─────────────┬───────────┬────────┬───────────╮
│ ID                     │ DIALER     │ ACCEPTOR                    │ STATIC COST │ SRC LATENCY │ DST LATENCY │ STATE     │ STATUS │ FULL COST │
├────────────────────────┼────────────┼─────────────────────────────┼─────────────┼─────────────┼─────────────┼───────────┼────────┼───────────┤
│ 7S1i1MwO8B2r14C8ROPstZ │ private.er │ ip-172-31-42-64-edge-router │           1 │      13.9ms │      13.8ms │ Connected │     up │        27 │
╰────────────────────────┴────────────┴─────────────────────────────┴─────────────┴─────────────┴─────────────┴───────────┴────────┴───────────╯
results: 1-1 of 1

If you see that link - you’re good to go

2 Likes

Thanks a lot for the explanation and the video. I'm having a problem when running the private routers :

ERROR fabric/router/handler_ctrl.(*dialHandler).handle |link, linkDialer|: {linkProtocol=[tls] error=[error dialing outgoing link [l/3p5xbPElhvJqfRoVQgTyHN]: error dialing payload channel for [l/3p5xbPElhvJqfRoVQgTyHN]: x509: certificate is valid for 127.0.0.1, not 144.24.198.122] routerVersion=[v0.26.3] linkId=[3p5xbPElhvJqfRoVQgTyHN] routerId=[xIsDkIK5a9] address=[tls:144.24.198.122:10080]} link dialing failed

Here's my router's config file :

v: 3

identity:
  cert:                 "/home/ubuntu/ziti/aws-private-router/pki/client.cert"
  server_cert:          "/home/ubuntu/ziti/aws-private-router/pki/server.cert"
  key:                  "/home/ubuntu/ziti/aws-private-router/pki/server.key"
  ca:                   "/home/ubuntu/ziti/aws-private-router/pki/cas.cert"

ctrl:
  endpoint:             tls:144.24.198.122:6262# Oracle Cloud instance public IP 

link:
  dialers:
    - binding: transport

listeners:
  - binding: edge
    address: tls:0.0.0.0:8442
    options:
      advertise: 172.31.43.155:8442# AWS EC2 instance private IP
      connectTimeoutMs: 1000
      getSessionTimeout: 60s
  - binding: tunnel
    options:
      mode: host


edge:
  csr:
    country: US
    province: NC
    locality: Charlotte
    organization: NetFoundry
    organizationalUnit: Ziti
    sans:
      dns:
        - ip-172-31-43-155# AWS EC2 instance hostname
        - localhost
      ip:
        - "127.0.0.1"
        - "172.31.43.155"# AWS EC2 instance private IP
        - "35.180.190.208"# AWS EC2 instance public IP

forwarder:
  latencyProbeInterval: 10
  xgressDialQueueLength: 1000
  xgressDialWorkerCount: 128
  linkDialQueueLength: 1000
  linkDialWorkerCount: 32

Inbound Ports 6262 and 8442 are open in both of my EC2 instance and Oracle Cloud VM.

Looks like the certificate is denying the controller's connextion certificate is valid for 127.0.0.1, not 144.24.198.122, do you know what may cause this issue ?

Thanks

This is the exact issue I was experiencing. I had to have a few days break to work on something else.. and will be revisiting this over the next few days.

I was getting this error when trying to setup a remote router that connected to a controller that I setup a while ago.. I believe the problem was with the controller PKI.. that caused this error

To resolve, @TheLumberjack provided the following instructions.. I have not gotten around to doing this just yet.. because I am trying to triangulate the specific cause of the problem

What I understand is that there is some problem with the controller certificates. I should have more updates on this over the next few days. I was hesitant to just do this in case it caused other problems.

In contrast, what I did do in parallel.. to help with understanding the problem ... was to setup a new controller on a new machine using a fresh install. This seems to have worked..

Hence.. I believe the issue relates to the controller PKI.

Maybe.. try setting up a new controller on a new server. If this works.. then you know it's something related to the controller config..

Hopefully I can pin point the exact issue.. of which.. this a great opportunity to learn more about how the controller PKI is setup.

@TheLumberjack also provided some great insights in this post.. definitely work taking a read through.

Brilliant video… Very helpful and picked up a few more things… like the need to add in the private and public IP addresses. thanks :slight_smile:

Hunh. I must say that this is not expected. At first I thought this was going to be straightforward and that the external IP wasn't in the cert, but when I probed it, I could see this wasn't gonna be 'easy'.

Using openssl and this command one can inspect the certificate that is being returned:

openssl s_client -connect 144.24.198.122:6262 -showcerts  | openssl x509 -text

When I did that, I can see in there that it is returning DNS/IP that 'seem' OK, but I think there's a bug in the quickstart. When we tell you to set the external dns to the external ip, it looks like the certificates that get generated by the controller will have entries for both the DNS name and the IP. Let me show you what I mean, here's what your server is returning (using that command above. i added line-wrapping in this post to make it more legible):

            X509v3 Subject Alternative Name:
                DNS:instance-20220723-2134, DNS:localhost, 
                DNS:instance-20220723-2134, DNS:144.24.198.122, DNS:144.24.198.122, 
                IP Address:127.0.0.1, IP Address:144.24.198.122, 
                IP Address:144.24.198.122

See how the SANS has DNS:144.24.198.122 and also has IP Address:144.24.198.122? I expect that the DNS is overriding the IP and this is what is causing the problem.

The easiest way to fix this will probably be to regenerate the server certificate, update the config, and restart the controller. I actually just covered this recently with markamind over at here. I think if you follow those instructions and get a "clean" server certificate where the SANS doesn't report a IP in the DNS section, it should work. I also think those steps are pretty clear, but if not, post back. We'll get this going, I'm sure of it! :slight_smile:

1 Like

And if it wasn’t clear - that’s all stuff you would do on the controller and leverages the environment variables the quickstart leaves behind after install. That’s probably clear, but if not, I figured I’d mention it… After doing this, the router should connect to the controller properly.

I also filed an issue for this in the github repo IP-only controller adds IP to DNS of SANS · Issue #811 · openziti/ziti · GitHub

2 Likes

I provisioned another Instance on OCI and followed the steps mentionned on markamind’s post and now when I do zitiLogin, I’m getting this error :

RESTY 2022/08/07 16:30:02 ERROR Get "https://144.24.195.21:8441/version": x509: certificate signed by unknown authority, Attempt 1
RESTY 2022/08/07 16:30:02 ERROR Get "https://144.24.195.21:8441/version": x509: certificate signed by unknown authority, Attempt 2
RESTY 2022/08/07 16:30:02 ERROR Get "https://144.24.195.21:8441/version": x509: certificate signed by unknown authority, Attempt 3
RESTY 2022/08/07 16:30:03 ERROR Get "https://144.24.195.21:8441/version": x509: certificate signed by unknown authority, Attempt 4
RESTY 2022/08/07 16:30:03 ERROR Get "https://144.24.195.21:8441/version": x509: certificate signed by unknown authority, Attempt 5
RESTY 2022/08/07 16:30:04 ERROR Post "https://144.24.195.21:8441/authenticate?method=password": x509: certificate signed by unknown authority, Attempt 1
RESTY 2022/08/07 16:30:04 ERROR Post "https://144.24.195.21:8441/authenticate?method=password": x509: certificate signed by unknown authority, Attempt 2
RESTY 2022/08/07 16:30:05 ERROR Post "https://144.24.195.21:8441/authenticate?method=password": x509: certificate signed by unknown authority, Attempt 3
RESTY 2022/08/07 16:30:05 ERROR Post "https://144.24.195.21:8441/authenticate?method=password": x509: certificate signed by unknown authority, Attempt 4
RESTY 2022/08/07 16:30:05 ERROR Post "https://144.24.195.21:8441/authenticate?method=password": x509: certificate signed by unknown authority, Attempt 5
error: unable to authenticate to https://144.24.195.21:8441. Error: Post "https://144.24.195.21:8441/authenticate?method=password": x509: certificate signed by unknown authority

The output of openssl s_client -connect 144.24.195.21:6262 -showcerts | openssl x509 -text looks fine.

I think that the problem might be with the expressInstall generating me two Root and intermediate PKIs :

Variables set before running the expressInstall :

export EXTERNAL_DNS="$(curl -s eth0.me)" 
export EXTERNAL_IP="$(curl -s eth0.me)"       
export ZITI_EDGE_CONTROLLER_IP_OVERRIDE="${EXTERNAL_IP}"
export ZITI_EDGE_ROUTER_IP_OVERRIDE="${EXTERNAL_IP}"
export ZITI_EDGE_CONTROLLER_HOSTNAME="${EXTERNAL_DNS}"
export ZITI_EDGE_ROUTER_HOSTNAME="${EXTERNAL_DNS}"
export ZITI_EDGE_CONTROLLER_PORT=8441
export ZITI_EDGE_ROUTER_PORT=8442

Variable set before creating the new server CA :

EDGE_CONTROLLER_EXTERNAL_DNS_NAME=144.24.195.21
EDGE_CONTROLLER_PRIVATE_DNS_NAME=10.0.0.235
EDGE_CONTROLLER_EXTERNAL_IP_ADDRESS=144.24.195.21
EDGE_CONTROLLER_PRIVATE_IP_ADDRESS=10.0.0.235
file_name="${ZITI_CONTROLLER_HOSTNAME}-$(date +'%Y-%m-%d_%H%M%S')"

Here’s my controller config :

v: 3

#trace:
#  path: "controller.trace"

#profile:
#  memory:
#    path: ctrl.memprof

db:                     "/home/ubuntu/.ziti/quickstart/instance-20220807-1808/db/ctrl.db"

identity:
  cert:                 "/home/ubuntu/.ziti/quickstart/instance-20220807-1808/pki/instance-20220807-1808-intermediate/certs/instance-20220807-1808-client.cert"
  server_cert:          "/home/ubuntu/.ziti/quickstart/instance-20220807-1808/pki/instance-20220807-1808-intermediate/certs/instance-20220807-1808-server.chain.pem"
  key:                  "/home/ubuntu/.ziti/quickstart/instance-20220807-1808/pki/instance-20220807-1808-intermediate/keys/instance-20220807-1808-server.key"
  ca:                   "/home/ubuntu/.ziti/quickstart/instance-20220807-1808/pki/cas.pem"

# Network Configuration
#
# Configure how the controller will establish and manage the overlay network, and routing operations on top of
# the network.
#
#network:

  # routeTimeoutSeconds controls the number of seconds the controller will wait for a route attempt to succeed.
  #routeTimeoutSeconds:  10

  # createCircuitRetries controls the number of retries that will be attempted to create a path (and terminate it)
  # for new circuits.
  #createCircuitRetries: 2  

  # pendingLinkTimeoutSeconds controls how long we'll wait before creating a new link between routers where
  # there isn't an established link, but a link request has been sent
  #pendingLinkTimeoutSeconds: 10

  # Defines the period that the controller re-evaluates the performance of all of the circuits
  # running on the network.
  #
  #cycleSeconds:         15
  
  # Sets router minimum cost. Defaults to 10
  #minRouterCost: 10

  # Sets how often a new control channel connection can take over for a router with an existing control channel connection
  # Defaults to 1 minute
  #routerConnectChurnLimit: 1m

  # Sets the latency of link when it's first created. Will be overwritten as soon as latency from the link is actually
  # reported from the routers. Defaults to 65 seconds.
  #initialLinkLatency: 65s
  
  #smart:
    #
    # Defines the fractional upper limit of underperforming circuits that are candidates to be re-routed. If 
    # smart routing detects 100 circuits that are underperforming, and `smart.rerouteFraction` is set to `0.02`,
    # then the upper limit of circuits that will be re-routed in this `cycleSeconds` period will be limited to 
    # 2 (2% of 100). 
    #
    #rerouteFraction:    0.02
    # 
    # Defines the hard upper limit of underperforming circuits that are candidates to be re-routed. If smart 
    # routing detects 100 circuits that are underperforming, and `smart.rerouteCap` is set to `1`, and 
    # `smart.rerouteFraction` is set to `0.02`, then the upper limit of circuits that will be re-routed in this 
    # `cycleSeconds` period will be limited to 1.
    #
    #rerouteCap:         4  

# the endpoint that routers will connect to the controller over.
ctrl:
  #options:
  # (optional) settings
  # set the maximum number of connect requests that are buffered and waiting to be acknowledged (1 to 5000, default 1)
  #maxQueuedConnects:      1
  # the maximum number of connects that have  begun hello synchronization (1 to 1000, default 16)
  #maxOutstandingConnects: 16
  # the number of milliseconds to wait before a hello synchronization fails and closes the connection (30ms to 60000ms, default: 1000ms)
  #connectTimeoutMs:       1000
  listener:             tls:0.0.0.0:6262

# the endpoint that management tools connect to the controller over.
mgmt:
  #options:
  # (optional) settings
  # set the maximum number of connect requests that are buffered and waiting to be acknowledged (1 to 5000, default 1)
  #maxQueuedConnects:      1
  # the maximum number of connects that have  begun hello synchronization (1 to 1000, default 16)
  #maxOutstandingConnects: 16
  # the number of milliseconds to wait before a hello synchronization fails and closes the connection (30ms to 60000ms, default: 1000ms)
  #connectTimeoutMs:       1000
  listener:             tls:0.0.0.0:10000

#metrics:
#  influxdb:
#    url:                http://localhost:8086
#    database:           ziti

# xctrl_example
#
#example:
#  enabled:              false
#  delay:                5s

healthChecks:
  boltCheck:
    # How often to try entering a bolt read tx. Defaults to 30 seconds
    interval: 30s
    # When to time out the check. Defaults to 20 seconds
    timeout: 20s
    # How long to wait before starting the check. Defaults to 30 seconds
    initialDelay: 30s

# By having an 'edge' section defined, the ziti-controller will attempt to parse the edge configuration. Removing this
# section, commenting out, or altering the name of the section will cause the edge to not run.
edge:
  # This section represents the configuration of the Edge API that is served over HTTPS
  api:
    #(optional, default 90s) Alters how frequently heartbeat and last activity values are persisted
    # activityUpdateInterval: 90s
    #(optional, default 250) The number of API Sessions updated for last activity per transaction
    # activityUpdateBatchSize: 250
    # sessionTimeout - optional, default 30m
    # The number of minutes before an Edge API session will time out. Timeouts are reset by
    # API requests and connections that are maintained to Edge Routers
    sessionTimeout: 30m
    # address - required
    # The default address (host:port) to use for enrollment for the Client API. This value must match one of the addresses
    # defined in this Controller.WebListener.'s bindPoints.
    address: 144.24.195.21:8441
  # This section is used to define option that are used during enrollment of Edge Routers, Ziti Edge Identities.
  enrollment:
    # signingCert - required
    # A Ziti Identity configuration section that specifically makes use of the cert and key fields to define
    # a signing certificate from the PKI that the Ziti environment is using to sign certificates. The signingCert.cert
    # will be added to the /.well-known CA store that is used to bootstrap trust with the Ziti Controller.
    signingCert:
      cert: /home/ubuntu/.ziti/quickstart/instance-20220807-1808/pki/instance-20220807-1808-signing-intermediate/certs/instance-20220807-1808-signing-intermediate.cert
      key:  /home/ubuntu/.ziti/quickstart/instance-20220807-1808/pki/instance-20220807-1808-signing-intermediate/keys/instance-20220807-1808-signing-intermediate.key
    # edgeIdentity - optional
    # A section for identity enrollment specific settings
    edgeIdentity:
      # duration - optional, default 180m
      # The length of time that a Ziti Edge Identity enrollment should remain valid. After
      # this duration, the enrollment will expire and no longer be usable.
      duration: 180m
    # edgeRouter - Optional
    # A section for edge router enrollment specific settings.
    edgeRouter:
      # duration - optional, default 180m
      # The length of time that a Ziti Edge Router enrollment should remain valid. After
      # this duration, the enrollment will expire and no longer be usable.
      duration: 180m

# web
# Defines webListeners that will be hosted by the controller. Each webListener can host many APIs and be bound to many
# bind points.
web:
  # name - required
  # Provides a name for this listener, used for logging output. Not required to be unique, but is highly suggested.
  - name: client-management
    # bindPoints - required
    # One or more bind points are required. A bind point specifies an interface (interface:port string) that defines
    # where on the host machine the webListener will listen and the address (host:port) that should be used to
    # publicly address the webListener(i.e. mydomain.com, localhost, 127.0.0.1). This public address may be used for
    # incoming address resolution as well as used in responses in the API.
    bindPoints:
      #interface - required
      # A host:port string on which network interface to listen on. 0.0.0.0 will listen on all interfaces
      - interface: 0.0.0.0:8441
        # address - required
        # The public address that external incoming requests will be able to resolve. Used in request processing and
        # response content that requires full host:port/path addresses.
        address: 144.24.195.21:8441
    # identity - optional
    # Allows the webListener to have a specific identity instead of defaulting to the root 'identity' section.
    identity:
      ca:          "/home/ubuntu/.ziti/quickstart/instance-20220807-1808/pki/144.24.195.21-intermediate/certs/144.24.195.21-intermediate.cert"
      key:         "/home/ubuntu/.ziti/quickstart/instance-20220807-1808/pki/144.24.195.21-intermediate/keys/144.24.195.21-server.key"
      server_cert: "/home/ubuntu/.ziti/quickstart/instance-20220807-1808/pki/instance-20220807-1808-intermediate/certs/instance-20220807-1808-2022-08-07_162827-server.chain.pem"
      cert:        "/home/ubuntu/.ziti/quickstart/instance-20220807-1808/pki/144.24.195.21-intermediate/certs/144.24.195.21-client.cert"
    # options - optional
    # Allows the specification of webListener level options - mainly dealing with HTTP/TLS settings. These options are
    # used for all http servers started by the current webListener.
    options:
      # idleTimeoutMs - optional, default 5000ms
      # The maximum amount of idle time in milliseconds allowed for pipelined HTTP requests. Setting this too high
      # can cause resources on the host to be consumed as clients remain connected and idle. Lowering this value
      # will cause clients to reconnect on subsequent HTTPs requests.
      idleTimeout: 5000ms  #http timeouts, new
      # readTimeoutMs - optional, default 5000ms
      # The maximum amount of time in milliseconds http servers will wait to read the first incoming requests. A higher
      # value risks consuming resources on the host with clients that are acting bad faith or suffering from high latency
      # or packet loss. A lower value can risk losing connections to high latency/packet loss clients.
      readTimeout: 5000ms
      # writeTimeoutMs - optional, default 100000ms
      # The total maximum time in milliseconds that the http server will wait for a single requests to be received and
      # responded too. A higher value can allow long-running requests to consume resources on the host. A lower value
      # can risk ending requests before the server has a chance to respond.
      writeTimeout: 100000ms
      # minTLSVersion - optional, default TLS1.2
      # The minimum version of TSL to support
      minTLSVersion: TLS1.2
      # maxTLSVersion - optional, default TLS1.3
      # The maximum version of TSL to support
      maxTLSVersion: TLS1.3
    # apis - required
    # Allows one or more APIs to be bound to this webListener
    apis:
      # binding - required
      # Specifies an API to bind to this webListener. Built-in APIs are
      #   - edge-management
      #   - edge-client
      #   - fabric-management
      - binding: edge-management
        # options - arg optional/required
        # This section is used to define values that are specified by the API they are associated with.
        # These settings are per API. The example below is for the 'edge-api' and contains both optional values and
        # required values.
        options: { }
      - binding: edge-client
        options: { }
      - binding: fabric
        options: { }

If you need to dig into my setup, just send me your SSH public key, I’ll add that to the OCI

I’ll start with OCI and make a video from start to finish. Gimme a bit. I can see the cert output from openssl still has this in it:

X509v3 Subject Alternative Name:
                DNS:144.24.195.21, DNS:10.0.0.235

If you want me to use your controller - cool. I’m happy to do that :slight_smile: i’ll DM you now

1 Like

Ok. I got to the bottom of the issue. It comes down to a few things but mainly it’s because the command I provided to @markamind on his other post was ever so slightly incorrect and that caused a lot of confusion! :frowning: I’ll update that post after this one.

@arslane gave me access to his OCI instance saving me a bit of time making an OCI machine. I wanted to use it anyway, to save you the time and so you’ll have your own instance that’s configured correctly as reference. Here’s what I needed to do.

Check all the ports

There are four main ports needed to add a second router to a quickstart-enabled instance. The first two are always opened as part of the quickstart, the second two are only opened when you add a second router. These are those ports:

  • 8441 - the client/management API port used in the quickstarts
  • 8442 - the edge router port that accepts edge connections used in the quickstarts
  • 6262 - the controller’s control plane port - routers connect to the controller on this port
  • 10080 - the port the quickstart uses for the edge router’s link listener

First thing I did was to check that all these “server” ports all returned a certificate with DNS names properly setup. Port 6262 was still returning the IP in the DNS field. That was when I realized that the instructions I gave to @markamind were incomplete.

Create TWO new certs

The quickstart comes with three PKI’s setup for the controller/edge controller as it’s meant to be educational. There’s a PKI for the overall network - the ‘controller’ (port 6262). One for the edge controller which is not quite the same as the ‘controller’. This is the PKI for the APIs (port 8441). Both certs need to be fixed. To do that I sourced the .env file that comes with the quickstart:

. $HOME/.ziti/quickstart/$(hostname)/$(hostname).env

Then I setup the variables but notice I did the DNS-related ones differently. Also note that I have two different files called out at the bottom of this code block… one for each cert we’re about to make. It’s vital that you do not set the DNS related fields to an IP address. Here you can see I made up one, and used the hostname for the other:

now="$(date +'%Y-%m-%d_%H%M%S')"

EDGE_CONTROLLER_EXTERNAL_DNS_NAME=my.private.ziti.controller
EDGE_CONTROLLER_PRIVATE_DNS_NAME=$(hostname)
EDGE_CONTROLLER_EXTERNAL_IP_ADDRESS=144.24.195.21
EDGE_CONTROLLER_PRIVATE_IP_ADDRESS=10.0.0.235

pki_allow_list_dns="${EDGE_CONTROLLER_EXTERNAL_DNS_NAME},${EDGE_CONTROLLER_PRIVATE_DNS_NAME},localhost,$(hostname)"
pki_allow_list_ip="127.0.0.1,${EDGE_CONTROLLER_EXTERNAL_IP_ADDRESS},${EDGE_CONTROLLER_PRIVATE_IP_ADDRESS}"

new_ctrl_cert_name="${ZITI_CONTROLLER_HOSTNAME}-${now}"
new_edge_ctrl_cert_name="${ZITI_EDGE_CONTROLLER_HOSTNAME}-${now}"

generated a new server cert for the controller by running:

"${ZITI_BIN_DIR}/ziti" pki create server \
  --pki-root="${ZITI_PKI_OS_SPECIFIC}" \
  --ca-name ${ZITI_CONTROLLER_INTERMEDIATE_NAME} \
  --key-file "${ZITI_CONTROLLER_HOSTNAME}-server" \
  --server-file "${new_ctrl_cert_name}-server" \
  --dns "${pki_allow_list_dns}" --ip "${pki_allow_list_ip}" \
  --server-name "${new_ctrl_cert_name} server certificate"

generated a new server cert for the edge controller by running:

"${ZITI_BIN_DIR}/ziti" pki create server \
  --pki-root="${ZITI_PKI_OS_SPECIFIC}" \
  --ca-name ${ZITI_EDGE_CONTROLLER_INTERMEDIATE_NAME} \
  --key-file "${ZITI_EDGE_CONTROLLER_HOSTNAME}-server" \
  --server-file "${new_edge_ctrl_cert_name}-server" \
  --dns "${pki_allow_list_dns}" --ip "${pki_allow_list_ip}" \
  --server-name "${new_edge_ctrl_cert_name} server certificate"

Finally, we can find and output these two files:

cat <<HERE

    CONTROLLER server cert use:
        $(find $ZITI_HOME -name "*${new_ctrl_cert_name}*chain.pem")
    EDGE CONTROLLER server cert use:
        $(find $ZITI_HOME -name "*${new_edge_ctrl_cert_name}*chain.pem")

HERE

which looked like this:

    CONTROLLER server cert use:
        /home/ubuntu/.ziti/quickstart/instance-20220807-1808/pki/instance-20220807-1808-intermediate/certs/instance-20220807-1808-2022-08-07_190024-server.chain.pem
    EDGE CONTROLLER server cert use:
        /home/ubuntu/.ziti/quickstart/instance-20220807-1808/pki/144.24.195.21-intermediate/certs/144.24.195.21-2022-08-07_190024-server.chain.pem

Edit the config file

With those files in hand, now I could edit the controller’s config file:

vi $ZITI_HOME/$(hostname).yaml

The first thing I did was edit the identity.server_cert near the top of the file and replaced the server_cert with the NEW server cert. Then I found the edge controller server_cert location in the web.identity.server_cert section (near the bottom of the file) and I updated that value.

Finally I restarted the controller:

sudo systemctl restart ziti-controller

Making sure it works

I wanted to make sure @arslane would be in good shape, so I make a “clint-aws” edge router:

ziti edge create edge-router clint-aws -o clint-aws.jwt -t

I scp’ed the jwt from @arslane’s instance, and scp’ed it to my AWS router. Then I cd’ed to that $ZITI_HOME, copied my existing config file, changed the necessary settings, enrolled it, and tried to start that router. When I did that the first time, my edge router in AWS reported “link dialing failed”:

certificate is valid for 127.0.0.1, not 144.24.195.21] linkId=[4xwLnaMbpcofMRpOsYnaR6]} link dialing failed

There seems to be another bug with the quickstart though because the edge-router’s config had an IP address in the SANS section. It looked like this:

    sans:
      dns:
        - 144.24.195.21
        - localhost
      ip:
        - "127.0.0.1"

I updated that section to use the hostname in the dns section and to use the external ip in the ip section. It now looks like this:

    sans:
      dns:
        - "instance-20220807-1808"
        - "localhost"
      ip:
        - "127.0.0.1"
        - "144.24.195.21"

The problem is that the certificates for routers are created during the enrollment process. That meant I had to delete the existing edge-router, and recreate it. Using the ziti-cli-functions.sh as a guide, I did that using the following commands:

ZITI_EDGE_ROUTER_RAWNAME="${ZITI_NETWORK}-edge-router"
"${ZITI_BIN_DIR-}/ziti" edge delete edge-router "${ZITI_EDGE_ROUTER_RAWNAME}"
"${ZITI_BIN_DIR-}/ziti" edge create edge-router "${ZITI_EDGE_ROUTER_RAWNAME}" -o "${ZITI_HOME_OS_SPECIFIC}/${ZITI_EDGE_ROUTER_RAWNAME}.jwt" -t -a "public"
"${ZITI_BIN_DIR-}/ziti-router" enroll "${ZITI_HOME_OS_SPECIFIC}/${ZITI_EDGE_ROUTER_RAWNAME}.yaml" --jwt "${ZITI_HOME_OS_SPECIFIC}/${ZITI_EDGE_ROUTER_RAWNAME}.jwt"
sudo systemctl start ziti-router

I verified the ‘public’ edge router started (Arslane’s edge router, it did start) and then went back to my AWS router and started that router. This time it connected to Arslane’s public edge-router. I could verify that by looking at the links formed:

ziti fabric list links
╭────────────────────────┬───────────┬────────────────────────────────────┬─────────────┬─────────────┬─────────────┬───────────┬────────┬───────────╮
│ ID                     │ DIALER    │ ACCEPTOR                           │ STATIC COST │ SRC LATENCY │ DST LATENCY │ STATE     │ STATUS │ FULL COST │
├────────────────────────┼───────────┼────────────────────────────────────┼─────────────┼─────────────┼─────────────┼───────────┼────────┼───────────┤
│ 7hx4qaAE1UlL4iPIH51Phg │ clint-aws │ instance-20220807-1808-edge-router │           1 │   65000.0ms │   65000.0ms │ Connected │     up │    130001 │
╰────────────────────────┴───────────┴────────────────────────────────────┴─────────────┴─────────────┴─────────────┴───────────┴────────┴───────────╯
results: 1-1 of 1

That was a lot more effort than I thought, and would be VERY hard for someone not as familiar with X509/the PKI setup of OpenZiti to figure out!!!

Thanks @arslane for letting me use your OCI instance. I look forward to your follow-up. :slight_smile:

2 Likes

Here’s a video taking you from nothing in OCI through the express install, through the addition of a second, private edge router. It’s basically the same steps as above, just cleaner, and narrated…

2 Likes

Thanks @TheLumberjack .. this is something that I have been trying to do for some time now.. and will be working on this today.. so this is great timing.

Something I also noticed is that you created the jwt file on the controller.. and then copied to the remote router.

My understanding is that this is because you have only allowed local host access to the controller.

Nah it's just because I'm lazy and when I use a quickstart-installed system I'll just ssh to that machine because it'll have zitiLogin on it. I absolutely could have ssh'ed onto the edge-router I was installing, used ziti login to login to my controller from there, and created the .jwt on the router. Then I wouldn't have to scp the .jwt around.

1 Like

Is there anyway you can restrict the Ziti Login from a remote machine? Curious to know.

Yes, we covered that on a separate discourse post about a month or two back here Making ZAC dark - #6 by dovholuknf

You need to “split” the api. That’s done in the controller. The relevant changelog entry talks about it too (also in that thread) Making ZAC dark - #3 by dovholuknf

1 Like

And for help - here’s what my current ‘web’ config looks like on my aws instance where i have the fabric and management apis split from the edge-client…

web:
  - name: dark-api
    bindPoints:
      - interface: 0.0.0.0:18441
        address: localhost:18441
    identity:
      ca:          "/home/ubuntu/.ziti/quickstart/ip-172-31-42-64/pki/ec2-18-188-201-183.us-east-2.compute.amazonaws.com-intermediate/certs/ec2-18-188-201-183.us-east-2.compute.amazonaws.com-intermediate.cert"
      key:         "/home/ubuntu/.ziti/quickstart/ip-172-31-42-64/pki/ec2-18-188-201-183.us-east-2.compute.amazonaws.com-intermediate/keys/ec2-18-188-201-183.us-east-2.compute.amazonaws.com-server.key"
      server_cert: "/home/ubuntu/.ziti/quickstart/ip-172-31-42-64/pki/ec2-18-188-201-183.us-east-2.compute.amazonaws.com-intermediate/certs/ec2-18-188-201-183.us-east-2.compute.amazonaws.com-server-2022-06-10-3915.chain.pem"
      _server_cert: "/home/ubuntu/.ziti/quickstart/ip-172-31-42-64/pki/ec2-18-188-201-183.us-east-2.compute.amazonaws.com-intermediate/certs/ec2-18-188-201-183.us-east-2.compute.amazonaws.com-server.chain.pem"
      cert:        "/home/ubuntu/.ziti/quickstart/ip-172-31-42-64/pki/ec2-18-188-201-183.us-east-2.compute.amazonaws.com-intermediate/certs/ec2-18-188-201-183.us-east-2.compute.amazonaws.com-client.cert"
    apis:
      - binding: edge-management
        options: { }
      - binding: fabric
        options: { }
  - name: client-management
    bindPoints:
      - interface: 0.0.0.0:8441
        address: ec2-18-188-201-183.us-east-2.compute.amazonaws.com:8441
    identity:
      ca:          "/home/ubuntu/.ziti/quickstart/ip-172-31-42-64/pki/ec2-18-188-201-183.us-east-2.compute.amazonaws.com-intermediate/certs/ec2-18-188-201-183.us-east-2.compute.amazonaws.com-intermediate.cert"
      key:         "/home/ubuntu/.ziti/quickstart/ip-172-31-42-64/pki/ec2-18-188-201-183.us-east-2.compute.amazonaws.com-intermediate/keys/ec2-18-188-201-183.us-east-2.compute.amazonaws.com-server.key"
      server_cert: "/home/ubuntu/.ziti/quickstart/ip-172-31-42-64/pki/ec2-18-188-201-183.us-east-2.compute.amazonaws.com-intermediate/certs/ec2-18-188-201-183.us-east-2.compute.amazonaws.com-server-2022-06-10-3915.chain.pem"
      _server_cert: "/home/ubuntu/.ziti/quickstart/ip-172-31-42-64/pki/ec2-18-188-201-183.us-east-2.compute.amazonaws.com-intermediate/certs/ec2-18-188-201-183.us-east-2.compute.amazonaws.com-server.chain.pem"
      cert:        "/home/ubuntu/.ziti/quickstart/ip-172-31-42-64/pki/ec2-18-188-201-183.us-east-2.compute.amazonaws.com-intermediate/certs/ec2-18-188-201-183.us-east-2.compute.amazonaws.com-client.cert"
    options:
      idleTimeout: 5000ms  #http timeouts, new
      readTimeout: 5000ms
      writeTimeout: 100000ms
      minTLSVersion: TLS1.2
      maxTLSVersion: TLS1.3
    apis:
      - binding: edge-client
        options: { }
1 Like

i do NOT have port 18441 exposed through the AWS ACL (security group) so you can only access the api from the local machine…

Oh - and if you DO do that, you’ll need to fix your zitiLogin alias too since it’ll only work on “localhost” now… instead of:

alias zitiLogin='ziti edge login "${ZITI_EDGE_CTRL_ADVERTISED}"/edge/management/v1/ -u "${ZITI_USER-}" -p "${ZITI_PWD}" -c "${ZITI_PKI}/${ZITI_EDGE_CONTROLLER_INTERMEDIATE_NAME}/certs/${ZITI_EDGE_CONTROLLER_INTERMEDIATE_NAME}.cert"'

you would use:

alias zitiLogin='ziti edge login localhost:18441/edge/management/v1/ -u "${ZITI_USER-}" -p "${ZITI_PWD}" -c "${ZITI_PKI}/${ZITI_EDGE_CONTROLLER_INTERMEDIATE_NAME}/certs/${ZITI_EDGE_CONTROLLER_INTERMEDIATE_NAME}.cert"'
1 Like