Missing or Invalid CSR

Support General Questions
1

Hi all, I have been trying to setup OpenZiti for a couple of weeks and tried on 3 servers and I get the same error every time I try to add a Edge Tunnel.

So this is my environment:
I have followed:

With 2 exceptions, I have setup a standard user called zitiadmin and run the installation steps as that user, the services are set to run as this user as well.
Before the installation I set local variables (PATH) for installation folder and set that to /opt/openziti and I also set the External_DNS to our public FQDN.

I also follow the guide to install the WebUI which is all working.
I have also followed the guides to get the Edge Tuennel Service installed on a Ubuntu server and opened the required firewall ports, I also dont see any blocks in the firewall logs.

So I use the WebUI to create a new Router which generates and JWT file. I copy the file to the Edge Tunnel server and restart the services and I get this error:

Apr 30 08:09:17 t1s-mms-tun01 ziti-edge-tunnel.sh[3879098]: (3879098)[ 0.000] INFO ziti-sdk:utils.c:200 ziti_log_set_level() set log level: root=3/INFO
Apr 30 08:09:17 t1s-mms-tun01 ziti-edge-tunnel.sh[3879098]: (3879098)[ 0.000] INFO ziti-sdk:utils.c:169 ziti_log_init() Ziti C SDK version 0.35.12 @5acfb13(HEAD) starting at (202>
Apr 30 08:09:17 t1s-mms-tun01 ziti-edge-tunnel.sh[3879098]: (3879098)[ 0.000] INFO ziti-sdk:ziti_enroll.c:88 ziti_enroll() Ziti C SDK version 0.35.12 @5acfb13(HEAD) starting enro>
Apr 30 08:09:18 t1s-mms-tun01 ziti-edge-tunnel.sh[3879098]: (3879098)[ 0.113] WARN ziti-sdk:ziti_ctrl.c:89 code_to_error() unmapped error code: MISSING_OR_INVALID_CSR
Apr 30 08:09:18 t1s-mms-tun01 ziti-edge-tunnel.sh[3879098]: (3879098)[ 0.113] ERROR ziti-sdk:ziti_enroll.c:233 enroll_cb() failed to enroll with controller: https://T1S-MGS-ZTG01:>
Apr 30 08:09:18 t1s-mms-tun01 ziti-edge-tunnel.sh[3879098]: (3879098)[ 0.113] ERROR ziti-edge-tunnel:ziti-edge-tunnel.c:2141 enroll_cb() enrollment failed: MISSING_OR_INVALID_CSR(>
Apr 30 08:09:18 t1s-mms-tun01 ziti-edge-tunnel.sh[3879094]: ERROR: failed to enroll MMS-Internal.jwt in /opt/openziti/etc/identities
Apr 30 08:09:18 t1s-mms-tun01 systemd[1]: ziti-edge-tunnel.service: Control process exited, code=exited, status=1/FAILURE

Can anyone help me with this error?
Many Thanks.

2

Hi @mphayesuk, welcome to the community and to OpenZiti! I'm sorry you're having these troubles. The good news though is that this seems to be reproducible if you've done it three times and get the same issue.

This appears to be a ziti-edge-tunnel issue, can you provide a detailed set of instructions to reproduce the problem?

Can you let us know what OS you're running ziti-edge-tunnel on too and how you run it?

I admit, I don't think I've seen this issue before. I'm going to scan the forum for anything similar. Let us know about the os and steps to reproduce, thanks.

3

One thing I just noticed is this error:

failed to enroll with controller: https://T1S-MGS-ZTG01:>

See how there's no port and it ends with a colon? This makes me think there's either a logging bug in the tunneler or there is a configuration issue with your controller, not advertising a port.

Can you also look into the controller and look at the config file to check the "address" entries in that file? Let's verify they're correct

4

Hi @TheLumberjack, thanks for your quick response:

Am I looking in: T1S-MGS-ZTG01.yaml

And this section:
ctrl:
options:
advertiseAddress: tls:T1S-MGS-ZTG01:6262

(optional) settings

set the maximum number of connect requests that are buffered and waiting to be acknowledged (1 to 5000, default 1)

#maxQueuedConnects: 1

the maximum number of connects that have begun hello synchronization (1 to 1000, default 16)

#maxOutstandingConnects: 16

the number of milliseconds to wait before a hello synchronization fails and closes the connection (30ms to 60000ms, default: 5000ms)

#connectTimeoutMs: 5000
listener: tls:0.0.0.0:6262

5

There's another address in there in the Web section too. Check that too please?

6

Just in case its important:
https://t1s-mgs-ztg01:6262/
Gives me an error of:

This site can’t provide a secure connection

7

web:

name - required

Provides a name for this listener, used for logging output. Not required to be unique, but is highly suggested.

  • name: client-management

    bindPoints - required

    One or more bind points are required. A bind point specifies an interface (interface:port string) that defines

    where on the host machine the webListener will listen and the address (host:port) that should be used to

    publicly address the webListener(i.e. mydomain.com, localhost, 127.0.0.1). This public address may be used for

    incoming address resolution as well as used in responses in the API.

    bindPoints:
    #interface - required

    A host:port string on which network interface to listen on. 0.0.0.0 will listen on all interfaces

    • interface: 0.0.0.0:1280

      address - required

      The public address that external incoming requests will be able to resolve. Used in request processing and

      response content that requires full host:port/path addresses.

      address: T1S-MGS-ZTG01:1280

    identity - optional

    Allows the webListener to have a specific identity instead of defaulting to the root 'identity' section.

    identity:
    ca: "/opt/openziti/T1SMGSZTG01/pki/T1S-MGS-ZTG01-edge-controller-root-ca/certs/T1S-MGS-ZTG01-edge-controller-root-ca.cert"
    key: "/opt/openziti/T1SMGSZTG01/pki/T1S-MGS-ZTG01-edge-controller-intermediate/keys/T1S-MGS-ZTG01-server.key"
    server_cert: "/opt/openziti/T1SMGSZTG01/pki/T1S-MGS-ZTG01-edge-controller-intermediate/certs/T1S-MGS-ZTG01-server.chain.pem"
    cert: "/opt/openziti/T1SMGSZTG01/pki/T1S-MGS-ZTG01-edge-controller-intermediate/certs/T1S-MGS-ZTG01-client.cert"
8

Ok thanks. There's a port there, so the config looks ok... If you give me those steps to reproduce, I'll try in a bit.

9

For https://t1s-mgs-ztg01/
ERR_CONNECTION_REFUSED

10

Also, can you use openssl s_client to conntect to port 1280 and 6262 from the tunneler machine?

openssl s_client -connect t1s-mgs-ztg01:1280 </dev/null
openssl s_client -connect t1s-mgs-ztg01:6262 </dev/null

both of those return certificate information?

11

Ok, so 6262 was not open on the firewall, even though that never appeared in the logs as a blocked port (but thats my issue to solve)

Now I get a different error:
Apr 30 10:23:11 t1s-mms-tun01 ziti-edge-tunnel.sh[3915701]: (3915701)[ 0.000] INFO ziti-sdk:utils.c:200 ziti_log_set_level() set log level: root=3/INFO
Apr 30 10:23:11 t1s-mms-tun01 ziti-edge-tunnel.sh[3915701]: (3915701)[ 0.000] INFO ziti-sdk:utils.c:169 ziti_log_init() Ziti C SDK version 0.35.12 @5acfb13(HEAD) starting at (2024-04-30T10:23:11.915)
Apr 30 10:23:11 t1s-mms-tun01 ziti-edge-tunnel.sh[3915701]: (3915701)[ 0.000] ERROR ziti-edge-tunnel:ziti-edge-tunnel.c:2196 enroll() failed to open file /opt/openziti/etc/identities/MMS-Internal.json: File exists(17)
Apr 30 10:23:11 t1s-mms-tun01 ziti-edge-tunnel.sh[3915697]: ERROR: failed to enroll MMS-Internal.jwt in /opt/openziti/etc/identities
Apr 30 10:23:11 t1s-mms-tun01 systemd[1]: ziti-edge-tunnel.service: Control process exited, code=exited, status=1/FAILURE

12

root@t1s-mms-tun01:/opt/openziti/etc/identities# ls -lah
total 12K
drwxrwx--- 2 root ziti 4.0K Apr 30 10:23 .
drwxr-xr-x 3 root root 4.0K Jan 16 10:56 ..
-rw------- 1 ziti ziti 0 Apr 30 10:23 MMS-Internal.json
-rw-rw---- 1 root ziti 994 Apr 30 08:09 MMS-Internal.jwt

13

The initialization script for ziti-edge-tunnel erroneously left an empty .json file when the previous enrollment attempt(s) failed. Can you remove the json file and restart ziti-edge-tunnel?

14

Thanks, that fixed that error but now we go back to the same error as before:

Apr 30 10:29:01 t1s-mms-tun01 ziti-edge-tunnel.sh[3916934]: (3916934)[ 0.000] INFO ziti-sdk:utils.c:200 ziti_log_set_level() set log level: root=3/INFO
Apr 30 10:29:01 t1s-mms-tun01 ziti-edge-tunnel.sh[3916934]: (3916934)[ 0.000] INFO ziti-sdk:utils.c:169 ziti_log_init() Ziti C SDK version 0.35.12 @5acfb13(HEAD) starting at (2024-04-30T10:29:01.416)
Apr 30 10:29:01 t1s-mms-tun01 ziti-edge-tunnel.sh[3916934]: (3916934)[ 0.000] INFO ziti-sdk:ziti_enroll.c:88 ziti_enroll() Ziti C SDK version 0.35.12 @5acfb13(HEAD) starting enrollment at (2024-04-30T10:29:01.416)
Apr 30 10:29:01 t1s-mms-tun01 ziti-edge-tunnel.sh[3916934]: (3916934)[ 0.112] WARN ziti-sdk:ziti_ctrl.c:89 code_to_error() unmapped error code: MISSING_OR_INVALID_CSR
Apr 30 10:29:01 t1s-mms-tun01 ziti-edge-tunnel.sh[3916934]: (3916934)[ 0.112] ERROR ziti-sdk:ziti_enroll.c:233 enroll_cb() failed to enroll with controller: https://T1S-MGS-ZTG01:1280 MISSING_OR_INVALID_CSR (The supplied enroll>
Apr 30 10:29:01 t1s-mms-tun01 ziti-edge-tunnel.sh[3916934]: (3916934)[ 0.112] ERROR ziti-edge-tunnel:ziti-edge-tunnel.c:2141 enroll_cb() enrollment failed: MISSING_OR_INVALID_CSR(-3)
Apr 30 10:29:01 t1s-mms-tun01 ziti-edge-tunnel.sh[3916923]: ERROR: failed to enroll MMS-Internal.jwt in /opt/openziti/etc/identities
Apr 30 10:29:01 t1s-mms-tun01 systemd[1]: ziti-edge-tunnel.service: Control process exited, code=exited, status=1/FAILURE

15

I bet the identity is burned. I bet it enrolled/consumed the .jwt but now that .jwt is not usable. Could you try removing the .jwt/.json files, make a new identity and see if that one will succeed?

In the UI, i would expect to see the identity and no "download" button. That would indicate the identity consumed the jwt, but the failure of port 6262 being closed, prevented the identity from being written...

16

I have removed the old Identity and created a new one and copied the file over, but still the same error message.

17

Build Debian 11 from Netinstall ISO

Standard tools with SSH server

Make sure you have tar , hostname , jq and curl installed before running the expressInstall one-liner.

Tar is gzip and is already installed,

As root

apt-get install jq curl git mlocate sudo

apt remove --assume-yes --purge apparmor

reboot

useradd -m zitiadmin -s /bin/bash

usermod -aG sudo zitiadmin

mkdir /opt/openziti

chown zitiadmin:zitiadmin /opt/openziti

// Run the below nvm install as zitiadmin

curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.3/install.sh | bash

source ~/.bashrc

nvm list-remote

nvm install v20.12.2
npm install -g @angular/cli@16

// Add the External_DNS name to /etc/hosts file
i.e.
10.10.5.252 xxx.xxx.co.uk

Set the below in home .bashrc files

Only set the External DNS and Home settings, the rest of them can be sorted out when you source the .env file in the openziti installation folder

****************** USE THESE ************************************

export EXTERNAL_DNS="xxx.xxx.co.uk"

export ZITI_HOME=”/opt/openziti/T1SMGSZTG01”

source /dev/stdin <<< "$(wget -qO- https://get.openziti.io/ziti-cli-functions.sh)"; expressInstall

As root run:

source /opt/openziti/T1SMGSZTG01/T1S-MGS-ZTG01.env

source /dev/stdin <<< "$(wget -qO- https://get.openziti.io/ziti-cli-functions.sh)"; createControllerSystemdFile "${ZITI_CTRL_NAME}"

source /dev/stdin <<< "$(wget -qO- https://get.openziti.io/ziti-cli-functions.sh)"; createRouterSystemdFile "${ZITI_ROUTER_NAME}"

cd /opt/openziti/T1SMGSZTG01/

cp T1S-MGS-ZTG01.service /etc/systemd/system
cp T1S-MGS-ZTG01-edge-router.service /etc/systemd/system

cd /etc/systemd/system

#update the user to zitiadmin

nano T1S-MGS-ZTG01-edge-router.service
nano T1S-MGS-ZTG01-edge-router.service

systemctl daemon-reload
systemctl enable --now T1S-MGS-ZTG01.service
systemctl enable --now T1S-MGS-ZTG01-edge-router.service

Install the Console Web UI

As zitiadmin run the following:

source /opt/openziti/T1SMGSZTG01/T1S-MGS-ZTG01.env

git clone GitHub - openziti/ziti-console "${ZITI_HOME}/ziti-console"

cd "${ZITI_HOME}/ziti-console"

npm install

ng build ziti-console-lib // select n twice

ng build ziti-console-node

As root run:

source /opt/openziti/T1SMGSZTG01/T1S-MGS-ZTG01.env

ln -s "${ZITI_PKI}/${ZITI_CTRL_EDGE_NAME}-intermediate/certs/${ZITI_CTRL_EDGE_ADVERTISED_ADDRESS}-server.chain.pem" "${ZITI_HOME}/ziti-console/server.chain.pem"

ln -s "${ZITI_PKI}/${ZITI_CTRL_EDGE_NAME}-intermediate/keys/${ZITI_CTRL_EDGE_ADVERTISED_ADDRESS}-server.key" "${ZITI_HOME}/ziti-console/server.key"

Create the service and change the run as user to zitiadmin

source /dev/stdin <<< "$(wget -qO- https://get.openziti.io/ziti-cli-functions.sh)"
createZacSystemdFile

cp "${ZITI_HOME}/ziti-console.service" /etc/systemd/system

systemctl daemon-reload
systemctl enable --now ziti-consolesystemctl status ziti-console --lines=0 --no-pager

*** Double check all folder permissions to make sure zitiadmin is the owner

https://${ZITI_CTRL_EDGE_ADVERTISED_ADDRESS}:8443

Tunnel Agent Setup

Create user ziti and group ziti and add to sudo and run the below commands as ziti user

useradd -m ziti -s /bin/bash

groupadd ziti

usermod -aG sudo ziti
usermod -aG ziti ziti

curl -sSLf https://get.openziti.io/tun/scripts/install-ubuntu.bash | bash

systemctl enable --now ziti-edge-tunnel.service

ziti-edge-tunnel add --jwt "$(< ./in-file.jwt)" --identity myIdentityName

Log on to the controller

Create a new Edge Router – the only entry was the “Name”

Load Identities Directory

The tunneller will load all enrolled identities in the --identity-dir directory at startup. The default location for identities is is /opt/openziti/etc/identities.

Add enrolled identity files to this directory by copying the JSON file into the directory and setting permissions for group ziti.

chown -cR :ziti /opt/openziti/etc/identities

chmod -cR ug=rwX,o-rwx /opt/openziti/etc/identities

package users can restart with systemd

systemctl restart ziti-edge-tunnel.service

18

Controller is on:
root@T1S-MGS-ZTG01:/opt/openziti/T1SMGSZTG01# lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description: Debian GNU/Linux 11 (bullseye)
Release: 11
Codename: bullseye

Edge server I am trying to enroll is on:
root@t1s-mms-tun01:/opt/openziti/etc/identities# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 20.04.4 LTS
Release: 20.04
Codename: focal

19

From this I'm assuming you are copying the JWT into the identities directory on the host that's running ziti-edge-tunnel? Are you also doing the ziti-edge-tunnel add command that you showed in your setup steps?

Only one of these two actions is necessary. If you're doing both, I suspect the second step is failing (because the JWT has already been used), which would explain the empty json file.

20

Hi, no I just copy the file and restart the service.