How can I create a service for ssh and add identities with the least amount of policies/configs/etc?

1

I am ready to start dogfooding OpenZiti on my own network. I would start by adding basic services for my own systems to access over the fabric. I would like to start with SSH.

I am going through the process of building the service. I can add the host.v1 config, but when I get to the intercept.v1 config it asks for a hostname. This is the part I am not sure how to proceed. There is probably a primitive I should be using.

Scenario

  • Let’s say I have 3 identities, linux, mac and windows (because I like pain)
  • I will ‘tag’ them with:
    • jptech is identifying them with my tenant (I am testing pseudo multi-tenant). I
    • admin is for my identities that will have admin rights
    • ssh is for any device that should have ssh accessible over the fabric.
  • Each of these identities will host ssh on port 22
  • Each of these identities will be on some disparate network
  • What I expect to happen is to…
    • ssh jp@linux.jptech.ziti from any host
    • ssh jp@mac.jptech.ziti from any host
    • ssh jp@win.jptech.ziti from any host.

What I am looking for is the minimum number of primitives that need to be configured to allow this to expand? I would like to simply add these tags to a new identity to add it the ssh tag to make it available to the admin tagged identities. That is the first iteration. The second iteration would be adding the tenant tag.

1 Like
2

Hello, this is a great question!

Here’s a setup that I think comes really close to your request. The one major difference between what I’ll describe here and your request is that the mechanism for access control with this setup will use service policies and role attributes rather than tags.

  • Identities with the “ssh” role will make their sshd server available
  • Identities with the “admin” role will be able to initiate connections to the ssh service

You mentioned the possibility of multitenancy, and I’m assuming other tenants might have an ssh service that is accessed by “admins”. FYI my personal convention is to name these roles so that it’s easy to tell at a glance which service and context the role specifies; e.g. “jptech.ssh.clients”, and “jptech.ssh.servers”. This is just my convention though. For this post I’ll stick with “ssh” and “admin” (and a separate “jptech” role that signifies the tenant).

By the way, all of what follows assumes that you will not restrict your tenants or services to specific edge routers. In other words, all of your edge routers will carry connections for any tenant and any service. So for this to work you’ll need open router policies:

ziti edge create service-edge-router-policy all --service-roles '#all' --edge-router-roles '#all'
ziti edge create edge-router-policy all --edge-router-roles '#all'  --identity-roles '#all'

You could deploy routers that will only be used by specific tenants and/or services. In that case you’d define your router policies accordingly, and the service policies that I show below could omit the “tenant” role. That last sentence might not make sense until you’ve seen the service policies, so read on if it isn’t

The service configuration is pretty easy, so let’s get that out of the way first. We’ll intercept the *.jptech.ziti" domain and the terminating side of the connection will always connect to 127.0.0.1:

ziti edge create config jptech.ssh.cfg.intercept intercept.v1 '{
    "addresses": ["*.jptech.ziti"],
    "protocols": ["tcp"],
    "portRanges": [ {"low":22,"high":22} ],
    "dialOptions": { "identity": "$dst_hostname" }
}'

ziti edge create config jptech.ssh.cfg.host host.v1 '{
    "address": "127.0.0.1",
    "protocol": "tcp",
    "port": 22,
    "listenOptions": { "identity": "$tunneler_id.name" }
}'

These configurations assume the identity names will match the intercept address wildcard.

We pull those configurations together into a service that specifies the roles that may access it:

ziti edge create service jptech.ssh \
    --configs jptech.ssh.cfg.intercept,jptech.ssh.cfg.host \
    --role-attributes admin,ssh

So the “jptech.ssh” service may be accessed by identities that have either or both of the role attributes “admin” and/or “ssh”. I’ll explain how to assign an identity to a role shortly.

First we need the service policies that give the role attributes their meaning. This is where the dial or bind (connect or serve) permission is mapped to the roles (there is a separate policy for dial/connect and bind/serve)

ziti edge create service-policy jptech.ssh.dial Dial --identity-roles "#jptech,#admin" --service-roles "@jptech.ssh"

ziti edge create service-policy jptech.ssh.bind Bind --identity-roles "#jptech,#ssh" --service-roles "@jptech.ssh"

These policies specify that identities with roles “jptech” AND “admin” may dial (a.k.a connect to) the “jptech.ssh” service, and identities with roles “jptech” AND “ssh” may bind (a.k.a. host) the “jptech.ssh” service. By the way, the comma-separated roles are AND’ed together by default. You could OR a list of roles together by passing --semantic AnyOf when creating a service policy.

Finally we can create the identities with role assignments as appropriate. For demonstration, I’ll create one identity that has “admin” access and one that does not. I’ll also create one that is a “jptech” tenant with no ssh access of any kind (probably wouldn’t want that, but just to illustrate the policies), and one identity that isn’t a “jptech” tenant at all:

ziti edge create identity device linux.jptech.ziti --role-attributes jptech,ssh,admin -o linux.jptech.ziti.jwt

ziti edge create identity device mac1.jptech.ziti --role-attributes jptech,admin -o mac1.jptech.ziti.jwt

ziti edge create identity device mac2.jptech.ziti --role-attributes jptech,ssh -o mac2.jptech.ziti.jwt

ziti edge create identity device win.jptech.ziti --role-attributes jptech -o win.jptech.ziti.jwt

ziti edge create identity device ios.notech.ziti --role-attributes notech,ssh,admin -o ios.notech.ziti.jwt

Note that it is possible to update an identities role attributes after it has been created.

It should look like this after the identities are enrolled:

ziti edge list identities
╭────────────┬───────────────────┬────────┬──────────────────┬─────────────╮
│ ID         │ NAME              │ TYPE   │ ATTRIBUTES       │ AUTH-POLICY │
├────────────┼───────────────────┼────────┼──────────────────┼─────────────┤
│ FgtJ-2309K │ mac1.jptech.ziti  │ Device │ admin,jptech     │ Default     │
│ Wul7VL0y8  │ Default Admin     │ User   │                  │ Default     │
│ ZSd4I2mXg  │ win.jptech.ziti   │ Device │ jptech           │ Default     │
│ gZdhILmX9K │ mac2.jptech.ziti  │ Device │ jptech,ssh       │ Default     │
│ jKExIL309K │ ios.notech.ziti   │ Device │ admin,notech,ssh │ Default     │
│ oqIQIL3XgK │ linux.jptech.ziti │ Device │ admin,jptech,ssh │ Default     │
╰────────────┴───────────────────┴────────┴──────────────────┴─────────────╯

And the policy-advisor shows us who can access what:

ziti edge policy-advisor identities
...
Output format: STATUS: ID (ID ROUTERS) -> SVC (SVC ROUTERS) Common Routers: (ONLINE COMMON ROUTERS/COMMON ROUTERS) Dial: DIAL_OK Bind: BIND_OK. ERROR_LIST
-------------------------------------------------------------------------------
OKAY : mac1.jptech.ziti (1) -> jptech.ssh (1) Common Routers: (1/1) Dial: Y Bind: N 

ERROR: Default Admin 
  - Identity does not have access to any services. Adjust service policies.

ERROR: win.jptech.ziti 
  - Identity does not have access to any services. Adjust service policies.

OKAY : mac2.jptech.ziti (1) -> jptech.ssh (1) Common Routers: (1/1) Dial: N Bind: Y 

ERROR: ios.notech.ziti 
  - Identity does not have access to any services. Adjust service policies.

OKAY : linux.jptech.ziti (1) -> jptech.ssh (1) Common Routers: (1/1) Dial: Y Bind: Y
2 Likes
3

Hello,

Just a heads up to let you know that I noticed a copy/pasta mistake in my previous post. The intercept configuration’s dial options identity was specified as “$tunneler_id.name”, but it should be “$dst_hostname” so that it matches the targeted hostname that the ssh client will look up. I have edited the post with this detail, but wanted to make sure you noticed the change incase you were in the process of trying things out.

1 Like
4

I am spinning up a new environment this week to test it. Thanks for including the commands, that will save a ton of clicking and understanding what is happening behind the clicking. :slight_smile:

5

I followed your steps, and I have most everything set but I get an access denied on some. I have also changed the attributes and it seems to randomly stop working from device to device. For instance, mac1 will work, mac2 wont; then I remove the admin role from macbian (my test host) and confirm it can no longer access mac1; then I add the admin attribute again and macbian can not ssh anymore.

One connects fine, the other three do not.

 $ ssh -v jp@win.jptech.ziti
OpenSSH_9.2p1 Debian-2, OpenSSL 3.0.9 30 May 2023
debug1: Reading configuration data /home/jp/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug1: Connecting to win.jptech.ziti [100.64.0.6] port 22.
debug1: connect to address 100.64.0.6 port 22: Connection refused
ssh: connect to host win.jptech.ziti port 22: Connection refused

Here is my output of the configuration.

ziti@2e61e0442f82:/persistent$ ziti edge policy-advisor identities | grep OKAY
  STATUS = The status of the identity -> service reachability. Will be OKAY or ERROR.
OKAY : mac1.jptech.ziti (1) -> hello_service (1) Common Routers: (1/1) Dial: Y Bind: Y
OKAY : mac1.jptech.ziti (1) -> jptech.ssh (1) Common Routers: (1/1) Dial: Y Bind: N
OKAY : macbian (1) -> hello_service (1) Common Routers: (1/1) Dial: Y Bind: Y
OKAY : macbian (1) -> jptech.ssh (1) Common Routers: (1/1) Dial: Y Bind: Y
OKAY : helloworld.jptech.corp (1) -> hello_service (1) Common Routers: (1/1) Dial: N Bind: Y
OKAY : falkor (1) -> hello_service (1) Common Routers: (1/1) Dial: Y Bind: Y
OKAY : mac2.jptech.ziti (1) -> jptech.ssh (1) Common Routers: (1/1) Dial: N Bind: Y
OKAY : win.jptech.ziti (1) -> hello_service (1) Common Routers: (1/1) Dial: Y Bind: Y
OKAY : win.jptech.ziti (1) -> jptech.ssh (1) Common Routers: (1/1) Dial: Y Bind: Y
OKAY : ios.jptech.ziti (1) -> hello_service (1) Common Routers: (1/1) Dial: Y Bind: Y
OKAY : linux.jptech.ziti (1) -> hello_service (1) Common Routers: (1/1) Dial: Y Bind: Y
OKAY : linux.jptech.ziti (1) -> jptech.ssh (1) Common Routers: (1/1) Dial: Y Bind: Y

ziti@2e61e0442f82:/persistent$ ziti edge list identities
╭────────────┬────────────────────────┬────────┬──────────────────┬─────────────╮
│ ID         │ NAME                   │ TYPE   │ ATTRIBUTES       │ AUTH-POLICY │
├────────────┼────────────────────────┼────────┼──────────────────┼─────────────┤
│ 1DDBNiwGV  │ mac1.jptech.ziti       │ Device │ admin,jptech,ssh │ Default     │
│ 6bNWqyt0o  │ macbian                │ Device │ admin,jptech,ssh │ Default     │
│ 7ARsJMakq  │ Default Admin          │ User   │                  │ Default     │
│ SgUAZ7mDo  │ helloworld.jptech.corp │ Device │                  │ Default     │
│ VI9UjJt0S  │ falkor                 │ Device │ admin            │ Default     │
│ euJKv7m0o  │ atreyu                 │ Device │                  │ Default     │
│ fM9mAb.Gs  │ mac2.jptech.ziti       │ Device │ admin,jptech,ssh │ Default     │
│ fmI.Nbw5Vl │ win.jptech.ziti        │ Device │ admin,jptech,ssh │ Default     │
│ gveVhi.Gs  │ ios.jptech.ziti        │ Device │ admin,notech,ssh │ Default     │
│ pE4Bv1tDS  │ ziti-edge-router       │ Router │                  │ Default     │
╰────────────┴────────────────────────┴────────┴──────────────────┴─────────────╯
results: 1-10 of 11
6

Update, I have this log in the edgerouter. [service 77OCwbctFezGe4tYTQ5ZgD has no terminators for instanceId win.jptech.ziti] 

ziti-edge-router_1                | [16360.410] WARNING edge/router/xgress_edge.(*edgeClientConn).processConnect [ch{edge}->u{classic}->i{Vq7p}]: {type=[EdgeConnectType] chSeq=[92] edgeSeq=[0] error=[service 77OCwbctFezGe4tYTQ5ZgD has no terminators for instanceId win.jptech.ziti] token=[be8332ac-0f5c-4adb-8ee6-f94a47670bc2] connId=[72]} failed to dial fabric

When it is successful there is no output on the logs.

Here is the full output of the identities. It’s late, and I can’t tell the difference between the mac1... identitity that I CAN connect to and all the other ones.

{
    "data": [
        {
            "_links": {
                "auth-policies": {
                    "href": "./auth-policies/default"
                },
                "authenticators": {
                    "href": "./identities/1DDBNiwGV/authenticators"
                },
                "edge-router-policies": {
                    "href": "./identities/1DDBNiwGV/edge-router-policies"
                },
                "edge-routers": {
                    "href": "./identities/1DDBNiwGV/edge-routers"
                },
                "enrollments": {
                    "href": "./identities/1DDBNiwGV/enrollments"
                },
                "failed-service-requests": {
                    "href": "./identities/1DDBNiwGV/failed-service-requests"
                },
                "posture-data": {
                    "href": "./identities/1DDBNiwGV/posture-data"
                },
                "self": {
                    "href": "./identities/1DDBNiwGV"
                },
                "service-configs": {
                    "href": "./identities/1DDBNiwGV/service-configs"
                },
                "service-policies": {
                    "href": "./identities/1DDBNiwGV/service-policies"
                },
                "services": {
                    "href": "./identities/1DDBNiwGV/services"
                }
            },
            "createdAt": "2023-08-08T03:05:22.342Z",
            "id": "1DDBNiwGV",
            "tags": {},
            "updatedAt": "2023-08-08T03:56:36.813Z",
            "appData": {},
            "authPolicy": {
                "_links": {
                    "self": {
                        "href": "./auth-policies/default"
                    }
                },
                "entity": "auth-policies",
                "id": "default",
                "name": "Default"
            },
            "authPolicyId": "default",
            "authenticators": {
                "cert": {
                    "id": "OYxUjbw5V"
                }
            },
            "defaultHostingCost": 0,
            "defaultHostingPrecedence": "default",
            "disabled": false,
            "enrollment": {},
            "envInfo": {
                "arch": "x86_64",
                "os": "Linux",
                "osRelease": "6.1.0-9-cloud-amd64",
                "osVersion": "#1 SMP PREEMPT_DYNAMIC Debian 6.1.27-1 (2023-05-08)"
            },
            "externalId": null,
            "hasApiSession": true,
            "hasEdgeRouterConnection": true,
            "isAdmin": false,
            "isDefaultAdmin": false,
            "isMfaEnabled": false,
            "name": "mac1.jptech.ziti",
            "roleAttributes": [
                "admin",
                "jptech",
                "ssh"
            ],
            "sdkInfo": {
                "appId": "ziti-edge-tunnel",
                "appVersion": "v0.21.5-local",
                "branch": "HEAD",
                "revision": "d7f329f",
                "type": "ziti-sdk-c",
                "version": "0.32.8"
            },
            "serviceHostingCosts": {},
            "serviceHostingPrecedences": {},
            "type": {
                "_links": {
                    "self": {
                        "href": "./identity-types/Device"
                    }
                },
                "entity": "identity-types",
                "id": "Device",
                "name": "Device"
            },
            "typeId": "Device"
        },
        {
            "_links": {
                "auth-policies": {
                    "href": "./auth-policies/default"
                },
                "authenticators": {
                    "href": "./identities/6bNWqyt0o/authenticators"
                },
                "edge-router-policies": {
                    "href": "./identities/6bNWqyt0o/edge-router-policies"
                },
                "edge-routers": {
                    "href": "./identities/6bNWqyt0o/edge-routers"
                },
                "enrollments": {
                    "href": "./identities/6bNWqyt0o/enrollments"
                },
                "failed-service-requests": {
                    "href": "./identities/6bNWqyt0o/failed-service-requests"
                },
                "posture-data": {
                    "href": "./identities/6bNWqyt0o/posture-data"
                },
                "self": {
                    "href": "./identities/6bNWqyt0o"
                },
                "service-configs": {
                    "href": "./identities/6bNWqyt0o/service-configs"
                },
                "service-policies": {
                    "href": "./identities/6bNWqyt0o/service-policies"
                },
                "services": {
                    "href": "./identities/6bNWqyt0o/services"
                }
            },
            "createdAt": "2023-08-01T03:45:41.002Z",
            "id": "6bNWqyt0o",
            "tags": {},
            "updatedAt": "2023-08-08T06:26:31.959Z",
            "appData": {},
            "authPolicy": {
                "_links": {
                    "self": {
                        "href": "./auth-policies/default"
                    }
                },
                "entity": "auth-policies",
                "id": "default",
                "name": "Default"
            },
            "authPolicyId": "default",
            "authenticators": {
                "cert": {
                    "id": "htX4UJt0o"
                }
            },
            "defaultHostingCost": 0,
            "defaultHostingPrecedence": "default",
            "disabled": false,
            "enrollment": {},
            "envInfo": {
                "arch": "x86_64",
                "os": "Linux",
                "osRelease": "6.1.0-10-amd64",
                "osVersion": "#1 SMP PREEMPT_DYNAMIC Debian 6.1.38-1 (2023-07-14)"
            },
            "externalId": null,
            "hasApiSession": true,
            "hasEdgeRouterConnection": true,
            "isAdmin": false,
            "isDefaultAdmin": false,
            "isMfaEnabled": false,
            "name": "macbian",
            "roleAttributes": [
                "admin",
                "jptech",
                "ssh"
            ],
            "sdkInfo": {
                "appId": "ziti-edge-tunnel",
                "appVersion": "v0.21.5-local",
                "branch": "HEAD",
                "revision": "d7f329f",
                "type": "ziti-sdk-c",
                "version": "0.32.8"
            },
            "serviceHostingCosts": {},
            "serviceHostingPrecedences": {},
            "type": {
                "_links": {
                    "self": {
                        "href": "./identity-types/Device"
                    }
                },
                "entity": "identity-types",
                "id": "Device",
                "name": "Device"
            },
            "typeId": "Device"
        },
        {
            "_links": {
                "auth-policies": {
                    "href": "./auth-policies/default"
                },
                "authenticators": {
                    "href": "./identities/7ARsJMakq/authenticators"
                },
                "edge-router-policies": {
                    "href": "./identities/7ARsJMakq/edge-router-policies"
                },
                "edge-routers": {
                    "href": "./identities/7ARsJMakq/edge-routers"
                },
                "enrollments": {
                    "href": "./identities/7ARsJMakq/enrollments"
                },
                "failed-service-requests": {
                    "href": "./identities/7ARsJMakq/failed-service-requests"
                },
                "posture-data": {
                    "href": "./identities/7ARsJMakq/posture-data"
                },
                "self": {
                    "href": "./identities/7ARsJMakq"
                },
                "service-configs": {
                    "href": "./identities/7ARsJMakq/service-configs"
                },
                "service-policies": {
                    "href": "./identities/7ARsJMakq/service-policies"
                },
                "services": {
                    "href": "./identities/7ARsJMakq/services"
                }
            },
            "createdAt": "2023-07-30T06:37:55.403Z",
            "id": "7ARsJMakq",
            "tags": {},
            "updatedAt": "2023-07-30T06:37:55.403Z",
            "appData": {},
            "authPolicy": {
                "_links": {
                    "self": {
                        "href": "./auth-policies/default"
                    }
                },
                "entity": "auth-policies",
                "id": "default",
                "name": "Default"
            },
            "authPolicyId": "default",
            "authenticators": {
                "updb": {
                    "id": "7pR6JoukZ5",
                    "username": "admin"
                }
            },
            "defaultHostingCost": 0,
            "defaultHostingPrecedence": "default",
            "disabled": false,
            "enrollment": {},
            "envInfo": {},
            "externalId": null,
            "hasApiSession": true,
            "hasEdgeRouterConnection": false,
            "isAdmin": true,
            "isDefaultAdmin": true,
            "isMfaEnabled": false,
            "name": "Default Admin",
            "roleAttributes": null,
            "sdkInfo": {},
            "serviceHostingCosts": {},
            "serviceHostingPrecedences": {},
            "type": {
                "_links": {
                    "self": {
                        "href": "./identity-types/User"
                    }
                },
                "entity": "identity-types",
                "id": "User",
                "name": "User"
            },
            "typeId": "User"
        },
        {
            "_links": {
                "auth-policies": {
                    "href": "./auth-policies/default"
                },
                "authenticators": {
                    "href": "./identities/fM9mAb.Gs/authenticators"
                },
                "edge-router-policies": {
                    "href": "./identities/fM9mAb.Gs/edge-router-policies"
                },
                "edge-routers": {
                    "href": "./identities/fM9mAb.Gs/edge-routers"
                },
                "enrollments": {
                    "href": "./identities/fM9mAb.Gs/enrollments"
                },
                "failed-service-requests": {
                    "href": "./identities/fM9mAb.Gs/failed-service-requests"
                },
                "posture-data": {
                    "href": "./identities/fM9mAb.Gs/posture-data"
                },
                "self": {
                    "href": "./identities/fM9mAb.Gs"
                },
                "service-configs": {
                    "href": "./identities/fM9mAb.Gs/service-configs"
                },
                "service-policies": {
                    "href": "./identities/fM9mAb.Gs/service-policies"
                },
                "services": {
                    "href": "./identities/fM9mAb.Gs/services"
                }
            },
            "createdAt": "2023-08-08T03:05:28.537Z",
            "id": "fM9mAb.Gs",
            "tags": {},
            "updatedAt": "2023-08-08T06:12:11.511Z",
            "appData": {},
            "authPolicy": {
                "_links": {
                    "self": {
                        "href": "./auth-policies/default"
                    }
                },
                "entity": "auth-policies",
                "id": "default",
                "name": "Default"
            },
            "authPolicyId": "default",
            "authenticators": {
                "cert": {
                    "id": "oY6Unb.5s"
                }
            },
            "defaultHostingCost": 0,
            "defaultHostingPrecedence": "default",
            "disabled": false,
            "enrollment": {},
            "envInfo": {
                "arch": "x86_64",
                "os": "Linux",
                "osRelease": "6.1.0-9-cloud-amd64",
                "osVersion": "#1 SMP PREEMPT_DYNAMIC Debian 6.1.27-1 (2023-05-08)"
            },
            "externalId": null,
            "hasApiSession": true,
            "hasEdgeRouterConnection": false,
            "isAdmin": false,
            "isDefaultAdmin": false,
            "isMfaEnabled": false,
            "name": "mac2.jptech.ziti",
            "roleAttributes": [
                "jptech",
                "ssh"
            ],
            "sdkInfo": {
                "appId": "ziti-edge-tunnel",
                "appVersion": "v0.21.5-local",
                "branch": "HEAD",
                "revision": "d7f329f",
                "type": "ziti-sdk-c",
                "version": "0.32.8"
            },
            "serviceHostingCosts": {},
            "serviceHostingPrecedences": {},
            "type": {
                "_links": {
                    "self": {
                        "href": "./identity-types/Device"
                    }
                },
                "entity": "identity-types",
                "id": "Device",
                "name": "Device"
            },
            "typeId": "Device"
        },
        {
            "_links": {
                "auth-policies": {
                    "href": "./auth-policies/default"
                },
                "authenticators": {
                    "href": "./identities/fmI.Nbw5Vl/authenticators"
                },
                "edge-router-policies": {
                    "href": "./identities/fmI.Nbw5Vl/edge-router-policies"
                },
                "edge-routers": {
                    "href": "./identities/fmI.Nbw5Vl/edge-routers"
                },
                "enrollments": {
                    "href": "./identities/fmI.Nbw5Vl/enrollments"
                },
                "failed-service-requests": {
                    "href": "./identities/fmI.Nbw5Vl/failed-service-requests"
                },
                "posture-data": {
                    "href": "./identities/fmI.Nbw5Vl/posture-data"
                },
                "self": {
                    "href": "./identities/fmI.Nbw5Vl"
                },
                "service-configs": {
                    "href": "./identities/fmI.Nbw5Vl/service-configs"
                },
                "service-policies": {
                    "href": "./identities/fmI.Nbw5Vl/service-policies"
                },
                "services": {
                    "href": "./identities/fmI.Nbw5Vl/services"
                }
            },
            "createdAt": "2023-08-08T03:05:35.962Z",
            "id": "fmI.Nbw5Vl",
            "tags": {},
            "updatedAt": "2023-08-08T06:09:29.573Z",
            "appData": {},
            "authPolicy": {
                "_links": {
                    "self": {
                        "href": "./auth-policies/default"
                    }
                },
                "entity": "auth-policies",
                "id": "default",
                "name": "Default"
            },
            "authPolicyId": "default",
            "authenticators": {
                "cert": {
                    "id": "yx6UjiwGV"
                }
            },
            "defaultHostingCost": 0,
            "defaultHostingPrecedence": "default",
            "disabled": false,
            "enrollment": {},
            "envInfo": {
                "arch": "x86_64",
                "os": "Linux",
                "osRelease": "6.1.0-9-cloud-amd64",
                "osVersion": "#1 SMP PREEMPT_DYNAMIC Debian 6.1.27-1 (2023-05-08)"
            },
            "externalId": null,
            "hasApiSession": true,
            "hasEdgeRouterConnection": false,
            "isAdmin": false,
            "isDefaultAdmin": false,
            "isMfaEnabled": false,
            "name": "win.jptech.ziti",
            "roleAttributes": [
                "jptech",
                "ssh"
            ],
            "sdkInfo": {
                "appId": "ziti-edge-tunnel",
                "appVersion": "v0.21.5-local",
                "branch": "HEAD",
                "revision": "d7f329f",
                "type": "ziti-sdk-c",
                "version": "0.32.8"
            },
            "serviceHostingCosts": {},
            "serviceHostingPrecedences": {},
            "type": {
                "_links": {
                    "self": {
                        "href": "./identity-types/Device"
                    }
                },
                "entity": "identity-types",
                "id": "Device",
                "name": "Device"
            },
            "typeId": "Device"
        },
        {
            "_links": {
                "auth-policies": {
                    "href": "./auth-policies/default"
                },
                "authenticators": {
                    "href": "./identities/gveVhi.Gs/authenticators"
                },
                "edge-router-policies": {
                    "href": "./identities/gveVhi.Gs/edge-router-policies"
                },
                "edge-routers": {
                    "href": "./identities/gveVhi.Gs/edge-routers"
                },
                "enrollments": {
                    "href": "./identities/gveVhi.Gs/enrollments"
                },
                "failed-service-requests": {
                    "href": "./identities/gveVhi.Gs/failed-service-requests"
                },
                "posture-data": {
                    "href": "./identities/gveVhi.Gs/posture-data"
                },
                "self": {
                    "href": "./identities/gveVhi.Gs"
                },
                "service-configs": {
                    "href": "./identities/gveVhi.Gs/service-configs"
                },
                "service-policies": {
                    "href": "./identities/gveVhi.Gs/service-policies"
                },
                "services": {
                    "href": "./identities/gveVhi.Gs/services"
                }
            },
            "createdAt": "2023-08-08T03:06:23.577Z",
            "id": "gveVhi.Gs",
            "tags": {},
            "updatedAt": "2023-08-08T03:06:23.577Z",
            "appData": {},
            "authPolicy": {
                "_links": {
                    "self": {
                        "href": "./auth-policies/default"
                    }
                },
                "entity": "auth-policies",
                "id": "default",
                "name": "Default"
            },
            "authPolicyId": "default",
            "authenticators": {},
            "defaultHostingCost": 0,
            "defaultHostingPrecedence": "default",
            "disabled": false,
            "enrollment": {
                "ott": {
                    "expiresAt": "2023-08-15T03:06:23.578Z",
                    "id": "gEeV4i.5Vl",
                }
            },
            "envInfo": {},
            "externalId": null,
            "hasApiSession": false,
            "hasEdgeRouterConnection": false,
            "isAdmin": false,
            "isDefaultAdmin": false,
            "isMfaEnabled": false,
            "name": "ios.jptech.ziti",
            "roleAttributes": [
                "admin",
                "notech",
                "ssh"
            ],
            "sdkInfo": {},
            "serviceHostingCosts": {},
            "serviceHostingPrecedences": {},
            "type": {
                "_links": {
                    "self": {
                        "href": "./identity-types/Device"
                    }
                },
                "entity": "identity-types",
                "id": "Device",
                "name": "Device"
            },
            "typeId": "Device"
        },
        {
            "_links": {
                "auth-policies": {
                    "href": "./auth-policies/default"
                },
                "authenticators": {
                    "href": "./identities/pE4Bv1tDS/authenticators"
                },
                "edge-router-policies": {
                    "href": "./identities/pE4Bv1tDS/edge-router-policies"
                },
                "edge-routers": {
                    "href": "./identities/pE4Bv1tDS/edge-routers"
                },
                "enrollments": {
                    "href": "./identities/pE4Bv1tDS/enrollments"
                },
                "failed-service-requests": {
                    "href": "./identities/pE4Bv1tDS/failed-service-requests"
                },
                "posture-data": {
                    "href": "./identities/pE4Bv1tDS/posture-data"
                },
                "self": {
                    "href": "./identities/pE4Bv1tDS"
                },
                "service-configs": {
                    "href": "./identities/pE4Bv1tDS/service-configs"
                },
                "service-policies": {
                    "href": "./identities/pE4Bv1tDS/service-policies"
                },
                "services": {
                    "href": "./identities/pE4Bv1tDS/services"
                }
            },
            "createdAt": "2023-07-30T06:38:01.645Z",
            "id": "pE4Bv1tDS",
            "tags": {},
            "updatedAt": "2023-08-08T06:13:26.473Z",
            "appData": {},
            "authPolicy": {
                "_links": {
                    "self": {
                        "href": "./auth-policies/default"
                    }
                },
                "entity": "auth-policies",
                "id": "default",
                "name": "Default"
            },
            "authPolicyId": "default",
            "authenticators": {},
            "defaultHostingCost": 0,
            "defaultHostingPrecedence": "default",
            "disabled": false,
            "enrollment": {},
            "envInfo": {
                "arch": "amd64",
                "os": "linux",
                "osRelease": "6.1.0-10-amd64",
                "osVersion": "#1 SMP PREEMPT_DYNAMIC Debian 6.1.38-1 (2023-07-14)"
            },
            "externalId": null,
            "hasApiSession": true,
            "hasEdgeRouterConnection": true,
            "isAdmin": false,
            "isDefaultAdmin": false,
            "isMfaEnabled": false,
            "name": "ziti-edge-router",
            "roleAttributes": null,
            "sdkInfo": {
                "appId": "ziti-router",
                "appVersion": "v0.29.0",
                "branch": "main",
                "revision": "3ca2dd2f4e7b",
                "type": "ziti-router:tunnel",
                "version": "v0.29.0"
            },
            "serviceHostingCosts": {},
            "serviceHostingPrecedences": {},
            "type": {
                "_links": {
                    "self": {
                        "href": "./identity-types/Router"
                    }
                },
                "entity": "identity-types",
                "id": "Router",
                "name": "Router"
            },
            "typeId": "Router"
        }
    ],
    "meta": {
        "filterableFields": [
            "tags",
            "roleAttributes",
            "externalId",
            "createdAt",
            "updatedAt",
            "isSystem",
            "name",
            "type",
            "authPolicyId",
            "isAdmin",
            "isDefaultAdmin",
            "id"
        ],
        "pagination": {
            "limit": 10,
            "offset": 0,
            "totalCount": 11
        }
    }
}
1 Like
7

A “terminator” is created when an identity successfully binds a service. You can see the terminators with the ziti cli:

ziti edge list terminators
╭────────────────────────┬────────────┬───────────────────────┬─────────┬─────────────────────────────────────────────┬───────────────────┬──────┬────────────┬──────────────╮
│ ID                     │ SERVICE    │ ROUTER                │ BINDING │ ADDRESS                                     │ IDENTITY          │ COST │ PRECEDENCE │ DYNAMIC COST │
├────────────────────────┼────────────┼───────────────────────┼─────────┼─────────────────────────────────────────────┼───────────────────┼──────┼────────────┼──────────────┤
│ 4xY7ArlxxDdiN31vRbnhaU │ jptech.ssh │ shawns-m1-mbp-0.29.0+ │ edge    │ hosted:e0cf2c22-9867-41f9-a0ca-e13796a7a427 │ linux.jptech.ziti │    0 │ default    │            0 │
╰────────────────────────┴────────────┴───────────────────────┴─────────┴─────────────────────────────────────────────┴───────────────────┴──────┴────────────┴──────────────╯
results: 1-1 of 1

If you aren’t seeing a terminator for identities that have been assigned the “ssh” role, the first place to look is in the logs from the client that is using that identity. I can see from your identity output that the “win” identity (and the others for that matter) are actually running on Debian and connecting with ziti-edge-tunnel. If you’re running ziti-edge-tunnel from systemd, you can see the logs with sudo journalctl -u ziti-edge-tunnel | cat. If you’re running from a script or the command line then you’ll need to capture the standard error to see the logs.

For the identities that aren’t working after you toggle the “admin” role, there’s likely something going on at the intercepting tunneler. ziti-edge-tunnel should notice when the role assignment has changed and you should see some activity in the logs about the service becoming available/unavailable shortly (~15s) after updating the identity’s role.

Could you share the logs from the “win” and “macbian” identities with me, either in this thread or as a DM if you have any privacy concerns?

8

Thanks for your help.

I started with just a bunch of linux hosts spun up real quick... named the mostly accordingly, then created the resources. Once I had it confirmed with the linux tunneler I would swap out a real windows and real mac device. As of this morning, macbian can ssh to mac1.jptech.ziti. There might be some errors in there as I have been working on making DNS work consistently. So my network's resolver is directing the jptech.ziti domain back to the 100.64.0.2 address instead of trying to resolve it locally.

Here is the win device

-- Boot 183e8ceea24b4b31a37ccc8475af0a10 --
Aug 08 16:50:45 win1 systemd[1]: Starting ziti-edge-tunnel.service - Ziti Edge Tunnel...
Aug 08 16:50:45 win1 ziti-edge-tunnel.sh[412]: NOTICE: no new JWT files in /opt/openziti/etc/identities/*.jwt
Aug 08 16:50:45 win1 systemd[1]: Started ziti-edge-tunnel.service - Ziti Edge Tunnel.
Aug 08 16:50:47 win1 ziti-edge-tunnel[414]: (414)[        1.993]   ERROR ziti-sdk:connect.c:974 connect_reply_cb() conn[0.0/Binding] failed to bind, reason=sibling terminator hdQDNO6Nc12QPrScVJblB with shared identity win.jptech.ziti belongs to different identity
Aug 08 16:50:47 win1 ziti-edge-tunnel[414]: (414)[        1.993]   ERROR tunnel-cbs:ziti_hosting.c:584 hosted_listen_cb() unable to host service jptech.ssh: connection is closed

Here is the mac1 device

Aug 08 16:57:20 mac1 systemd[1]: Stopping ziti-edge-tunnel.service - Ziti Edge Tunnel...
Aug 08 16:57:20 mac1 systemd[1]: ziti-edge-tunnel.service: Deactivated successfully.
Aug 08 16:57:20 mac1 systemd[1]: Stopped ziti-edge-tunnel.service - Ziti Edge Tunnel.
Aug 08 16:57:20 mac1 systemd[1]: ziti-edge-tunnel.service: Consumed 5min 48.875s CPU time.
-- Boot 092e71a4014047e3bbfa5b6519d32d94 --
Aug 08 16:57:28 mac1 systemd[1]: Starting ziti-edge-tunnel.service - Ziti Edge Tunnel...
Aug 08 16:57:28 mac1 ziti-edge-tunnel.sh[412]: NOTICE: no new JWT files in /opt/openziti/etc/identities/*.jwt
Aug 08 16:57:28 mac1 systemd[1]: Started ziti-edge-tunnel.service - Ziti Edge Tunnel.

Here is the macbian device

Aug 08 08:54:30 macbian ziti-edge-tunnel[49315]: sed: couldn't open temporary file /etc/sedXej5SY: Per
mission denied
Aug 08 08:54:30 macbian ziti-edge-tunnel[2597]: (2597)[    22021.954]   ERROR ziti-edge-tunnel:utils.c
:31 run_command_va() cmd{sed -z -i 's/nameserver/nameserver 100.64.0.2\nnameserver/' /etc/resolv.conf}
 failed: 1024/0/Success
Aug 08 08:54:32 macbian ziti-edge-tunnel[2597]: (2597)[    22023.703]   ERROR ziti-sdk:ziti_ctrl.c:154
 ctrl_resp_cb() ctrl[ziti.jptech.ninja] request failed: -3008(unknown node or service)
Aug 08 08:54:32 macbian ziti-edge-tunnel[2597]: (2597)[    22023.703]   ERROR ziti-sdk:ziti_ctrl.c:154
 ctrl_resp_cb() ctrl[ziti.jptech.ninja] request failed: -3008(unknown node or service)
Aug 08 08:54:32 macbian ziti-edge-tunnel[2597]: (2597)[    22023.703]   ERROR ziti-sdk:ziti.c:1034 upd
ate_services() ztx[0] failed to get service updates err[CONTROLLER_UNAVAILABLE/unknown node or service
] from ctrl[https://ziti.jptech.ninja:8441]
Aug 08 08:54:32 macbian ziti-edge-tunnel[2597]: (2597)[    22023.703]    WARN tunnel-cbs:ziti_tunnel_c
trl.c:740 on_ziti_event() ziti_ctx controller connections failed: Ziti Controller is not available
Aug 08 08:54:32 macbian ziti-edge-tunnel[2597]: (2597)[    22023.703]   ERROR ziti-edge-tunnel:ziti-ed
ge-tunnel.c:1199 on_event() ztx[/opt/openziti/etc/identities/macbian.json] failed to connect to contro
ller due to Ziti Controller is not available
Aug 08 08:54:34 macbian ziti-edge-tunnel[49482]: sed: couldn't open temporary file /etc/sed31Hurj: Per
mission denied
Aug 08 08:54:34 macbian ziti-edge-tunnel[2597]: (2597)[    22025.248]   ERROR ziti-edge-tunnel:utils.c
:31 run_command_va() cmd{sed -z -i 's/nameserver/nameserver 100.64.0.2\nnameserver/' /etc/resolv.conf}
 failed: 1024/0/Success
Aug 08 08:54:44 macbian ziti-edge-tunnel[2597]: (2597)[    22035.337]   ERROR ziti-sdk:ziti.c:1034 upd
ate_services() ztx[0] failed to get service updates err[UNAUTHORIZED/The request could not be complete
d. The session is not authorized or the credentials are invalid] from ctrl[https://ziti.jptech.ninja:8
441]
Aug 08 08:54:44 macbian ziti-edge-tunnel[2597]: (2597)[    22035.337]    WARN ziti-sdk:ziti.c:1037 upd
ate_services() ztx[0] api session is no longer valid. Trying to re-auth
Aug 08 08:55:22 macbian ziti-edge-tunnel[2597]: (2597)[    22073.945]   ERROR ziti-sdk:channel.c:577 l
atency_timeout() ch[2] no read/write traffic on channel since before latency probe was sent, closing c
hannel
Aug 08 08:55:57 macbian ziti-edge-tunnel[2597]: (2597)[    22108.092]   ERROR ziti-sdk:connect.c:974 c
onnect_reply_cb() conn[0.122/Connecting] failed to connect, reason=service 1P8vfj5YwzobHCx6qVJDxY has
no terminators for instanceId win.jptech.ziti
Aug 08 08:55:57 macbian ziti-edge-tunnel[2597]: (2597)[    22108.092]   ERROR tunnel-cbs:ziti_tunnel_c
bs.c:103 on_ziti_connect() ziti dial failed: connection is closed
Aug 08 08:56:45 macbian ziti-edge-tunnel[2597]: (2597)[    22156.785]   ERROR ziti-sdk:connect.c:974 c
onnect_reply_cb() conn[0.123/Connecting] failed to connect, reason=service 1P8vfj5YwzobHCx6qVJDxY has
no terminators for instanceId linux.jptech.ziti
Aug 08 08:56:45 macbian ziti-edge-tunnel[2597]: (2597)[    22156.785]   ERROR tunnel-cbs:ziti_tunnel_c
bs.c:103 on_ziti_connect() ziti dial failed: connection is closed
Aug 08 08:56:50 macbian ziti-edge-tunnel[2597]: (2597)[    22161.493]   ERROR ziti-sdk:connect.c:974 c
onnect_reply_cb() conn[0.124/Connecting] failed to connect, reason=service 1P8vfj5YwzobHCx6qVJDxY has
no terminators for instanceId mac2.jptech.ziti
Aug 08 08:56:50 macbian ziti-edge-tunnel[2597]: (2597)[    22161.493]   ERROR tunnel-cbs:ziti_tunnel_c
bs.c:103 on_ziti_connect() ziti dial failed: connection is closed
9

I forgot… as an update. I deleted all the non-identity resources and recreated them in my testing. I was looking to see how durable/fragile they are. It seems like by using the role attributes (which I was referring to when I said tags… but I guess there are tags too) that there is a lot of flexibility.

I can also see that my thoughts on the flexibility on the role names being separate actually kind of open me up to error. Whereas strings like jptech.admin or jptech.ssh and client.ssh would be less likely to get separated from ‘loose’ role strings. Excellent observation, thank you!

BTW… thanks for helping me get over the hump of using the UI. This is a beautiful thing, to be able to recreate an environment and build the resources from a single command!

root@ziti:~/ziti/configscript# sh ./build.sh
New service edge router policy all created with id: 20MZjuVZuL9A2uCB8z4jKy
New edge router policy all created with id: 2vctClPw8O9iTaMHrARvcm
New config jptech.ssh.cfg.intercept created with id: 929pxd0ARCRgIYEm7okt
New config jptech.ssh.cfg.host created with id: 7J4Npd8Kmj6jG0aoifF0x8
New service jptech.ssh created with id: 6dxVqc21YVpJ2kuGDtnkt3
New service policy jptech.ssh.dial created with id: 2eWzdCIkzhdQHzsg5utyBi
New service policy jptech.ssh.bind created with id: 1aGQCqMP2iYiynVG7CbMS6
New identity linux.jptech.ziti created with id: JSABqhry7
Enrollment expires at 2023-08-15T18:59:54.441Z
New identity mac1.jptech.ziti created with id: CgbB7hrY7w
Enrollment expires at 2023-08-15T18:59:54.715Z
New identity mac2.jptech.ziti created with id: mjAB7SrYq
Enrollment expires at 2023-08-15T18:59:54.965Z
New identity win.jptech.ziti created with id: ifLd7hryq
Enrollment expires at 2023-08-15T18:59:55.243Z
New identity falkor.jptech.ziti created with id: 1hXBqSryqw
Enrollment expires at 2023-08-15T18:59:55.476Z
New identity macbian.jptech.ziti created with id: JeXd7Sry7
Enrollment expires at 2023-08-15T18:59:55.721Z
10

Are you still seeing this? If so could you please show me the output from ziti edge list terminators? This error suggests that there are two or more identities trying to use the same listen options identity value ("win.jptech.ziti" in this case) when binding the service, but that shouldn't be happening if your host.v1 configuration specifies $tunneler_id.name.

Is the host running macbian a different distro than the others? ziti-edge-tunnel thinks that it needs to modify /etc/resolve.conf in order to add it's internal DNS server to the host's resolver, but it isn't able to because it lacks permission. btw our systemd unit file runs the process with a non-root user named "ziti", and we configure the system (via polkit) to allow the ziti user to manipulate the per-link DNS server list. Modifying resolv.conf may be necessary if the distro isn't using systemd-resolved, but I wonder if it's somehow confused for two reasons:

  1. I noticed that all of your identities reported the same osRelease and osVersion (which happens to be a fairly recent kernel version), and the other identities aren't going down this path. I'd expect they are setting the DNS Server on the tun interface that ziti-edge-tunnel creates (can be seen with resolvectl when ziti-edge-tunnel is running).
  2. The logs show that your controller hostname "ziti.jptech.ninja" cannot be resolved, and I wonder if the system's resolver (possibly resolv.conf) is in working order.

Can you show me the following from the macbian host?

  • ls -l /etc/resolv.conf
  • cat /etc/resolv.conf
  • ls -l /run/systemd/resolv
  • cat /run/systemd/resolve/resolv.conf
  • cat /run/systemd/resolve/stub-resolv.conf
  • systemctl is-active systemd-resolved.service
  • systemctl cat ziti-edge-tunnel.service

Sorry for the long list of items. I'm hoping to avoid more round trips for both of us :slight_smile:

Thanks

11

These are good helps! In part of my testing I have destroyed the environment and started clean again. I am testing all on the same os containers for simplicity. I have scripted the creation of the resources for this test, and I added the clearer named resources per your earliest suggestion on the role names. Below is my creation script. Falkor and Macbian are the two systems I am usually sitting at that I am testing with. Falkor is a mac, Macbian is debian.

Yes, one of the hosts is displaying this.
Aug 08 19:40:49 linux ziti-edge-tunnel[161]: (161)[ 2.039] ERROR ziti-sdk:connect.c:974 connect_reply_cb() conn[0.0/Binding] failed to bind, reason=sibling terminator 1YSIX2WgKYBaqrP0wP77aP with shared identity linux.jptech.ziti belongs to different identity

Here is my output for terminators and identities:

t# ziti edge list terminators
╭────────────────────────┬────────────┬──────────────────┬─────────┬─────────────────────────────────────────────┬────────────────────┬──────┬────────────┬──────────────╮
│ ID                     │ SERVICE    │ ROUTER           │ BINDING │ ADDRESS                                     │ IDENTITY           │ COST │ PRECEDENCE │ DYNAMIC COST │
├────────────────────────┼────────────┼──────────────────┼─────────┼─────────────────────────────────────────────┼────────────────────┼──────┼────────────┼──────────────┤
│ 51yA6F1TtdHjEJt1FMLc3Q │ jptech.ssh │ ziti-edge-router │ edge    │ hosted:33ee33c9-9ae0-4c66-9d32-6d91e393e601 │ falkor.jptech.ziti │    0 │ default    │            0 │
╰────────────────────────┴────────────┴──────────────────┴─────────┴─────────────────────────────────────────────┴────────────────────┴──────┴────────────┴──────────────╯
results: 1-1 of 1
root@ziti:~/ziti/configscript# ziti edge list identities
╭────────────┬─────────────────────┬────────┬────────────────────────────────┬─────────────╮
│ ID         │ NAME                │ TYPE   │ ATTRIBUTES                     │ AUTH-POLICY │
├────────────┼─────────────────────┼────────┼────────────────────────────────┼─────────────┤
│ 1hXBqSryqw │ falkor.jptech.ziti  │ Device │ jptech.admin,jptech.ssh.server │ Default     │
│ AxLx9Bi-D  │ Default Admin       │ User   │                                │ Default     │
│ CgbB7hrY7w │ mac1.jptech.ziti    │ Device │ jptech.admin                   │ Default     │
│ JSABqhry7  │ linux.jptech.ziti   │ Device │ jptech.admin,jptech.ssh.server │ Default     │
│ JeXd7Sry7  │ macbian.jptech.ziti │ Device │ jptech.admin,jptech.ssh.server │ Default     │
│ Yff-7SFyqw │ ziti-edge-router    │ Router │                                │ Default     │
│ ifLd7hryq  │ win.jptech.ziti     │ Device │ jptech.ssh.server              │ Default     │
│ mjAB7SrYq  │ mac2.jptech.ziti    │ Device │ jptech.ssh.server              │ Default     │
╰────────────┴─────────────────────┴────────┴────────────────────────────────┴─────────────╯
results: 1-8 of 8

regarding resolv.conf... is it necessary to work properly beyond just resolving? i.e. if I can dig linux.jptech.ziti @100.64.0.2 and get the correct IP, can I then just ssh root@100.64.0.6 and accomplish the same test? I don't want to mix DNS issues on the hosts if I can avoid it.

Here are the outputs you asked for:

root@win:~# ls -l /etc/resolv.conf
-rw-r--r-- 1 root root 80 Aug  8 19:49 /etc/resolv.conf
root@win:~# cat /etc/resolv.conf
domain jptech.corp
search jptech.corp
nameserver 100.64.0.2
nameserver 10.1.1.1
root@win:~# ls -l /run/systemd/resolv
ls: cannot access '/run/systemd/resolv': No such file or directory
root@win:~# cat /run/systemd/resolve/resolv.conf
cat: /run/systemd/resolve/resolv.conf: No such file or directory
root@win:~# cat /run/systemd/resolve/stub-resolv.conf
cat: /run/systemd/resolve/stub-resolv.conf: No such file or directory
root@win:~# systemctl is-active systemd-resolved.service
inactive
root@win:~# systemctl cat ziti-edge-tunnel.service
# /etc/systemd/system/ziti-edge-tunnel.service
[Unit]
Description=Ziti Edge Tunnel
After=network-online.target

[Service]
Type=simple
EnvironmentFile=/opt/openziti/etc/ziti-edge-tunnel.env
User=ziti
UMask=0007
AmbientCapabilities=CAP_NET_ADMIN
ExecStartPre=/opt/openziti/bin/ziti-edge-tunnel.sh
ExecStart=/opt/openziti/bin/ziti-edge-tunnel run --verbose=${ZITI_VERBOSE} --dns-ip-range=${ZITI_DNS_IP_RANGE} --identity-dir=${ZITI_IDENTITY_DIR}
Restart=always
RestartSec=3

[Install]
WantedBy=multi-user.target

image

Here is my entire install script content:

ziti edge create service-edge-router-policy all --service-roles '#all' --edge-router-roles '#all'
ziti edge create edge-router-policy all --edge-router-roles '#all'  --identity-roles '#all'

ziti edge create config jptech.ssh.cfg.intercept intercept.v1 '{
    "addresses": ["*.jptech.ziti"],
    "protocols": ["tcp"],
    "portRanges": [ {"low":22,"high":22} ],
    "dialOptions": { "identity": "$dst_hostname" }
}'
ziti edge create config jptech.ssh.cfg.host host.v1 '{
    "address": "127.0.0.1",
    "protocol": "tcp",
    "port": 22,
    "listenOptions": { "identity": "$tunneler_id.name" }
}'

ziti edge create service jptech.ssh \
    --configs jptech.ssh.cfg.intercept,jptech.ssh.cfg.host \
    --role-attributes jptech.admin,jptech.ssh.server


ziti edge create service-policy jptech.ssh.dial Dial --identity-roles "#jptech.admin" --service-roles "@jptech.ssh"

ziti edge create service-policy jptech.ssh.bind Bind --identity-roles "#jptech.admin,#jptech.ssh.server" --service-roles "@jptech.ssh"


ziti edge create identity device linux.jptech.ziti --role-attributes jptech.ssh.server,jptech.admin -o linux.jptech.ziti.jwt
ziti edge create identity device mac1.jptech.ziti --role-attributes jptech.admin -o mac1.jptech.ziti.jwt
ziti edge create identity device mac2.jptech.ziti --role-attributes jptech.ssh.server -o mac2.jptech.ziti.jwt
ziti edge create identity device win.jptech.ziti --role-attributes jptech.ssh.server -o win.jptech.ziti.jwt
ziti edge create identity device falkor.jptech.ziti --role-attributes jptech.ssh.server,jptech.admin -o falkor.jptech.ziti.jwt
ziti edge create identity device macbian.jptech.ziti --role-attributes jptech.ssh.server,jptech.admin -o macbian.jptech.ziti.jwt



# Install the tunneler

(
set -euo pipefail

curl -sSLf https://get.openziti.io/tun/package-repos.gpg \
  | sudo gpg --dearmor --output /usr/share/keyrings/openziti.gpg

echo 'deb [signed-by=/usr/share/keyrings/openziti.gpg] https://packages.openziti.org/zitipax-openziti-deb-stable jammy main' \
  | sudo tee /etc/apt/sources.list.d/openziti.list >/dev/null

sudo apt update
sudo apt install ziti-edge-tunnel
)

sudo mv *.jwt /opt/openziti/etc/identities/
sudo chown -cR :ziti        /opt/openziti/etc/identities
sudo chmod -cR ug=rwX,o-rwx /opt/openziti/etc/identities

sudo systemctl enable --now ziti-edge-tunnel.service
sudo systemctl restart ziti-edge-tunnel.service
12

If you're comfortable keeping track if the assigned IP addresses for your wildcard domain then that certainly works for the ziti services. However it looks like the system can't resolve your ziti controller's hostname, and therefor the tunneler won't be able to get its list of services. I believe this was from ziti-edge-tunnel on your macbian host:

ERROR ziti-sdk:ziti_ctrl.c:154 ctrl_resp_cb() ctrl[ziti.jptech.ninja] request failed: -3008(unknown node or service)

I'm guessing you can't resolve other hostnames too, e.g. curl https://openziti.discourse.group/ when this happens. Looking at your resolv.conf I think I see why. The traditional resolv.conf resolver uses a single name server for resolving all queries until that name server fails entirely (e.g. cannot be reached or perhaps some number of failed requests). Once the resolver decides that the current name server is lame it starts using the next one that it knows about for all requests. And so on.

Other resolvers work differently. For example systemd-resolved lets you define DNS servers for each network interface on the hosts. By default systemd-resolved sends queries to the DNS servers for all of the interfaces simultaneously, and uses the first successful response regardless of which DNS server provided it. This works nicely for using DNS servers that only know a limited set of hostnames.

Going back to your resolv.conf - it starts with ziti-edge-tunnel's DNS server IP. This means that it will be used for all hostname lookups, probably until you stop ziti-edge-tunnel and the server no longer responds. Once that happens the resolver will start using 10.1.1.1 and I'd guess your non-ziti lookups start working again, but you won't be able to look up ziti hostnames.

You probably want to tell ziti-edge-tunnel about your upstream DNS server (with the --dns-upstream or -u option), so queries for non-ziti hostnames can be forwarded to a server that has answers. You could use a systemd drop-in to change the command line options without copying the entire service unit:

# mkdir /etc/systemd/system/ziti-edge-tunnel.service.d
# echo 'ExecStart=/opt/openziti/bin/ziti-edge-tunnel run --verbose=${ZITI_VERBOSE} --dns-ip-range=${ZITI_DNS_IP_RANGE} --identity-dir=${ZITI_IDENTITY_DIR} --dns-upstream 10.1.1.1' > /etc/systemd/system/ziti-edge-tunnel.service.d/50-add-dns-upstream.conf
# systemctl daemon-reload
# systemctl restart ziti-edge-tunnel.service

So that's DNS.

I think the other major issue you're having is the inability to bind the service for more than one identity? I'm wondering if that is a bug... Which controller and router versions are you running? I'm going to try an environment with multiple identities here.

13

Actually, I did have some resolution issues. The Macbian is running gnome, and I am used to editing resolv.conf directly, and the wifi manager didn't like it, so it wiped out the nameservers. So we can ignore that. The rest of the hosts have basic DNS resolution and can access the controller.

I am not certain if this info below is enough for your testing. I have been doing a docker compose down -v && rm -rf ziti-fs/* && docker system prune -a && docker compose up -d each time I iterate on the environment. So I think I am pulling down a new image each time. But, maybe I need to do a docker pull to get something specific?

# ziti controller version
v0.29.0

# ziti edge version
Version     : v0.29.0
GIT revision: 3ca2dd2f4e7b
Build Date  : 2023-07-13T15:53:37Z
Runtime     : go1.20.5

# cat .env
# OpenZiti Variables
ZITI_IMAGE=openziti/quickstart
ZITI_VERSION=latest

# the user and password to use
# Leave password blank to have a unique value generated or set the password explicitly
ZITI_USER=admin
ZITI_PWD="ahahah,youdidn'tsaythemagicword" (not really)

# controller name, address/port information
ZITI_CTRL_NAME=ziti-controller
ZITI_CTRL_EDGE_ADVERTISED_ADDRESS=ziti.jptech.ninja
ZITI_CTRL_ADVERTISED_ADDRESS=ziti.jptech.ninja
ZITI_CTRL_EDGE_IP_OVERRIDE=45.79.71.170
ZITI_CTRL_EDGE_ADVERTISED_PORT=8441
ZITI_CTRL_ADVERTISED_PORT=8440

# The duration of the enrollment period (in minutes), default if not set. shown - 7days
ZITI_EDGE_IDENTITY_ENROLLMENT_DURATION=10080
ZITI_ROUTER_ENROLLMENT_DURATION=10080

# router address/port information
ZITI_ROUTER_NAME=ziti-edge-router
ZITI_ROUTER_ADVERTISED_HOST=ziti.jptech.ninja
ZITI_ROUTER_PORT=8442
ZITI_ROUTER_IP_OVERRIDE=45.79.71.170
ZITI_ROUTER_LISTENER_BIND_PORT=8444
ZITI_ROUTER_ROLES=public

version: '2.4'
services:
  ziti-controller:
    image: "${ZITI_IMAGE}:${ZITI_VERSION}"
    env_file:
      - ./.env
    ports:
      - ${ZITI_CTRL_EDGE_ADVERTISED_PORT:-1280}:${ZITI_CTRL_EDGE_ADVERTISED_PORT:-1280}
      - ${ZITI_CTRL_ADVERTISED_PORT:-6262}:${ZITI_CTRL_ADVERTISED_PORT:-6262}
    environment:
      - ZITI_CTRL_NAME=${ZITI_CTRL_NAME:-ziti-edge-controller}
      - ZITI_CTRL_EDGE_ADVERTISED_ADDRESS=${ZITI_CTRL_EDGE_ADVERTISED_ADDRESS:-ziti-edge-controller}
      - ZITI_CTRL_EDGE_ADVERTISED_PORT=${ZITI_CTRL_EDGE_ADVERTISED_PORT:-1280}
      - ZITI_CTRL_EDGE_IP_OVERRIDE=${ZITI_CTRL_EDGE_IP_OVERRIDE:-127.0.0.1}
      - ZITI_CTRL_ADVERTISED_PORT=${ZITI_CTRL_ADVERTISED_PORT:-6262}
      - ZITI_EDGE_IDENTITY_ENROLLMENT_DURATION=${ZITI_EDGE_IDENTITY_ENROLLMENT_DURATION}
      - ZITI_ROUTER_ENROLLMENT_DURATION=${ZITI_ROUTER_ENROLLMENT_DURATION}
      - ZITI_USER=${ZITI_USER:-admin}
      - ZITI_PWD=${ZITI_PWD}
    networks:
      ziti:
        aliases:
          - ziti-edge-controller
    volumes:
      - ./ziti-fs:/persistent
    entrypoint:
      - "/var/openziti/scripts/run-controller.sh"

  ziti-controller-init-container:
    image: "${ZITI_IMAGE}:${ZITI_VERSION}"
    depends_on:
      - ziti-controller
    environment:
      - ZITI_CTRL_EDGE_ADVERTISED_ADDRESS=${ZITI_CTRL_EDGE_ADVERTISED_ADDRESS:-ziti-edge-controller}
      - ZITI_CTRL_EDGE_ADVERTISED_PORT=${ZITI_CTRL_EDGE_ADVERTISED_PORT:-1280}
    env_file:
      - ./.env
    networks:
      ziti:
    volumes:
      - ./ziti-fs:/persistent
    entrypoint:
      - "/var/openziti/scripts/run-with-ziti-cli.sh"
    command:
      - "/var/openziti/scripts/access-control.sh"

  ziti-edge-router:
    image: "${ZITI_IMAGE}:${ZITI_VERSION}"
    env_file:
      - ./.env
    depends_on:
      - ziti-controller
    ports:
      - ${ZITI_ROUTER_PORT:-3022}:${ZITI_ROUTER_PORT:-3022}
      - ${ZITI_ROUTER_LISTENER_BIND_PORT:-10080}:${ZITI_ROUTER_LISTENER_BIND_PORT:-10080}
    environment:
      - ZITI_CTRL_ADVERTISED_ADDRESS=${ZITI_CTRL_ADVERTISED_ADDRESS:-ziti-controller}
      - ZITI_CTRL_ADVERTISED_PORT=${ZITI_CTRL_ADVERTISED_PORT:-6262}
      - ZITI_CTRL_EDGE_ADVERTISED_ADDRESS=${ZITI_CTRL_EDGE_ADVERTISED_ADDRESS:-ziti-edge-controller}
      - ZITI_CTRL_EDGE_ADVERTISED_PORT=${ZITI_CTRL_EDGE_ADVERTISED_PORT:-1280}
      - ZITI_ROUTER_NAME=${ZITI_ROUTER_NAME:-ziti-edge-router}
      - ZITI_ROUTER_ADVERTISED_HOST=${ZITI_ROUTER_ADVERTISED_HOST:-ziti-edge-router}
      - ZITI_ROUTER_PORT=${ZITI_ROUTER_PORT:-3022}
      - ZITI_ROUTER_LISTENER_BIND_PORT=${ZITI_ROUTER_LISTENER_BIND_PORT:-10080}
      - ZITI_ROUTER_ROLES=public
    networks:
      - ziti
    volumes:
      - ./ziti-fs:/persistent
    entrypoint: /bin/bash
    command: "/var/openziti/scripts/run-router.sh edge"

  ziti-console:
    image: openziti/zac
    working_dir: /usr/src/app
    environment:
      - ZAC_SERVER_CERT_CHAIN=/persistent/pki/${ZITI_CTRL_EDGE_ADVERTISED_ADDRESS:-ziti-edge-controller}-intermediate/certs/${ZITI_CTRL_EDGE_ADVERTISED_ADDRESS:-ziti-edge-controller}-server.cert
      - ZAC_SERVER_KEY=/persistent/pki/${ZITI_CTRL_EDGE_ADVERTISED_ADDRESS:-ziti-edge-controller}-intermediate/keys/${ZITI_CTRL_EDGE_ADVERTISED_ADDRESS:-ziti-edge-controller}-server.key
      - ZITI_CTRL_EDGE_ADVERTISED_ADDRESS=${ZITI_CTRL_EDGE_ADVERTISED_ADDRESS:-ziti-edge-controller}
      - ZITI_CTRL_EDGE_ADVERTISED_PORT=${ZITI_CTRL_EDGE_ADVERTISED_PORT:-1280}
      - ZITI_CTRL_NAME=${ZITI_CTRL_NAME:-ziti-edge-controller}
      - PORTTLS=8443
    depends_on:
      - ziti-controller
    ports:
      - 8443:8443
    volumes:
      - ./ziti-fs:/persistent
    networks:
      - ziti

networks:
  ziti:

    # volumes:
    #   ziti-fs:
# create a local ziti-fs folder with `mkdir ziti-fs && chmod 777 ziti-fs`, I don't have time to figure out the permissions right now, don't judgeme.
14

Ijust tried from my ipad to falkor (ziti desktop) and it let me connect fine. Perhaps it’s in the Linux tunneler?

Edit: I also downloaded the raw binaries and ran the tunnels manually with ziti-edge-tunnel run --identity-dir /opt/openziti/etc/identities and nothing is changed. I will try tomorrow with another OS/VM for grins.

Edit 2: I am happy to tar up and share my ziti-fs with you if you think it will help.

15

I tried a test with a newer ziti-edge-tunnel than you are using and although I’m not seeing exactly what you see, I’m unable to get more than one terminator for the service even though multiple identities are binding it.

At any point, such as when your iPad->Mac test worked, do you see more than one terminator from the ziti cli? This should be the case - you should see a terminator for each identity that offers the ssh service.

I’m going to shuffle some things around today and see if I can pinpoint what’s going on, and where.

16

The problem that prevents multiple identities from binding the same service is being tracked with this issue: Lookup of terminators with same instance id isn't filtering by instance id · Issue #766 · openziti/fabric · GitHub

My colleague has already made a fix that I have tested, with a development build of the controller. I’m not sure when this change will be folded into a release. Hopefully soon but probably not today. I’ll work on getting this fix into a docker image so you can kick the tires.

17

tenor-226420165

Thanks, I am glad it wasn’t just my noobness!

18

Hehe, cool. This allows us to delete 20+ services :slight_smile:
I’ve thought about how to do this myself but wasn’t aware of a possibility until now.

19

fyi the quickstart docker image with the fix mentioned above is available.You should be able to set ZITI_IMAGE=openziti/quickstart:fabric-766-fix-amd64 in the .env file.

20

I think I might still need some help, it still isn’t working for me. I can ssh from my iPad to my mac (falkor), but not any of the linux vms. Here is what I have done.

I have done the following to upgrade my environment

cat .env
# OpenZiti Variables
ZITI_IMAGE=openziti/quickstart
ZITI_VERSION=fabric-766-fix-amd64

docker compose down -v
docker system prune -a
docker compose pull
docker compose up -d

I confirmed that ziti is up again.

I ssh’d into all the vms [linux,win,mac1,mac2].jptech.corp and confirmed the tunneler status is green. I tailed the logs with journalctl -u ziti-edge-tunnel -f. Then is did a ssh ...@jptech.zit and got all ssh: connect to host mac1.jptech.ziti port 22: Connection refused. I did a dig mac1.jptech.ziti @100.64.0.2 to confirm the IP addresses from my host and then a ssh root@100.64.0.8 and got the same connection refused.

Below is the screenshot of the tail of the log BEFORE attempting to connect via ssh to the devices and after a restart of the service.

image
image1328×1316 525 KB

The following is a screenshot AFTER attempting to ssh to the ziti services. I don’t see a connection request or denial… nothing shows on the tunnel log.

image
image1328×1316 525 KB

Here is my ziti desktop log on my mac, I cleared it BEFORE connection attempts.

image
image658×652 107 KB

Here is the log AFTER I make the connection attempt.

image
image658×652 188 KB

Here is what my controller looks like, as far as identities, services and terminators.

jp@falkor ~ $ ziti edge list identities
╭────────────┬─────────────────────┬────────┬────────────────────────────────┬─────────────╮
│ ID         │ NAME                │ TYPE   │ ATTRIBUTES                     │ AUTH-POLICY │
├────────────┼─────────────────────┼────────┼────────────────────────────────┼─────────────┤
│ 1hXBqSryqw │ falkor.jptech.ziti  │ Device │ jptech.admin,jptech.ssh.server │ Default     │
│ AxLx9Bi-D  │ Default Admin       │ User   │                                │ Default     │
│ CgbB7hrY7w │ mac1.jptech.ziti    │ Device │ jptech.ssh.server              │ Default     │
│ JSABqhry7  │ linux.jptech.ziti   │ Device │ jptech.ssh.server              │ Default     │
│ JeXd7Sry7  │ macbian.jptech.ziti │ Device │ jptech.admin,jptech.ssh.server │ Default     │
│ Ml5U-IzcG  │ admin.jp            │ User   │                                │ Default     │
│ Yff-7SFyqw │ ziti-edge-router    │ Router │                                │ Default     │
│ b7NoaVzZG  │ jpipad              │ Device │ jptech.admin                   │ Default     │
│ ifLd7hryq  │ win.jptech.ziti     │ Device │ jptech.ssh.server              │ Default     │
│ mjAB7SrYq  │ mac2.jptech.ziti    │ Device │ jptech.ssh.server              │ Default     │
╰────────────┴─────────────────────┴────────┴────────────────────────────────┴─────────────╯
results: 1-10 of 10
jp@falkor ~ $ ziti edge list services
╭────────────────────────┬────────────┬────────────┬─────────────────────┬───────────────────╮
│ ID                     │ NAME       │ ENCRYPTION │ TERMINATOR STRATEGY │ ATTRIBUTES        │
│                        │            │  REQUIRED  │                     │                   │
├────────────────────────┼────────────┼────────────┼─────────────────────┼───────────────────┤
│ 6dxVqc21YVpJ2kuGDtnkt3 │ jptech.ssh │ true       │ smartrouting        │ jptech.admin      │
│                        │            │            │                     │ jptech.ssh.server │
╰────────────────────────┴────────────┴────────────┴─────────────────────┴───────────────────╯
results: 1-1 of 1
jp@falkor ~ $ ziti edge list terminators
╭────────────────────────┬────────────┬──────────────────┬─────────┬─────────────────────────────────────────────┬────────────────────┬──────┬────────────┬──────────────╮
│ ID                     │ SERVICE    │ ROUTER           │ BINDING │ ADDRESS                                     │ IDENTITY           │ COST │ PRECEDENCE │ DYNAMIC COST │
├────────────────────────┼────────────┼──────────────────┼─────────┼─────────────────────────────────────────────┼────────────────────┼──────┼────────────┼──────────────┤
│ 39uYskaOtAetSZt5XV0X8y │ jptech.ssh │ ziti-edge-router │ edge    │ hosted:dc10692a-3f1d-4ff6-a423-32d128beda3b │ falkor.jptech.ziti │    0 │ default    │            0 │
╰────────────────────────┴────────────┴──────────────────┴─────────┴─────────────────────────────────────────────┴────────────────────┴──────┴────────────┴──────────────╯
results: 1-1 of 1
jp@falkor ~ $ ziti edge list service-policies
╭────────────────────────┬─────────────────┬──────────┬───────────────┬──────────────────────────────────┬─────────────────────╮
│ ID                     │ NAME            │ SEMANTIC │ SERVICE ROLES │ IDENTITY ROLES                   │ POSTURE CHECK ROLES │
├────────────────────────┼─────────────────┼──────────┼───────────────┼──────────────────────────────────┼─────────────────────┤
│ 1aGQCqMP2iYiynVG7CbMS6 │ jptech.ssh.bind │ AllOf    │ @jptech.ssh   │ #jptech.admin #jptech.ssh.server │                     │
│ 2eWzdCIkzhdQHzsg5utyBi │ jptech.ssh.dial │ AllOf    │ @jptech.ssh   │ #jptech.admin                    │                     │
╰────────────────────────┴─────────────────┴──────────┴───────────────┴──────────────────────────────────┴─────────────────────╯
results: 1-2 of 2
jp@falkor ~ $ ziti edge list configs
╭────────────────────────┬──────────────────────────┬──────────────╮
│ ID                     │ NAME                     │ CONFIG TYPE  │
├────────────────────────┼──────────────────────────┼──────────────┤
│ 7J4Npd8Kmj6jG0aoifF0x8 │ jptech.ssh.cfg.host      │ host.v1      │
│ 929pxd0ARCRgIYEm7okt   │ jptech.ssh.cfg.intercept │ intercept.v1 │
╰────────────────────────┴──────────────────────────┴──────────────╯
results: 1-2 of 2
jp@falkor ~ $ ziti edge list summary
╭───────────────────────────┬───────╮
│ ENTITY TYPE               │ COUNT │
├───────────────────────────┼───────┤
│ apiSessionCertificates    │    22 │
│ apiSessions               │    25 │
│ authPolicies              │     1 │
│ authenticators            │     8 │
│ cas                       │     0 │
│ configTypes               │     5 │
│ configs                   │     2 │
│ edgeRouterPolicies        │     2 │
│ enrollments               │     2 │
│ eventualEvents            │     0 │
│ externalJwtSigners        │     0 │
│ identities                │    10 │
│ identityTypes             │     4 │
│ mfas                      │     0 │
│ postureCheckTypes         │     5 │
│ postureChecks             │     0 │
│ routers                   │     1 │
│ routers.edge              │     1 │
│ serviceEdgeRouterPolicies │     1 │
│ servicePolicies           │     2 │
│ services                  │     1 │
│ services.edge             │     1 │
│ sessions                  │     2 │
│ terminators               │     1 │
╰───────────────────────────┴───────╯