I apologize for intruding into JP's question but can you please explain what dialOptions and listenOptions are for ? Never saw these before on the forum nor on the documentation.
How can I create a service for ssh and add identities with the least amount of policies/configs/etc?
My apologies. Our automated build process for these images doesn’t directly support building/pushing ziti versions that won’t be released. E.g. the automated builds always push the :latest tag. So I built the image manually and I made a mistake in putting the fixed binary in a place where it wouldn’t be picked up by the docker build.
I’ve rebuilt and re-pushed the image. Please try again. I’ll be watching this thread closely incase we’re not on track.
Sure. dial and listen options let you specify some parameters that will be passed to the ziti SDK dial or bind function when the tunneler intercepts a connection or begins hosting a service. The schema for these fields are defined in the intercept.v1 and host.v1 config type entities:
"dialOptions": {
"additionalProperties": false,
"properties": {
"connectTimeoutSeconds": {
"$ref": "#/definitions/timeoutSeconds",
"description": "defaults to 5 seconds if no dialOptions are defined. defaults to 15 if dialOptions are defined but connectTimeoutSeconds is not specified."
},
"identity": {
"description": "Dial a terminator with the specified identity. '$dst_protocol', '$dst_ip', '$dst_port', and '$dst_hostname' are resolved to the corresponding value of the destination address.",
"type": "string"
}
},
"type": "object"
},
"listenOptions": {
"additionalProperties": false,
"properties": {
"bindUsingEdgeIdentity": {
"description": "Associate the hosting terminator with the name of the hosting tunneler's identity. Setting this to 'true' is equivalent to setting 'identiy=$tunneler_id.name'",
"type": "boolean"
},
"connectTimeoutSeconds": {
"$ref": "#/definitions/timeoutSeconds",
"description": "defaults to 5"
},
"cost": {
"description": "defaults to 0",
"maximum": 65535,
"minimum": 0,
"type": "integer"
},
"identity": {
"description": "Associate the hosting terminator with the specified identity. '$tunneler_id.name' resolves to the name of the hosting tunneler's identity. '$tunneler_id.tag[tagName]' resolves to the value of the 'tagName' tag on the hosting tunneler's identity.",
"type": "string"
},
"maxConnections": {
"description": "defaults to 3",
"minimum": 1,
"type": "integer"
},
"precedence": {
"description": "defaults to 'default'",
"enum": [
"default",
"required",
"failed"
],
"type": "string"
}
},
"type": "object"
},
1 Like
Just to clarify, pull latest or fabric-766-fix-amd64?
fabric-766-fix-amd64
I get this in the logs of my mac
[2023-08-10T22:01:31:527Z] INFO PacketTunnelProvider:UserNotifications.swift:100 post() Attempting to post Info notification, subitile:Optional("Controller: Available"), body:Optional("falkor.jptech.ziti\nhttps://ziti.jptech.ninja:8441"), zid:Optional("falkor.jptech.ziti")
[2023-08-10T22:01:31:528Z] WARN PacketTunnelProvider:UserNotifications.swift:107 post() Not authorized to send notifications
(98340)[2023-08-10T22:01:37.705Z] INFO ziti-sdk:posture.c:204 ziti_send_posture_data() ztx[0] first run or potential controller restart detected
(98340)[2023-08-10T22:02:07.168Z] INFO ziti-sdk:channel.c:234 new_ziti_channel() ch[4] (ziti-edge-router@tls://ziti.jptech.ninja:8442) new channel for ztx[0] identity[falkor.jptech.ziti]
(98340)[2023-08-10T22:02:07.168Z] INFO tunnel-cbs:ziti_tunnel_ctrl.c:797 on_ziti_event() ztx[falkor.jptech.ziti] added edge router ziti-edge-router@tls://ziti.jptech.ninja:8442@ziti.jptech.ninja
[2023-08-10T22:02:07:179Z] INFO PacketTunnelProvider:ZitiTunnelDelegate.swift:195 excludeRoute() excludeRoute 45.79.71.170 => 45.79.71.170, 255.255.255.255
[2023-08-10T22:02:07:179Z] INFO PacketTunnelProvider:ZitiTunnelDelegate.swift:195 excludeRoute() excludeRoute 45.79.71.170 => 45.79.71.170, 255.255.255.255
(98340)[2023-08-10T22:02:07.168Z] INFO ziti-sdk:channel.c:733 reconnect_channel() ch[4] reconnecting NOW
(98340)[2023-08-10T22:02:07.540Z] INFO ziti-sdk:channel.c:631 hello_reply_cb() ch[4] connected. EdgeRouter version: v0.0.0|local|2020-01-01 01:01:01|linux|amd64
(98340)[2023-08-10T22:02:07.540Z] INFO tunnel-cbs:ziti_tunnel_ctrl.c:801 on_ziti_event() ztx[falkor.jptech.ziti] router ziti-edge-router@tls://ziti.jptech.ninja:8442 connected
(98340)[2023-08-10T22:02:37.553Z] INFO tunnel-cbs:ziti_dns.c:500 format_resp() found record[100.64.0.6] for query[1:mac1.jptech.ziti]
(98340)[2023-08-10T22:02:37.553Z] INFO tunnel-cbs:ziti_dns.c:500 format_resp() found record[100.64.0.5] for query[1:mac2.jptech.ziti]
(98340)[2023-08-10T22:02:37.554Z] INFO tunnel-cbs:ziti_dns.c:500 format_resp() found record[100.64.0.7] for query[1:linux.jptech.ziti]
(98340)[2023-08-10T22:02:37.556Z] INFO tunnel-cbs:ziti_dns.c:500 format_resp() found record[100.64.0.4] for query[1:win.jptech.ziti]
(98340)[2023-08-10T22:02:37.643Z] ERROR ziti-sdk:connect.c:965 connect_reply_cb() conn[0.28/Connecting] failed to connect, reason=service 6dxVqc21YVpJ2kuGDtnkt3 has no terminators for instanceId win.jptech.ziti
(98340)[2023-08-10T22:02:37.643Z] ERROR tunnel-cbs:ziti_tunnel_cbs.c:103 on_ziti_connect() ziti dial failed: connection is closed
(98340)[2023-08-10T22:02:37.643Z] ERROR ziti-sdk:connect.c:965 connect_reply_cb() conn[0.27/Connecting] failed to connect, reason=service 6dxVqc21YVpJ2kuGDtnkt3 has no terminators for instanceId linux.jptech.ziti
(98340)[2023-08-10T22:02:37.643Z] ERROR tunnel-cbs:ziti_tunnel_cbs.c:103 on_ziti_connect() ziti dial failed: connection is closed
(98340)[2023-08-10T22:02:37.646Z] ERROR ziti-sdk:connect.c:965 connect_reply_cb() conn[0.25/Connecting] failed to connect, reason=service 6dxVqc21YVpJ2kuGDtnkt3 has no terminators for instanceId mac2.jptech.ziti
(98340)[2023-08-10T22:02:37.646Z] ERROR tunnel-cbs:ziti_tunnel_cbs.c:103 on_ziti_connect() ziti dial failed: connection is closed
(98340)[2023-08-10T22:02:37.653Z] ERROR ziti-sdk:connect.c:965 connect_reply_cb() conn[0.26/Connecting] failed to connect, reason=service 6dxVqc21YVpJ2kuGDtnkt3 has no terminators for instanceId mac1.jptech.ziti
(98340)[2023-08-10T22:02:37.653Z] ERROR tunnel-cbs:ziti_tunnel_cbs.c:103 on_ziti_connect() ziti dial failed: connection is closed
(98340)[2023-08-10T22:02:46.868Z] INFO tunnel-cbs:ziti_hosting.c:409 on_hosted_client_connect() hosted_service[jptech.ssh], client[jpipad] dst_addr[tcp:falkor.jptech.ziti:22]: incoming connection
(98340)[2023-08-10T22:03:02.536Z] INFO tunnel-cbs:ziti_hosting.c:409 on_hosted_client_connect() hosted_service[jptech.ssh], client[jpipad] dst_addr[tcp:falkor.jptech.ziti:22]: incoming connection
(98340)[2023-08-10T22:03:21.410Z] INFO tunnel-cbs:ziti_dns.c:500 format_resp() found record[100.64.0.3] for query[1:asdf.jptech.ziti]
I can ssh to my mac from my iPad, but nothing can ssh into the linux VMs.
Thse controller is scrolling non-stop this error
ziti-controller_1 | [ 227.570] ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:8441]: {error=[EOF]} handshake failed
Did you clear your docker cache and/or re-pull the image (at the fabric-766-fix-amd64 tag)? That’s my best guess right now.
Is there anything in the logs from the linux tunnelers? I’m guessing you still only have a single terminator that represents the first of the #ssh identities that started?
BTW we can forget about this docker tag now. The 0.30 release was published a few hours ago and a docker image was pushed (to ‘:latest’).
I believe so, unless there is a step I'm missing.
docker compose down -v
docker system prune -a
docker pull openziti/quickstart:fabric-766-fix-amd64
docker compose up -d
The only thing I haven't done is delete the volumes. I know the dash v implies deleting the volumes, but I'm redirecting the volume binds to a folder. If it would help I can blow the whole thing away and start fresh.
I'll double check my logs on the identities when I get home.
Ok thanks. One thing that could explain those handshake errors in your controller log is if you have any identities that were enrolled with a previous incarnation of the controller trying to connect after the controller was reinitialized.
Just in case I’ll blow it away and start fresh.
We are golden… it works. I did a clean install of the whole shebang and ssh of the linux hosts works great.
Thank you so much for the help. I will write up a howto on this not before too long.
1 Like
Awesome! Thanks for checking out openziti. I look forward to seeing your how-to when you get to it!
Edit: I got it again. There is something really delicate I am just not getting. I am gonna rinse and repeat till I can figure it out.
I am wrapping up my docs, but I found an interesting twist. It seems I can only get it working if I am using a controller on the public cloud. I can’t make it work via a local VM. I can get all the tunnelers to handshake. But any attempts to dial the server gives an error that there is no terminator for that service. It pings/resolves. If I do the exact same setup on a public cloud it works just fine.
Any thoughts?
Edit: I can’t reproduce it working now. I get this error.
(86285)[2023-08-12T01:08:48.041Z] INFO tunnel-cbs:ziti_dns.c:500 format_resp() found record[100.64.0.3] for query[1:linux1.jptech.ziti]
(86285)[2023-08-12T01:08:48.123Z] ERROR ziti-sdk:connect.c:965 connect_reply_cb() conn[0.8/Connecting] failed to connect, reason=service 7aNqEvnkF55PNUKVkLKGa1 has no terminators for instanceId linux1.jptech.ziti
(86285)[2023-08-12T01:08:48.123Z] ERROR tunnel-cbs:ziti_tunnel_cbs.c:103 on_ziti_connect() ziti dial failed: connection is closed
Last question @scareything
Am I correct in assuming that the identity name must match the *.jptech.ziti pattern, otherwise you can’t connect to it? I tested this by creating a couple of identities whose name is simply admin and linux, and I can see the policies have been applied and the services are OK. But it doesn’t actually work.
i.e. these will work
$ zli
╭────────────┬───────────────────┬────────┬──────────────┬─────────────╮
│ ID │ NAME │ TYPE │ ATTRIBUTES │ AUTH-POLICY │
├────────────┼───────────────────┼────────┼──────────────┼─────────────┤
│ 0vok.hYa.q │ linux.jptech.ziti │ Device │ jptech.ssh │ Default │
│ yAik.hYc.q │ admin.jptech.ziti │ Device │ jptech.admin │ Default │
╰────────────┴───────────────────┴────────┴──────────────┴─────────────╯
results: 1-4 of 4
these will not
$ zli
╭────────────┬──────────────────┬────────┬──────────────┬─────────────╮
│ ID │ NAME │ TYPE │ ATTRIBUTES │ AUTH-POLICY │
├────────────┼──────────────────┼────────┼──────────────┼─────────────┤
│ PvYg9Pbc3 │ admin │ Device │ jptech.admin │ Default │
│ RuVd91ba3q │ linux │ Device │ jptech.ssh │ Default │
╰────────────┴──────────────────┴────────┴──────────────┴─────────────╯
Here is my error in the tunnel of the dialing identity.
Aug 13 01:11:59 admin ziti-edge-tunnel[568]: (568)[ 219.256] ERROR ziti-sdk:connect.c:974 connect_reply_cb() conn[0.1/Connecting] failed to connect, reason=service 3K6t788tyr3slYiPdvHk7B has no terminators for instanceId linux.jptech.ziti
$ ziti policy-advisor identities -q
ERROR: Default Admin
- Identity does not have access to any services. Adjust service policies.
OKAY : admin (1) -> jptech.ssh (1) Common Routers: (1/1) Dial: Y Bind: N
OKAY : linux (1) -> jptech.ssh (1) Common Routers: (1/1) Dial: N Bind: Y
ERROR: ziti-edge-router
- Identity does not have access to any services. Adjust service policies.
Yes, the identity names need to match the wildcard domain. You could also create identities that are IP addresses that fall within a cidr range.
The key point is that the intercepting tunneler really only knows the intercepted protocol:IP:port (the IP can be looked up for a hostname match), and so the target identity needs to be expressed with those primitives.
Btw we have some ideas for additional primitives described in support domain substitutions · Issue #540 · openziti/ziti-tunnel-sdk-c · GitHub
1 Like
I commented on the issue, if that’s not a place for that type of comment please let me know, I don’t want to make extra work for anyone.
1 Like
Picking up on this.
If I didn't want a wildcard intercept but instead I'd like to define a service for each SSH client/identity, how would the config/policy configuration look like for the following scenario:
- Attribute
sshfor identities to allow SSH access to respective services - Multiple servers running
ziti-edge-tunnelwhere SSH is hosted and the connection should be let out
I'd love a setup with as few configurations/policies as possible as we currently have 1 service, 2 policies, 2 configs for each SSH server.
Thanks!
Okay, so I've upgraded to the newest version controller but something seems to be not working.
I can see one terminator for each ssh server for each router but a connections is weirdly enough only possible for a single server - the other server won't work and give me a timeout. I can't find any relevant logs on the controller side. For the SSH servers, there are no ziti-edge-tunnel logs...
Hi @dmuensterer.
If for whatever reason you don't want to use wildcard (or CIDR) intercepts for to reach multiple ssh servers then you'll definitely need one service for each server.
The services could all use the same host.v1 configuration if you're targeting sshd at 127.0.0.1, but each service will need its own intercept configuration with a unique intercept ip and/or port. For example:
ziti edge create config ssh-host1-intercept.v1 intercept.v1 '{"addresses":["20.0.1.11"],"protocols":["tcp"],"portRanges":[{"low":22,"high":22}]}'
ziti edge create config ssh-host2-intercept.v1 intercept.v1 '{"addresses":["20.0.1.12"],"protocols":["tcp"],"portRanges":[{"low":22,"high":22}]}'
ziti edge create config ssh-all-host.v1 host.v1 '{"address":"127.0.0.1","port":22,"protocol":"tcp"}'
And then create the services with references to the configurations. Each service will need its own identity and bind policy (assuming you want proper access control):
ziti edge create service ssh-host1 -c ssh-host1-intercept.v1,ssh-all-host.v1 --role-attributes ssh-services
ziti edge create identity host1
ziti edge create sp ssh-host1-bind Bind --identity-roles '@host1' --service-roles '@ssh-host1'
ziti edge create service ssh-host2 -c ssh-host2-intercept.v1,ssh-all-host.v1 --role-attributes ssh-services
ziti edge create identity host2
ziti edge create sp ssh-host2-bind Bind --identity-roles '@host2' --service-roles '@ssh-host2'
The service role that we added when defining the services makes it possible to share a single dial policy between all of the services. We use a policy that specifies groups for both the identity-roles and the service-roles, so identities with the ssh-dialers identity role can see all of the services that have the ssh-services service role:
ziti edge create sp ssh-any-dial Dial --service-roles '#ssh-services' --identity-roles '#ssh-dialers'
And finally we are ready to create some client identities:
ziti edge create identity client1 --role-attributes ssh-dialers
ziti edge create identity client2 --role-attributes ssh-dialers
And the policy-advisor subcommand will show us a summary of who can do what:
ziti edge create identity client1 --role-attributes ssh-dialers
...
OKAY : client1 (2) -> ssh-host1 (2) Common Routers: (1/2) Dial: Y Bind: N
OKAY : client2 (2) -> ssh-host1 (2) Common Routers: (1/2) Dial: Y Bind: N
OKAY : host1 (2) -> ssh-host1 (2) Common Routers: (1/2) Dial: N Bind: Y
OKAY : client1 (2) -> ssh-host2 (2) Common Routers: (1/2) Dial: Y Bind: N
OKAY : client2 (2) -> ssh-host2 (2) Common Routers: (1/2) Dial: Y Bind: N
OKAY : host2 (2) -> ssh-host2 (2) Common Routers: (1/2) Dial: N Bind: Y
To summarize, we have:
- 1 shared host configuration
- 1 shared dial policy
- as many client identities as you need
And for each service we have:
- 1 intercept configuration
- 1 identity
- 1 bind policy
Please let me know if this doesn't answer your question. I'm also curious to know why you wouldn't want to use a wildcard or CIDR intercept address?
-Shawn
1 Like
This is definitely weird. If you still have this issue could you please tell me a little more? Questions that come to mind for me:
- are you accessing the servers from different clients/tunnelers or are they all on the same box?
- it sounds like this didn't happen before you upgraded the controller? did you also update the routers?
- are there any interesting log messages on the routers?
1 Like