Where to begin with openziti?

Building/Development
21

@bottles , this info :point_up_2: will be helpful, and will you also please verify that the device where the hosting tunneler is running is also able to reach the server that is described by that same host.v1 config (address, port, and protocol from the config). When the packets arrive at the hosting tunneler they will be forwarded to that destination.

Do I understand correctly that the hosting tunneler is running on the same device that is providing the app.example.com application server? In that case I would expect the host.v1 config to have contents like this assuming the app server is listening on 80/tcp on localhost.

{
  "protocol": "tcp",
  "address": "127.0.0.1",
  "port": 80
}

I see these client tunneler log messages.

[     2371.359]   ERROR ziti-sdk:connect.c:906 connect_reply_cb() conn[0.18] failed to connect, reason=service lQNWuZBdSC has no terminators
[     2371.359]   ERROR tunnel-cbs:ziti_tunnel_cbs.c:104 on_ziti_connect() ziti dial failed: connection is closed

This indicates to me that the hosting tunneler is either offline or lacks the necessary policy permissions to join the network and "bind" aka "host" aka "terminate" that particular Ziti service.

22

Do I understand correctly that the hosting tunneler is running on the same device that is providing the app.example.com application server? In that case I would expect the host.v1 config to have contents like this assuming the app server is listening on 80/tcp on localhost.

{
  "protocol": "tcp",
  "address": "127.0.0.1",
  "port": 80
}

Yes, the hosting tunneler and app server is on the same machine.

I have configured the service like this.

root@ip-172-31-16-247:/home/admin# ziti edge list configs
RESTY 2022/04/27 02:45:41 ERROR read /home/admin: is a directory
id:   CU1WuZqgSC
name: normalSvc-intercept.v1
type: intercept.v1
data: {
          "addresses": [
              "app.example.com"
          ],
          "portRanges": [
              {
                  "high": 80,
                  "low": 80
              }
          ],
          "protocols": [
              "tcp"
          ]
      }

id:   nbiWkZqgUC
name: normalSvc-host.v1
type: host.v1
data: {
          "address": "127.0.0.1",
          "port": 80,
          "protocol": "tcp"
      }

results: 1-2 of 2

service policies like this.

root@ip-172-31-16-247:/home/admin# ziti edge list service-policies
RESTY 2022/04/27 02:47:21 ERROR read /home/admin: is a directory
id: FONtkmBgUC    name: normalSvc-dialing    type: Dial    service roles: [@normalSvc]    identity roles: [@normalSvcClient ] posture check roles: []
id: RtNtkmqgU    name: normalSvc-binding    type: Bind    service roles: [@normalSvc]    identity roles: [@normalSvcServer] posture check roles: []

app.example.com and openziti.example.com are on the same subnet (172.31.16.247/20). I can run wget http://app.example.com successfully on openziti.example.com.

root@ip-172-31-16-247:/home/admin# wget http://app.example.com
--2022-04-27 02:38:20--  http://app.example.com
Resolving app.example.com(http://app.example.com)... 3.138.77.34
Connecting to app.example.com (app.example.com)|3.138.77.34|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 612 [text/html]
Saving to: 'index.html'

index.html                                         100%[===============================================================================================================>]     612  --.-KB/s    in 0s
23

i’d change your host.v1 to match what you’re dialing: app.example.com. since it’s on the same machine that “shouldn’t” matter - however… if “app.example.com” has bound to a specific IP address (not 0.0.0.0) you will NEED to supply the ip, 127.0.0.1 won’t work. on the ‘far’ machine - if you dig app.example.com does it show you an ip address? when you look at netstat or ss - do you see port 80 listening on all interfaces

tcp        0      0 0.0.0.0:80            0.0.0.0:*               LISTEN      -

or a single ip?

tcp        0      10.20.30.40:80            0.0.0.0:*               LISTEN      -
24

Yep. The service is listening on all interfaces - 0.0.0.0:80. I just installed nginx on the server and let it running.

on the ‘far’ machine - if you dig app.example.com does it show you an ip address?

I ran this command on both app.example.com and the local machine, they all show the same.

app.example.com.241INA3.138.77.34

25

and that's your local IP address I take it right? humor me - could you change your host.v1 config (using ziti edge update config -d {}) or via ZAC to use "eth0.me:80"?

then on the local machine when you curl to app.example.com it should exit on the 'far' side going to eth0.me:80 and you should get your ip address.

This is a very common sample I run - and have running right now.

cd@172.27.89.105:sg3: ~$ curl eth0.me
174.210.70.48
cd@172.27.89.105:sg3: ~$ curl eth0.ziti
129.80.199.88

curl'ing to eth0.me shows my ACTUAL ip address - while eth0.ziti (notice .ziti is not a valid top level domain, cool eh? :slight_smile: ) shows the IP of my edge router (where it's leaving my overlay network)

You're still getting "timeouts" when curling to app.example.com? Would you try that just to see what happens?

Maybe I'm missing something you're doing. I can give you the exact set of ziti cli commands if needed but it's similar to the ones before from Where to begin with openziti? - #11 by TheLumberjack.

26

and that’s your local IP address I take it right?

I think that's the public IP address of app.example.com.

could you change your host.v1 config (using ziti edge update config -d {}) or via ZAC to use “eth0.me:80”?

okay, I changed it to be

type: intercept.v1
data: {
          "addresses": [
              "app.example.com"
          ],
          "portRanges": [
              {
                  "high": 80,
                  "low": 80
              }
          ],
          "protocols": [
              "tcp"
          ]
      }

name: normalSvc-host.v1.eth0.me
type: host.v1
data: {
          "address": "eth0.me",
          "port": 80,
          "protocol": "tcp"
      }

I recreated the service and added it to the policies again. Now I ran the command wget http://app.example.com on local machine. I got this error.

[       65.710]   ERROR tunnel-cbs:ziti_tunnel_cbs.c:104 on_ziti_connect() ziti dial failed: Operation did not complete in time
[       65.729]   ERROR ziti-sdk:channel.c:464 dispatch_message() ch[0] could not find waiter for reply_to = 3

I checked ziti-router on openziti.example.com and I got this error again.

Apr 27 04:36:15 ip-172-31-16-247 ziti-router[575]: {"_channels":["establishPath"],"apiSessionId":"cl2h3267i04mceeqrv95x0clf","attempt":0,"attemptNumber":"1","binding":"edge","circuitId":"GD9HaDbpZ","context":"ch{ctrl}-\u003eu{reconnecting}-\u003ei{n7l5}","destination":"hosted:fb513c94-0727-4ea9-ba5f-fe298587011a","error":"error creating route for [c/GD9HaDbpZ]: timeout waiting for message reply: context deadline exceeded","file":"github.com/openziti/fabric@v0.17.88/router/handler_ctrl/route.go:120","func":"github.com/openziti/fabric/router/handler_ctrl.(*routeHandler).fail","level":"error","msg":"failed to connect egress","sessionId":"cl2h32e0404mqeeqrjip6hz1y","time":"2022-04-27T04:36:15.089Z"}
Apr 27 04:36:20 ip-172-31-16-247 ziti-router[575]: {"_channels":["establishPath"],"apiSessionId":"cl2h3267i04mceeqrv95x0clf","attempt":1,"attemptNumber":"2","binding":"edge","circuitId":"GD9HaDbpZ","context":"ch{ctrl}-\u003eu{reconnecting}-\u003ei{n7l5}","destination":"hosted:fb513c94-0727-4ea9-ba5f-fe298587011a","error":"error creating route for [c/GD9HaDbpZ]: timeout waiting for message reply: context deadline exceeded","file":"github.com/openziti/fabric@v0.17.88/router/handler_ctrl/route.go:120","func":"github.com/openziti/fabric/router/handler_ctrl.(*routeHandler).fail","level":"error","msg":"failed to connect egress","sessionId":"cl2h32e0404mqeeqrjip6hz1y","time":"2022-04-27T04:36:20.092Z"}
Apr 27 04:36:20 ip-172-31-16-247 ziti-router[575]: {"_context":"ch{edge}-\u003eu{classic}-\u003ei{W8EP}","chSeq":6,"connId":2,"edgeSeq":0,"error":"exceeded maximum [2] retries creating circuit [c/GD9HaDbpZ]: error creating route for [s/GD9HaDbpZ] on [r/NtcRLAFfm] (error creating route for [c/GD9HaDbpZ]: timeout waiting for message reply: context deadline exceeded)","file":"github.com/openziti/edge@v0.21.169/router/xgress_edge/listener.go:160","func":"github.com/openziti/edge/router/xgress_edge.(*edgeClientConn).processConnect","level":"warning","msg":"failed to dial fabric","time":"2022-04-27T04:36:20.095Z","token":"bc254af1-c8bf-4679-addf-d26e6c559c4f","type":"EdgeConnectType"}

ziti-controller has the following errors.

Apr 27 05:06:25 ip-172-31-16-247 ziti-controller[518]: {"_channels":["establishPath"],"apiSessionId":"cl2h43ext05z0eeqrvxn9hmlf","attemptNumber":1,"circuitId":"ETbwRD6zZ","file":"github.com/openziti/fabric@v0.17.88/controller/network/routesender.go:172","func":"github.com/openziti/fabric/controller/network.(*routeSender).handleRouteSend","level":"warning","msg":"received failed route status from [r/NtcRLAFfm] for attempt [#0] of [s/ETbwRD6zZ] (error creating route for [c/ETbwRD6zZ]: timeout waiting for message reply: context deadline exceeded)","sessionId":"cl2h46zro063xeeqra7wxhb3p","time":"2022-04-27T05:06:25.456Z"}
Apr 27 05:06:25 ip-172-31-16-247 ziti-controller[518]: {"_channels":["selectPath"],"apiSessionId":"cl2h43ext05z0eeqrvxn9hmlf","attemptNumber":1,"circuitId":"ETbwRD6zZ","error":"error creating route for [s/ETbwRD6zZ] on [r/NtcRLAFfm] (error creating route for [c/ETbwRD6zZ]: timeout waiting for message reply: context deadline exceeded)","file":"github.com/openziti/fabric@v0.17.88/controller/network/network.go:430","func":"github.com/openziti/fabric/controller/network.(*Network).CreateCircuit","level":"warning","msg":"route attempt for circuit failed","sessionId":"cl2h46zro063xeeqra7wxhb3p","time":"2022-04-27T05:06:25.457Z"}

I can feel that something wrong in the openziti server, maybe a silly mistake I made before..

I almost copy the same config as what you told me, and differences are I used port 80 instead. I also used slightly different commands in the following.

# enroll the 'server' identity (where you will ssh to)
ziti edge enroll normalSshSvcServer.jwt  **# I used ./ziti-edge-tunnel enroll -j normalSshSvcServer.jwt -i normalSshSvcServer.json**
sudo ./ziti-edge-tunnel run -i normalSshSvcServer.json

# enroll/use this identity in your "desktop edge" or ziti-edge-tunnel
sudo ./ziti-edge-tunnel run normalSshSvcServer.json # I thought this is a typo ?? and I ran the following commands instead on the local client machine. 
# ./ziti-edge-tunnel enroll -j normalSshSvcClient.jwt -i normalSshSvcClient.json
# ./ziti-edge-tunnel run -i normalSshSvcClient.json
27

some update.

I have found a log on ziti-router like this.

Apr 27 14:08:42 ip-172-31-16-247 ziti-router[1921]: {"_channels":["establishPath"],"apiSessionId":"cl2hn6sfm00rfgnqry2yye2sx","attemptNumber":"2","bindConnId":0,"binding":"edge","circuitId":"oJxCLJQGu","destination":"hosted:acb7b84e-372b-4124-8ecc-96058eb083c6","file":"github.com/openziti/edge@v0.21.169/router/xgress_edge/dialer.go:148","func":"github.com/openziti/edge/router/xgress_edge.(*dialer).Dial","level":"debug","msg":"router not assigning connId for dial","sessionId":"cl2hna2co00vsgnqrl6nrp7cg","time":"2022-04-27T14:08:42.134Z"}

I have traced the source code and found out that assignId is not false.

The place to assign assignId is here:

assignIds is false. For some reason edge.RouterProvidedConnId is not correct or something else.

28

I think maybe I should start from creating everything on one node. Having app servics running behind a buston server is a bit too hard to set up.

29

Yeah, let’s start by making sure your environment is setup correctly.

Are you advertising an address that is contactable by both the client and the host? You are correct that it was a typo! oops. i must have copied the wrong window when I pasted it into discourse.

Can you tell me what the “advertise” url of the router looks like for you? for example in my router it looks like:

listeners:
# bindings of edge and tunnel requires an "edge" section below
  - binding: edge
    address: tls:0.0.0.0:8442
    options:
      advertise: ec2-18-188-201-183.us-east-2.compute.amazonaws.com:8442

These exact commands I just ran myself on my environment. Something strange is happening for sure.

# make some identities
ziti edge create identity device eth0.client -o eth0.client.jwt
ziti edge create identity device eth0.server -o eth0.server.jwt

# make the overlay objects
ziti edge create config 'eth0.host.v1' host.v1 '{"protocol":"tcp", "address":"eth0.me","port":80}'
ziti edge create config 'eth0.intercept.v1' intercept.v1 '{"protocols":["tcp"],"addresses":["eth0.discourse.ziti"], "portRanges":[{"low":80, "high":80}]}'
ziti edge create service 'eth0' --configs 'eth0.intercept.v1','eth0.host.v1'
ziti edge create service-policy 'eth0.binding' Bind --service-roles '@eth0' --identity-roles '@eth0.server'
ziti edge create service-policy 'eth0.dialing' Dial --service-roles '@eth0' --identity-roles '@eth0.client'

# enroll the 'server' identity (where you will ssh to)
ziti edge enroll eth0.client.jwt
sudo ./ziti-edge-tunnel run -i eth0.server.json

# enroll/use this identity in your "desktop edge" or ziti-edge-tunnel
#
# NOTE: skip this if you use ziti desktop edge for windows or mac
# ziti edge enroll eth0.server.jwt
#
# sudo ./ziti-edge-tunnel run eth0.client.json

# now run curl:
curl eth0.discourse.ziti
18.188.201.183

I’ll ask our router fella about that error:

router not assigning connId for dial

I’ve never seen that one yet

30

Sure, you can. I'm sure that's a bummer for you (and for me). Realistically there shouldn't be any difference whatsoever. But - that's not always how it is :). I didn't realize your app was behind a 'bastion'. That also should absolutely not matter.

Thanks for sticking with it - we'll get it, I'm sure!

31

Once we get this sorted too @bottles, maybe you will want to look at making your bastions dark and close inbound 22… we did it and wrote a piece recently - Bastion Dark Mode -

1 Like
32

Oh, thanks so much for all of these. I would want to learn more about the concept of how openziti is designing.

I think I might be missing some fundamental things. I thought zero network should only install on a bastion server, and all other application servers are behind it. In this way, a normal user will not be able to visit the application without opening a tunnel, but once they are on openziti network, they can get access to the applications through bastion server.

33

This is a great point to make… as from my viewpoint… ziti provides a paradigm shift.

ie. its not about using traditional tools… rather… its about creating a network fabric around everything

It’s also not about routing traffic over TCP/IP… as ziti operates also at a lower level… within the TLS at layer 2… using server and client certificates to create end to end trust.

This is what allows remote traffic to access servers when all of the ports are closed… because its operating at a deeper layer within the network… ie. you can now use a zitified version of ssh to ssh into a machine even when port 22 is closed… because you don’t need to have port 22 open anymore

I hope that helps…

This is what I understand… which I think is pretty close.

1 Like
34

Some update.

I have found that ziti-tunnel will be using tproxy as a default choice and when it is using tproxy to operate a tunnel, it utilises iptables [1]. In AWS, I used security group to control all the firewall traffic so I am unsure that if iptables would still be in use. I don’t know which platform openziti is supporting or maybe I need to set it on the on-premises machines. I will try to open all the ports in security group and see.

[1] ziti/ziti-tunnel at release-next · openziti/ziti · GitHub

35

It’s also not about routing traffic over TCP/IP… as ziti operates also at a lower level… within the TLS at layer 2… using server and client certificates to create end to end trust.

Shall I know where do you find the reference of this?

36

You will find it in general comments re the networking stack…

You can learn more by learning about the OSI model… Ziti operates between layers 2 to 5… as I understand

The OSI model explained and how to easily remember its 7 layers | Network World.

37

Hmm…I am asking where do you find “as ziti operates also at a lower level… within the TLS at layer 2” - not the explanation of OSI model.

38

This is a good reference point… a great demo also :slight_smile:

39

Specifically to layer 2… it was more of a conversation with the Ziti / NetFoundry team. Not sure I have seen it mentioned specifically in any documentation… though it could be mentioned in a video recording somewhere

Hope that helps… I have watched / read a lot of content… and remember these arbitrary bits of things :slight_smile:

40

Oh, okay. Thanks. Just a bit curious. I mean it is only a software, although it has changed a lot. I can see it changed iptables so maybe it works on network layer. But not sure how can it work on data link layer.