Hi, I am in the early learning of this new technique. I have strugged this for a week and still can’t make a simple example working. I have installed openziti on one server called openziti.example.com (private ip is 172.31.16.11) and installed a webserver called app.example.com (private ip is 172.31.16.10) on another server but they are in the same subnet. Those servers both on the aws.
To make everything easy, I use python simple http server to host an index.html file on port 80 at app.example.com.
I make app.example.com not accessible to public through firewalls - well, only port 22 for ssh access. I have opened 8441 (controller),8442 (router),8443 (zac),22 (ssh access) port to the public on openziti.example.com server.
I have followed Host Ziti Anywhere | Ziti to install openziti, and I also installed ZAC on the same node of openziti.
There is a critical bug if following the quick start doc that edge router needs to change hostname to a public ip and I changed that.
Through ZAC, I created a user client identity, a service with incept.v1, a service dial policy that uses identity client and the service.
Then I installed ziti-edge-tunnel on a Ubuntu (20.4) machine, and I can enroll the identity. I changed the /etc/hosts file on Ubuntu machine and make app.example.com to be 172.31.16.10. I then run with
./ziti-edge-tunnel run -i output and visit http://app.example.com:80 on the browser, but I still got the following errors.
[ 7203.096] ERROR ziti-sdk:connect.c:902 connect_reply_cb() conn[0.8] failed to connect, reason=service 1GtVCTL9U has no terminators for identity gitlab.desktop.client
[ 7203.096] ERROR tunnel-cbs:ziti_tunnel_cbs.c:104 on_ziti_connect() ziti dial failed: connection is closed
[ 7203.116] INFO tunnel-cbs:ziti_tunnel_cbs.c:428 intercept_match_addr() matching 172.31.16.10
[ 7203.325] ERROR ziti-sdk:connect.c:902 connect_reply_cb() conn[0.9] failed to connect, reason=service 1GtVCTL9U has no terminators for identity gitlab.desktop.client
[ 7203.325] ERROR tunnel-cbs:ziti_tunnel_cbs.c:104 on_ziti_connect() ziti dial failed: connection is closed
[ 7203.555] ERROR ziti-sdk:connect.c:902 connect_reply_cb() conn[0.10] failed to connect, reason=service 1GtVCTL9U has no terminators for identity gitlab.desktop.client
[ 7203.555] ERROR tunnel-cbs:ziti_tunnel_cbs.c:104 on_ziti_connect() ziti dial failed: connection is closed
If someone can tell me why I would very be appreciated. Thanks.
I am still learning the ropes.. but though to add a tips that I have learned along the way.
My understanding is that the "/ziti-edge-tunnel" is only required when you are building a custom SDK applications
i.e when you want to use zitified ssh or scp
What I understand in your example is that you need to install the mobile / desktop tunneller apps instead.. this is what uses the intercept configuration you have setup... the /ziti-edge-tunnel' does not use an intercept.. rather a host.. as I understand
To get your example to work.. what I believe you need to do is to zitify the application server... you can find the code here for a gloang server.. not sure if there is any code base for a python server
I would encourage you go give golang a go.. as its actually not hard to learn and will help you later one.. when you want to implement more advanced capabilities.. what is great is that it gets complied into C so that it runs fast
.. Also.. to get access to the dark application server from a desktop / mobile.. you need to enrol the identities on each desktop / mobile app.. so you will need to download the jwt from the server .. or use the QR enrollment feature as needed.
After having gone through the learning curve..I would suggest starting out with a simpler starting piece.. and only deploy to a single server..
here is how I got started.. after a lot of time spent learning
Once I worked through all of this.. I was then able to tackle implementing a reverse proxy to provide access to a private web server. I used a golang server to implement this.. and will be documenting my learnings later next week
After implementing a reverse proxy… you can go further by building a custom SDK to integrate directly into the Python application server.
However… I wanted to minimise the effort to get a working demo… which I why I started with the golang example … most of the SKD development has already been provided for you… making it much easier to get started.
The golang example is all open source… and is a great place to learn how to build your own SDK … which in your case… would be into the Python application server… once this is done… you no longer need the reverse proxy.
This is something I am planning next… but for now… I am quite content with a reverse proxy… to showcase what can be done… when you show people that you can use some crazy domain names… they freak out a bit… and go … that is really cool.
“What I understand in your example is that you need to install the mobile / desktop tunneller apps instead… this is what uses the intercept configuration you have setup… the /ziti-edge-tunnel’ does not use an intercept… rather a host… as I understand”
Could you tell me where the another mobile / desktop tunneller apps? The only client I can find is GitHub - openziti/ziti-tunnel-sdk-c. The documentation said if I use some ziti unware application, I will need this ziti-tunnel-sdk-c. Or I should choose to use ziti sdk to make ziti network be aware.
But now I guess maybe I need to create a service identity for the app and run ziti-edge-tunnel on the app.example.com too? In this way, I can make my simple python http web server be able to be identified by ziti network??
I thought either Go or python should be the same but I would try Go example if I am still not successful.
@bottles - welcome to the community! I'm sorry to hear you're having troubles but it looks like you've gotten quite far already!!! Nicely done.
Going back to your OP. When I see this particular log message:
service 1GtVCTL9U has no terminators for identity gitlab.desktop.client
that means the service you have setup is either wrong somehow, or more likely, it means the identity that is supposed to be hosting your service is not online.
I don't have a real clear picture of your overall solution but it seems like you've only created one identity? Is that correct? Later on you wrote: But now I guess maybe I need to create a service identity for the app and run ziti-edge-tunnel on the app.example.com too?".
Generally speaking that is what I would have expected. It usually looks something like this. Is that close to what you're trying to to?
Assuming that's correct - on the "local machine" running ziti-edge-tunnel - that's where you would be able to type into your browser app.example.com assuming you gave your client identity a service policy allowing it to 'dial'. On the remote machine you'd want another identity that was set to 'bind' the service. Then, when ziti-edge-tunnel turns on you'll get that terminator.
You should be able to do all this with ziti-edge-tunnel. It's easiest to start with that too imo.
Yes, the fix it out for that. It just needs to be released.
Can we first agree that the diagram I displayed above is what you're trying to do? Or pretty close?
OHHH!! Thanks for this really nice picture. This network image is exactually what I want to build. I just guess that both service and local machine might need to be identified by ziti.network. That’s right I only created one identity. I can feel that I am very close now and I will certainly try add a service identity soon.
OpenZiti is all about security. You can’t access services defined on the network without having a strong identity. So you are correct - you will need an identity at both locations in order to get your solution to work as shown.
I think you’re really close - seems like you just need a tunneller on the far end (ziti-edge-tunnel it sounds like, but it could be windows, or mac too of course).
If I were building this solution it would contain:
one intercept.v1 config - providing ‘intercept’ capabilities to tunneling-based clients
one host.v1 config - providing “offload” of my service on “Private Network 2” (as shown above)
one service which references the two configs mentioned above
one client identity - representing the identity looking to access the service
one “server” (loose term) identity - representing the far side of the service the client identity wants to access
one dial service-policy authorizing “clients” to access the service
one bind service-policy authorizing the ‘server’ identity to provide the offload from the ziti overlay network (the ‘server’)
You need these things too - which the quickstart provides:
one edge router in public IP space
one edge-router-policy to allow ALL endpoints (#all) to use this edge-router
one service-edge-router-policy allowing ALL services to use this egde-router
I think i captured all the pieces needed above. If i forgot any i’ll edit this post
Thanks so much for these. I have to say that after adding another server identity & bind policy, I still got the same error. I think the terminator should be automatically added, but there is no terminator. I will delete all these settings and try again tomorrow to see.
when you run the ‘far/server’ ziti-edge-tunnel, assuming you made a ‘bind’ service policy for that identity correctly, it will create a terminator automatically. yes
Example of me using this to login to an AWS box from my machine. notice i have to specify ubuntu and feed my identity file ~/.ssh/demoaws.pem :
ssh ubuntu@normalSshSvc.ziti -i ~/.ssh/demoaws.pem
Welcome to Ubuntu 20.04.3 LTS (GNU/Linux 5.11.0-1022-aws x86_64)
* Documentation: https://help.ubuntu.com
* Management: https://landscape.canonical.com
* Support: https://ubuntu.com/advantage
System information as of Fri Apr 22 16:44:16 UTC 2022
System load: 0.0 Users logged in: 1
Usage of /: 86.2% of 7.69GB IPv4 address for docker0: 172.17.0.1
Memory usage: 47% IPv4 address for eth0: 172.31.42.64
Swap usage: 0% IPv4 address for tun0: 100.64.0.1
Processes: 133
=> / is using 86.2% of 7.69GB
* Ubuntu Pro delivers the most comprehensive open source security and
compliance features.
https://ubuntu.com/aws/pro
43 updates can be applied immediately.
To see these additional updates run: apt list --upgradable
*** System restart required ***
Last login: Fri Apr 22 16:34:58 2022 from 67.246.244.61
I am playing this again today. That terminator issue seems gone once I recreate all the services/identities/policies, but I still can’t connect to the server I want. While I am still unsuccessful, I think I find some clue. I found that once I ran this in my local machine.
./ziti-edge-tunnel enroll -j client.jwt -i client
./ziti-edge-tunnel enroll -i client
ip a
I got a tunnel with ip like this
6: tun0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UNKNOWN group default qlen 500
link/none
inet 100.64.0.1/32 scope global tun0
valid_lft forever preferred_lft forever
inet6 fe80::ddef:430c:f3d7:4a30/64 scope link stable-privacy
valid_lft forever preferred_lft forever
then I tried to run
wget http://app.example.com
on my own machine, I can see the log like this:
[ 0.299] INFO tunnel-cbs:ziti_dns.c:251 new_ipv4_entry() registered DNS entry app.example.com -> 100.64.0.10
The system has found app.example.com as ip with 100.64.0.10.
./ziti-edge-tunnel enroll -j server.jwt -i server
./ziti-edge-tunnel enroll -i server
This runs without any error message so it seems good.
I then checked ip a
and found that the ip address of tun0 is also 100.64.0.1.
4: tun0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 500
link/none
inet 100.64.0.1/32 scope global tun0
valid_lft forever preferred_lft forever
inet6 fe80::f290:fcd9:f465:7ffc/64 scope link stable-privacy
valid_lft forever preferred_lft forever
I think this is where the issue is. I tought they should give app.example.com 100.64.0.10 so that my client machine can connect to app.example.com?? For some reason they always assign 100.64.0.1 to any machine that joins the vpn.
I am non expert on this, but do you think this is the right direction?
hey! you did a great job getting this far. way to go! one thing that’s ‘hard’ with ziti is that you can’t really use standard IP tools unless you really understand what ziti is doing under the hood. to help you more - let’s get a few more bits of data.
What is the machine you are running the ziti-edge-tunnel on? I assume that it’s linux? What version of linux is it? it sounds like your system isn’t sending dns requests to the local dns server. I’m not “the expert” on DNS on linux - but you can test it by running a dig at the IP address of the DNS server. I think it’s usually
dig @100.64.0.3 app.example.com
I’ll try to find someone who’s running linux to check out your post too
@bottles You’re nearly there. You’ve successfully configured things so that your tunneler’s nameserver is responding with an address for your Ziti service’s domain name, and so there may be one more thing you need to do to complete the connection. Let’s figure out what is that missing piece.
Below I’ll explain a little more about the tunneler’s built-in nameserver, but I don’t believe you are having a problem with that piece. The identical client and server IPs is not actually a problem (). Will you share more of the verbose log from ziti-edge-tunnel run --verbose=4 --identity=client so that we can diagnose the problem? If you see any ERROR messages from either client or server tunneler those will be most interesting.
A little more about the tunneler’s built-in nameserver / domain name resolver
One thing to note is that the IPv4 address the Ziti tunneler’s built-in nameserver provides is only addressable on the same device. That is, there’s no expectation that those addresses are unique for all devices on the Ziti network. Rather, they will always know each other by their Ziti identities, not an IP address.
The tunneler can be configured to utilize any CIDR for these ephemeral IPv4 addresses that are uniquely addressable to the device where the tunneler is running. The default CIDR is 100.64.0.0/10, and so the default interface address is 100.64.0.1 (the second address in the range), and the default nameserver address is 100.64.0.2 (third address in the range). If you were to override the default then the interface and nameserver addresses too would shift to the same offset positions of second and third addresses in the CIDR, respectively.
The built-in nameserver will answer any query that matches your Ziti services’ configurations. You may of course validate that any service’s domain name is “known” to your tunneler by directly addressing the query to the built-in nameserver, similar to the prior example @TheLumberjack provided, but with a minor adjustment to use the correct nameserver address.
$ dig @100.64.0.2 app.example.com
100.64.0.10
As an aside, you may also change the behavior of the built-in nameserver by specifying an “upstream” nameserver with --dns-upstream=208.67.222.222. This particular example configuration would cause the built-in nameserver to forward queries that do not match any authorized Ziti service domain name to an OpenDNS global recursive nameserver. In this way it is possible to configure the device’s OS to have a single resolver address that will answer queries for both Ziti DNS and global DNS.
This is really interesting... can you provide more specifics on what commands you need to run for this.. I think this would make what I have done a lot easier re reverse proxy server
Thanks a lot for the response. I learnt a lot from this project -well - although I am still unsuccessful.
I can run dig @100.64.0.2 app.example.com successfully on my desktop client machine and it will return 100.64.0.10, but I can’t run this successfully either on openziti server or the app.example.com server. However, if I added the identity running on app.example.com to dail service policy, dig command will return 100.64.0.10. I assume DNS is correct?
Now I ran wget http://app.example.com again on my desktop machine, it still doesn’t connect correctly.
I now checked openziti server and found the following log on ziti-router service.
Apr 26 14:03:45 ip-172-31-16-247 ziti-router[748]: {"_channels":["establishPath"],"apiSessionId":"cl2g7fke303w5ejqrwli6p0n8","attempt":0,"attemptNumber":"1","binding":"edge","circuitId":"dahuHcaJD","context":"ch{ctrl}-\u003eu{reconnecting}-\u003ei{xA87}","destination":"hosted:03a904fc-934a-4989-862c-f31edd8696be","error":"error creating route for [c/dahuHcaJD]: timeout waiting for message reply: context deadline exceeded","file":"github.com/openziti/fabric@v0.17.88/router/handler_ctrl/route.go:120","func":"github.com/openziti/fabric/router/handler_ctrl.(*routeHandler).fail","level":"error","msg":"failed to connect egress","sessionId":"cl2g7hok903yzejqrfm9y9lt5","time":"2022-04-26T14:03:45.206Z"}
Apr 26 14:03:50 ip-172-31-16-247 ziti-router[748]: {"_channels":["establishPath"],"apiSessionId":"cl2g7fke303w5ejqrwli6p0n8","attempt":1,"attemptNumber":"2","binding":"edge","circuitId":"dahuHcaJD","context":"ch{ctrl}-\u003eu{reconnecting}-\u003ei{xA87}","destination":"hosted:03a904fc-934a-4989-862c-f31edd8696be","error":"error creating route for [c/dahuHcaJD]: timeout waiting for message reply: context deadline exceeded","file":"github.com/openziti/fabric@v0.17.88/router/handler_ctrl/route.go:120","func":"github.com/openziti/fabric/router/handler_ctrl.(*routeHandler).fail","level":"error","msg":"failed to connect egress","sessionId":"cl2g7hok903yzejqrfm9y9lt5","time":"2022-04-26T14:03:50.209Z"}
Apr 26 14:03:50 ip-172-31-16-247 ziti-router[748]: {"_context":"ch{edge}-\u003eu{classic}-\u003ei{Py4v}","chSeq":29,"connId":7,"edgeSeq":0,"error":"exceeded maximum [2] retries creating circuit [c/dahuHcaJD]: error creating route for [s/dahuHcaJD] on [r/NtcRLAFfm] (error creating route for [c/dahuHcaJD]: timeout waiting for message reply: context deadline exceeded)","file":"github.com/openziti/edge@v0.21.169/router/xgress_edge/listener.go:160","func":"github.com/openziti/edge/router/xgress_edge.(*edgeClientConn).processConnect","level":"warning","msg":"failed to dial fabric","time":"2022-04-26T14:03:50.212Z","token":"48a4e17d-e1fc-4621-9086-e62ad46f1bed","type":"EdgeConnectType"}
My next guess is I have changed /root/.ziti/quickstart/ip-172-31-16-247/ip-172-31-16-247-edge-router.yaml and remove/readd edge router but maybe it is still not correct. I have changed the yaml file like this.
listeners:
# bindings of edge and tunnel requires an "edge" section below
- binding: edge
address: tls:0.0.0.0:8442
options:
**advertise: 13.58.67.111:8442** # this is the public ip address of openziti server
connectTimeoutMs: 1000
getSessionTimeout: 60s
- binding: tunnel
options:
mode: host #tproxy|host
edge:
csr:
country: US
province: NC
locality: Charlotte
organization: NetFoundry
organizationalUnit: Ziti
sans:
dns:
- ip-172-31-16-247
- localhost
ip:
- "127.0.0.1"
**- "13.58.67.111"** # this is the public ip address of openziti server, I guess maybe I should also add public ip of app.example.com server?
PS. The logs of app.example.com and my desktop machine are massive especially with verbose=4. I failed to find anything useful from them.
The log on app.example.com is here - note that the real DNS name is gitlab.hackercat.site.
everything looks like it’s correct to me. my guess is that your offload service is sending to a host/port that is not online. it looks like you are successfully sending traffic correctly.
what does the “bind” service, host.v1 config look like?
@markamind This is a new option for the Linux tunneler CLI ziti-edge-tunnelas of 0.17.23. Normally the built-in resolver behaves as an "authoritative" nameserver, and this new option changes the behavior to "recursive", meaning it will forward the query to the specified upstream nameserver if unable to answer it authoritatively.
While powerful for special cases, there is a reason the built-in nameserver behaves as "authoritative" by default. It is intrinsic to DNS to try to resolve a query with multiple nameservers, and so it is more transparent and simple to let Ziti DNS be the first nameserver and the naturally fail the query if it does not match any authorized services' domain names. That way, the next nameserver can answer non-Ziti queries normally.
It is not normally necessary for a Ziti tunneler user to manipulate their operating system's resolver configuration i.e. DNS settings. For example, when ziti-edge-tunnel runs it will auto-configure a Ziti resolver in the OS. Please do start a new Discourse topic or poke me on an existing topic if you have found a scenario where it is necessary to manually configure the Ziti resolver settings!
correct - you will need an identity at both locations in order to get your solution to work as shown.
You’ve successfully configured things so that your tunneler’s nameserver is responding with an address for your Ziti service’s domain name, and so there may be one more thing you need to do to complete the connection. Let’s figure out what is that missing piece.
). Will you share more of the verbose log from