Routing to the web server

Hi, I have a few questions. First of all, I want to ask regarding the DNS name in the addresses. Is it can be a random name? For example, “webserver.ziti”. Or is it need the real DNS name?

ziti edge create config http.intercept.v1 intercept.v1 '{"protocols":["tcp"],"addresses":["http.ziti"], "portRanges":[{"low":80, "high":80}]}'

Currently in the progress of doing the Totally Private Postgres. My environment is quite the same as in the Youtube video. The difference is that I’m using droplet (Digital Ocean Cloud), the VM1 is the platform (using the docker-compose setup), and the VM2 is just a simple web server using httpd (not inside a docker). After setting up everything, I still cannot access the web server from the client side.
ping private.webserver.ziti

When I try to curl the private.webserver.ziti, it will give the timeout.

Curl: (28) Failed to connect to private.webserver.ziti port 80 after 16670 ms: Timed out

Also, I’m curious about the ping IP, the reply from 100.64.0.4 and what it means by smartrouting in the terminator strategy.
Besides, do I need to configure routing in the router or anything related?
Thank you in advance again for sharing and helping me.

Yep. whatever name you want. Isn’t that neat? It just needs to be ‘valid’. So mostly ascii characters and a few other. Any valid DNS entry.

You can’t use ping. Ping is a network level tool and uses ICMP. It’s really only useful to query the IP associated with the DNS name. Curl should work though. If it’s not working, and your setup is pretty much the same as the video, you should tail (or look at) the tunneler log, and you should look at the log on the router. If there are no helpful messages in there (there usually is) then you can look at the controller log. Finally, I will often use ziti edge policy-advisor identities -q (or policy-adisor services, shown below) to check my policies are correct. You might start with policy advisor first though. More specifically you can use a service name like this:

ziti edge policy-advisor services svc.device-virtual -q
OKAY : svc.device-virtual.identity (1) -> svc.device-virtual (1) Common Routers: (1/1) Dial: N Bind: Y

Notice here, I have a service that can be BOUND – but no identities can Dial it (lol - not very useful!)


Now look at this example:

ziti edge policy-advisor services svc.core-data -q
OKAY : _cdaws_clint (1) -> svc.core-data (1) Common Routers: (1/1) Dial: Y Bind: N

OKAY : svc.core-data.identity (1) -> svc.core-data (1) Common Routers: (1/1) Dial: N Bind: Y

That service has an identity ‘binding’ it – and an identity that can dial it. It’s useful! That make sense?

The most common problem is forgetting one or the other service policy. I can tell that because your tunneler has the service, you have the dial properly set correctly. You might have a missing bind policy or the policy might be setup wrong. policy-advisor will show you if you have it wrong, but you’ll have to figure out ‘how’ you did it wrong by reviewing the policy you have in place.

Another common problem is that the router/identity offloading the traffic to the final destination doesn’t have access to the service you’re trying to access. Maybe a firewall is in the way, or the IP in the host.v1 config is wrong etc.

It’s almost certainly some small configuration issue, sometimes it’s hard to find them but I’m sure you’ll spot it!

I forgot abt the policy advisor.

ziti@1e1b254c6c5d:/persistent$ ziti edge policy-advisor services private-postgres -q
OKAY : http-client (2) -> private-postgres (5) Common Routers: (2/2) Dial: Y Bind: N 

If the bind is N, so it is not binding yet?

Besides, 1 question regarding the video. Why would you update the router?

ziti edge update identity ip-172-31-42-64-edge-router -a "private-postgres-servers"

The private-postgres-servers is an identity role, right? This is the only step (update) I didn’t follow in my setup.

ziti edge create service-policy private-postgres-binding \
    Bind --service-roles @private-postgres --identity-roles '#private-postgres-servers'

I do understand that we need to bind the service representing the HTTP server. So for the identity server, do we just randomly eventhough it’s an id? I’m confused about this part.

Correct. In tunneler-based solutions, like that example, you need one identity that is doing the dialing (the client side) and you need one identity that’s offloading from the overlay network (the far/server side, doing the ‘bind’). You’re missing the bind.

If I remember correctly, this is what will end up matching the bind service policy. I think that’s what gets you the bind side.

Yep. That was my guess above. But you need to. You could run the same command just use ‘private-web-servers’ instead of postgres. Make sure you run both of those commands though. You need a bind service policy and you need identities that will match the identity-roles attribute you use in the service policy

Not sure exactly what you mean by randomly. Here you’re explicitly telling the overlay that any identity with the matching attribute is expected to be able to bind the service. It’s how you could provide redundancy/additional throughout to your back end. For test stuff, you rarely need that redundancy/throughout. It’s just used to be instructive that it works this way.

Hth

Update:
I already add the bind services and the attribute

ziti edge policy-advisor services private.web -q
OKAY : web-client (2) -> private.web (5) Common Routers: (2/2) Dial: Y Bind: N 

OKAY : ziti-edge-router (2) -> private.web (5) Common Routers: (2/2) Dial: N Bind: Y

Here is step by step

7. ziti edge create identity user web-client -a 'web-clients' -o web.client.jwt

//copy jwt file into any of root file
8. docker compose cp ziti-controller:/persistent/web.client.jwt /etc

9. Open winSCP to transfer the file from the droplet by using SFTP into the host machine as client side.

10. Open ZDEW, then 'Add Identity, choose http.client.jwt.

11. Automatically show up in the interface of ZDEW. 

12. ziti edge list identities "limit none" | grep (id router)

//ip bla2 is the router name, the private-web-servers is attribute
extra step: ziti edge update identity (id router) -a "private-web-servers"

//pws=private web server
//This config is used instruct the server-side tunneler how to offload the traffic from the overlay, back to the underlay.
13. ziti edge create config pws.intercept.v1 intercept.v1 '{"protocols":["tcp"], "addresses":["web.ziti"], "portRanges":[{"low":80, "high":80}]}'

//Create a host.v1 config. This config is used instruct the server-side tunneler how to offload the traffic from the overlay, back to the underlay. 
14. ziti edge create config pws.host.v1 host.v1 '{"protocol":"tcp", "address":"(private IP web server)", "port":80}'

//step 15 if want to update the addresses can use ziti edge update 
15. ziti edge create service private.web --configs pws.intercept.v1,pws.host.v1

//private web dial =pwd
//Create a service-policy to authorize "Private Web Clients" to "dial" the service representing the HTTP server.
16. ziti edge create service-policy pwd Dial --service-roles "@private.web" --identity-roles '#web-clients'

//private web bind = pwb
//Create a service-policy to authorize the "Private Web Server" to "bind" the service representing the HTTP server. 
//the private-web-servers is the attribute for the router
17. ziti edge create service-policy pwb Bind --service-roles '@private.web' --identity-roles "#private-web-servers"

Then i try to curl the web.ziti.

curl: (7) Failed to connect to web.ziti port 80: Connection refused

I already add the inbound rules of HTTP for the droplet. Hopefully, I can find what I miss in this mini-experiment. Thank you for the help. It helps me to understand each of the commands.

Yes that’s looking better. What do the logs in the local tunneler and the remote router show? Any hints in there? I’m expecting one or the other to have a helpful clue.

You’re sure that on the edge router, it can curl to (private IP web server):80, right? Can you verify that ziti-edge-router can do that? Let’s look at the last 40 ish lines of logs for both sides and verify the curl works.

If you don’t want to publish your logs, you can send them via email to clint at openziti.org

It seems like my ziti-ziti-edge-router.log is empty.

cat ziti-ziti-edge-router.log

As for the tunneller log, I’ll send it to you through email.
For better understanding, I’ve included this image.

Is it possible to route from docker to non-docker?

If you’re running using systemd, that’s totally normal. If you followed the quickstart, you’ll have run a systemctl enable command (using systemd). In that case you need to look at the logs using journalctl: journalctl -u ziti-router “should” be the command you want. Can you check those? From your logs you emailed me (thank you) it looks like your controller is totally offline – that’d cause problems too.

Your image is great. It’s very possible to route like that from docker, sure. I would expect that it would work fine. You can “exec” into the docker container using something like docker exec -it ubuntu_ziti-controller_1 bash (replace ubuntu_ziti-controller_1 with whatever the name is of your docker container).

Once inside the container, you should be able to use a curl command to probe that endpoint.

We’ll figure this out :wink: Talk to you soon, cheers

1 Like

Update:

root@network:~# journalctl -u ziti-router
-- No entries --
root@network:~# journalctl -u ziti-router --verify
3905a8: Data object references invalid entry at 3e21850             
File corruption detected at /var/log/journal/37c3f6b4cf7334a41d5a243163a55b87/system.journal:3e21678 (of 67108864 bytes, 97%).
FAIL: /var/log/journal/37c3f6b4cf7334a41d5a243163a55b87/system.journal (Bad message)
PASS: /var/log/journal/37c3f6b4cf7334a41d5a243163a55b87/system@8be4fa9389d442d8a9afb8994735e241-000000000003d268-0005f1827d8fd004.journal
PASS: /var/log/journal/37c3f6b4cf7334a41d5a243163a55b87/system@8be4fa9389d442d8a9afb8994735e241-000000000000ec00-0005f0eaf9c1d15f.journal
PASS: /var/log/journal/37c3f6b4cf7334a41d5a243163a55b87/system@8be4fa9389d442d8a9afb8994735e241-000000000001e10e-0005f11dd4797752.journal
PASS: /var/log/journal/37c3f6b4cf7334a41d5a243163a55b87/system@8be4fa9389d442d8a9afb8994735e241-000000000002d849-0005f15053d9e153.journal
PASS: /var/log/journal/37c3f6b4cf7334a41d5a243163a55b87/system@8be4fa9389d442d8a9afb8994735e241-0000000000000001-0005f0c483b164f8.journal
PASS: /var/log/journal/37c3f6b4cf7334a41d5a243163a55b87/system@8be4fa9389d442d8a9afb8994735e241-000000000004ceaa-0005f1a675f54c35.journal

From the client side, I was able to reach the controller:

C:\Users\..>curl https://pssnet.ddns.net:1280/version -k
{"data":{"apiVersions":{"edge":{"v1":{"apiBaseUrls":["https://pssnet.ddns.net:1280/edge/client/v1"],"path":"/edge/client/v1"}},"edge-client":{"v1":{"apiBaseUrls":["https://pssnet.ddns.net:1280/edge/client/v1"],"path":"/edge/client/v1"}},"edge-management":{"v1":{"apiBaseUrls":["https://pssnet.ddns.net:1280/edge/management/v1"],"path":"/edge/management/v1"}}},"buildDate":"2022-12-20T20:53:53Z","revision":"d9282a0b4e84","runtimeVersion":"go1.19.4","version":"v0.27.1"},"meta":{}}

Part of the tunneller logs:

[2023-01-11T05:20:55.210Z]    WARN ziti-sdk:ziti.c:1458 api_session_cb() ztx[0] failed to get api session from ctrl[https://pssnet.ddns.net:1280] api_session_state[1] CONTROLLER_UNAVAILABLE[-15] software caused connection abort
[2023-01-11T05:21:00.210Z]    INFO ziti-sdk:ziti.c:866 ziti_re_auth_with_cb() ztx[0] starting to re-auth with ctlr[https://pssnet.ddns.net:1280] api_session_status[0] api_session_expired[TRUE]
[2023-01-11T05:21:00.456Z]   ERROR ziti-sdk:ziti_ctrl.c:154 ctrl_resp_cb() ctrl[pssnet.ddns.net] request failed: -4079(software caused connection abort)
[2023-01-11T05:21:00.456Z]    WARN ziti-sdk:ziti.c:1458 api_session_cb() ztx[0] failed to get api session from ctrl[https://pssnet.ddns.net:1280] api_session_state[1] CONTROLLER_UNAVAILABLE[-15] software caused connection abort
[2023-01-11T05:21:05.314Z]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:637 on_cmd() received cmd <{"Data":{"DumpPath":"C:\\Program Files (x86)\\NetFoundry, Inc\\Ziti Desktop Edge\\logs\\service"},"Command":"ZitiDump"}
>
[2023-01-11T05:21:05.314Z]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:274 process_cmd() ziti dump started 
[2023-01-11T05:21:05.316Z]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:325 process_cmd() ziti dump finished 
[2023-01-11T05:21:05.316Z]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:223 on_command_resp() resp[1,len=25] = {"Success":true,"Code":0}
[2023-01-11T05:21:05.464Z]    INFO ziti-sdk:ziti.c:866 ziti_re_auth_with_cb() ztx[0] starting to re-auth with ctlr[https://pssnet.ddns.net:1280] api_session_status[0] api_session_expired[TRUE]

This is quite interesting to me. Of course, the first thought when doing something like this “It’s going to work as I’m following it, just change the environment and use a different setup”. Yet, it didn’t work, and I learned many things while trying to configure how it went wrong or which part I forgot. If everything is work, not very thrilling in the research part, right? :joy: I am just taking my sweet time learning as much as possible about zero trust and experimenting few things like this before finishing my internship soon.

I keep forgetting you’re using docker compose to run the router. What do you see when you run docker logs on the router container?

Docker logs:

[83695.508]    INFO : http: TLS handshake error from 115.164.92.206:51978: read tcp 172.31.0.3:1280->115.164.92.206:51978: i/o timeout
[83695.816]    INFO : http: TLS handshake error from 115.164.92.206:52003: EOF
[83701.044]    INFO : http: TLS handshake error from 115.164.92.206:52012: read tcp 172.31.0.3:1280->115.164.92.206:52012: read: connection reset by peer
[83706.248]    INFO : http: TLS handshake error from 115.164.92.206:51974: EOF
[83711.439]    INFO : http: TLS handshake error from 115.164.92.206:51996: EOF
[83716.648]    INFO : http: TLS handshake error from 115.164.92.206:52008: EOF
[83721.868]    INFO : http: TLS handshake error from 115.164.92.206:52008: EOF
[83727.178]    INFO : http: TLS handshake error from 115.164.92.206:52017: EOF

This is just a part of it. Mostly it is the same INFO.

At this point I think I’m going to produce a video that shows how to make this work. I’ll probably use aws, not digital ocean, but it should not matter (other than the obvious differences in cloud providers). Then, you’ll be able to watch that step by step and we’ll see if that helps.

You can you check the controller logs too and see if there are any hints in there.

I’ll post back after I have a video walkthrough

Docker Ziti-Controller logs:



Thank you for the upcoming video.

Hi @himeose. I don’t have a video yet but I wanted to follow up. I think this is what you’re doing, and I did it in my own lab:

To accomplish this, here’s exactly what I did:

Prerequisites:

  1. Provisioned two VMs in a single AWS VPC
  2. Ran the quickstart on one VM and verified it works as expected
  3. Ran the docker container crccheck/hello-world and exposed it on port 80 (this is the web server that shows the docker whale) docker run --rm --name web-test -p 80:8000 crccheck/hello-world

Enable “private.web” access

  1. obtain a docker env file and the simplified compose file in whatever folder you like:

     curl -sO https://raw.githubusercontent.com/openziti/ziti/release-next/quickstart/docker/.env
     curl -s https://raw.githubusercontent.com/openziti/ziti/release-next/quickstart/docker/simplified-docker-compose.yml > docker-compose.yml
    
  2. update the .env file to look similar to this. Hopefully obvious, replace advertised.address with your externally accessible address. like ‘ec2-3-134-108-218.us-east-2.compute.amazonaws.com’ for example etc

     # OpenZiti Variables
     ZITI_IMAGE=openziti/quickstart
     ZITI_VERSION=latest
    
     # OpenZiti Variables
     ZITI_IMAGE=openziti/quickstart
     ZITI_VERSION=latest
    
     # The duration of the enrollment period (in minutes), default if not set
     # shown - 7days
     ZITI_EDGE_IDENTITY_ENROLLMENT_DURATION=10080
     ZITI_EDGE_ROUTER_ENROLLMENT_DURATION=10080
    
     # controller address/port information
     ZITI_CONTROLLER_RAWNAME=ziti-controller
     ZITI_CONTROLLER_HOSTNAME=ec2-3-134-108-218.us-east-2.compute.amazonaws.com
     ZITI_CTRL_PORT=8440
    
     ZITI_EDGE_CONTROLLER_RAWNAME=ziti-edge-controller
     ZITI_EDGE_CONTROLLER_HOSTNAME=ec2-3-134-108-218.us-east-2.compute.amazonaws.com
     ZITI_EDGE_CONTROLLER_PORT=8441
     #ZITI_EDGE_CONTROLLER_IP_OVERRIDE=172.17.0.1
    
     # router address/port information
     ZITI_EDGE_ROUTER_RAWNAME=ec2-3-134-108-218.us-east-2.compute.amazonaws.com
     ZITI_EDGE_ROUTER_PORT=8442
     #ZITI_EDGE_ROUTER_IP_OVERRIDE=172.17.0.1
    
  3. start docker - IMPORTANT: notice I used -p openziti to start docker. That produces a predictable name used in step 5 below. Make sure you either understand what the -p flag is doing here, or just run it exactly as shown! :slight_smile:

     # this removes EVERYTHING:
     docker-compose -p openziti down -v
     # start docker here
     docker-compose -p openziti up
    
  4. optional: If needed, get the latest ziti binaries and have them added to the path for you if needed

     source /dev/stdin <<< "$(wget -qO- https://get.openziti.io/quick/ziti-cli-functions.sh)"; getZiti "yes"
    
  5. setup a file to source to make it easy to login to the environment. This pulls information from the container and makes a “docker.env” file that you can source and use to ziti login

     docker exec openziti_ziti-controller_1 grep 'export ZITI_EDGE_CTRL_ADVERTISED' /persistent/ziti.env > docker.env
     docker exec openziti_ziti-controller_1 grep 'export ZITI_USER' /persistent/ziti.env >> docker.env
     docker exec openziti_ziti-controller_1 grep 'export ZITI_PWD' /persistent/ziti.env >> docker.env
     source docker.env
    
     ziti edge login $ZITI_EDGE_CTRL_ADVERTISED -u $ZITI_USER -p $ZITI_PWD -y
    
  6. Configure the OpenZiti overlay for the private web service and identity. (notice I used MY ip=172.31.50.50 of my private web server)

     service="private.web"
     private_web_server_ip=172.31.50.50
     ziti edge create config "${service}.intercept.v1" intercept.v1 '{"protocols":["tcp"],"addresses":["'${service}'"], "portRanges":[{"low":80, "high":80}]}'
     ziti edge create config "${service}.host.v1" host.v1 '{"protocol":"tcp", "address":"'${private_web_server_ip}'","port":80}'
     ziti edge create service ${service} --configs "${service}.intercept.v1","${service}.host.v1"
     ziti edge create service-policy "${service}.bind" Bind --service-roles "@${service}" --identity-roles "#${service}.binders"
     ziti edge create service-policy "${service}.dial" Dial --service-roles "@${service}" --identity-roles "#${service}.dialers"
    
     ## authorize dialers...
     ziti edge create identity user awsdockerclint -a "${service}.dialers" -o awsdockerclint.jwt
    
  7. authorize the router identity to bind the service. Notice i used the name of MY edge router here (ec2-3-134-108-218.us-east-2.compute.amazonaws.com) - you have to replace with yours:

     ziti edge update identity ec2-3-134-108-218.us-east-2.compute.amazonaws.com -a "${service}.binders"
    

I’ll make a video demo’ing all this in a bit

And a companion video for the steps outlined above

Thank you for following up and providing your insight. Really appreciate your willingness to share your own experience and knowledge. Gotta try it later and can’t wait to see the outcome of my own through this setup.