Configuring ziti-tunneler-client.v1

[markamind]

@gooseleggs

You have used some slightly different configs which I have not yet come acros

I think they were an older config style.. I referenced them from an old version of the Java JDBC driver example

From what @TheLumberjack mentions... your approach is on the money

[gooseleggs]

These are a rinse and repeat from the configs given here: Your First Service - Zero Trust Host Access | Ziti (openziti.github.io)

[TheLumberjack]

Looks like @gooseleggs got you sorted? Awesome!

@markamind

I referenced them from an old version of the Java JDBC driver example

I just looked at the cheatsheet from the jdbc-postgres example and it looks to me like it has been changed away from using the terminator approach. @markamind where did you find the example commands you referenced? I want to update them. If they are embedded in the video, I'll have to edit/redo that video but if they are text somewhere, would you let me know so that I can update it?

Thanks!

[markamind]

This was the original video.. before you made the updates..

I do watch these videos in lot of detail

You will find it referenced from 5:42.. so I will need to update my approach now

Zero Trust Postgres - UPDATED see https://youtu.be/s-skpw7bUfI

PS.. Once I get this working.. I will provide more feedback.. as I feel there is a high level cookbook needed to navigate the different approaches..

[TheLumberjack]

Thanks. I’ve added some “updated Jun 2022” sections on that video to try to get people to use the new CLI commands.

[gooseleggs]

@markamind There is one question that I think we have not answered for you, and I walked away from the computer and thought that I should loop back around, and I have finally got to a keybaord to do this.

The question - Why does it work when I am doing everything “fat”, but doesn’t work so well when using docker. What is the best way? (Well if it wasn’t, I am writing about this anyway LOL). Grab a coffee - this could get long!

As said earlier, when you run something inside docker, it effectively runs on its own network. You can find the IP Address that a container is using by using docker inspect <container_name>. When you start/stop a container the IP address that it has may change. By using the “-networks” configuration inside the compose file, you tell docker what networks it can see. That is why httpbin is connected to zitiblue and zitired, so other containers can see it. Kind of like VLAN. Now, those networks are only routable on the host that is running docker. For the httpbin case, you can get to it through a container running on that network, (connect to the container with docker exec -it <container_name> /bin/bash, or you can connect to it by using the docker inspect command and browsing to that IP Address.

In fact, I use NGINX with dockergen and letsencrypt to automatically add a container to the load balancer, and go and get a Lets Encrypt cert for me automagically. Basically, it listens for a container start/stop event, and if it is connected to my NGINX network, and based on environment variables will configure the world for me. This means, that scripts are taking care to ensure that the load balancer is sending the packets to whatever address the upstream container is listening on.

Now, back to your plight…

There are two options where you can mix docker containers, and ‘fat’ applications:
a) Make the docker container bind to a host IP address, which is what we do by using the -ports docker configuration options. The downside is that you need firewalling to prevent access over the network.
b) (this is what I do when developing CakePHP Applications): use a bind mount to load in a folder on the host into a container

More on option B
So, how would this look like. In your case, you would create another service (duplicate httpbin example) but make it python. Then bind mount a folder into that container. The plus side of this, is that the container is effectively joined to the same network, and can see it at the docker layer. You would access the container like normal (docker exec -it <container name>) and away you go.

Then, when you happy with everything, and you want to distribute it, you could create a Dockerfile to build the container, and then you would not need the bind mount going forward. One of the advantages of a docker container is that I can 100% guarantee that if it works on my machine, then it will work on yours in terms of functionality as you bundle all the requirements up into it. This means there is no configuration drift that occurs between production and development that invariably sneaks in.

An untested addition to the docker-file could be

python:
  image: python:3.8.13-slim-bullseye
  volumes:
    - type: bind
      source: ./pythonsource
      target: /usr/src/myapp
  networks:
    - zitiblue
    - zitired

[markamind]

Ahh awesome. . super helpful… I will read in more detail and revert back shortly

[markamind]

Quick update… as I have made progress…

I think I get it now… because the nameserver can only find what is located within the Docker instance… rather than what is outside the Docker container

Hence… when you want to access a ziti intercept… the call needs to be made within the Docker instance… as that is where the nameserver runs from

If you try to run it outside the Docker container… it cannot find it

For example… if you setup a tunneller (in this case an old version as it is better to use ziti-edge-tunnel)…

./ziti-tunnel proxy -i /mnt/v/temp/tunneler-id.json private-httpbin:2000 -v

you can only use localhost to connect to the service

curl http://localhost:2000/json

If you use the intercept… it will not will not resolve the host

curl http://httpbin.ziti:2000/json

…
What confused me for a long time is that this is not how the service works

For instance, you can use this zitified curl command to test a ziti service

sdk-golang/example/curlz at main · openziti/sdk-golang

When you run the following command from outside the Docker container it works as expected

So…this approach must work different because its not accessing the namesever inside the Docker container

@TheLumberjack … more insights on this would be greatly appreciated

export ZITI_SDK_CONFIG=/mnt/v/temp/tunneler-id.json
./curlz http://private-httpbin:2000/json

[markamind]

In relation to your docker file, I can confirm that this worked a treat..

  python:
    image: python:3.8.13-slim-bullseye
    volumes:
      - type: bind
        source: ./pythonsource
        target: /usr/src/myapp
    entrypoint: /bin/bash
    networks:
      - zitiblue
      - zitired

What I did notice is that after running the following command

docker-compose up -d

I needed to run the following to restart it

docker run -d -it python:3.8.13-slim-bullseye

I am not sure why it stopped automatically stopped.. I could not work that out.

While I was hoping that this is all I needed to get an end to end test working.. I came up with another unexpected issue with installing the Python SDK in the container.

For some reason, installing the Python SDK missed making a copy of the lib folder..

Missing lib directory after installing Python SDK on a Docker Container

I am working through creating a Docker container for the Python SDK. I want to build this so that I can test the samples without needing to run them locally on my desktop. Steps followed apt-get -y update && apt-get -y install git git clone https://github.com/openziti/ziti-sdk-py.git cd /ziti-sdk-py/ python setup.py install ZITI_IDENTITIES=/ziti-sdk-py/sample/tunneler-id.json python3 h-ziti-p.py The error I received is OSError: /usr/local/lib/python3.8/site-packages/openziti-0.5.2-py3.…

[TheLumberjack]

@markamind

@dovholuknf … more insights on this would be greatly appreciated

Sure, on which bit? I must admit I'm having a hard time tracking what "the issue" is here and in your follow up you seem to say that things are working? What's the bit you want insight on?

[markamind]

sorry… its a bit of a mess in my head also… as something is not adding up … but don’t know how to explain it.

test 1:
When I run the ./curlz… as shown below… it works as expected… calling the ziti service…and returning httbin result … as expected

./curlz http://private-httpbin:2000/json

Assuming that if this works… I should also be able to run the Python SDK samples that use the intercept

test 2:

python3 h-ziti-p.py

however… when I run this, it does not find the intercept… even through I have confirmed with curlz that its setup correctly

Failed to establish a new connection: [Errno -2] Name or service not known’)

So… I am trying to find out what is the difference between calling an intercept and doing a curlz to work out what is causing this problem.

I thought that maybe… if I created a Python container… it would resolve the issue… but I just finished performing this test… which failed…

So… there is something that I am missing

Any tips on what next to test?

[markamind]

What makes this more confusing to me… is that when you run the Python SDK samples from your desktop… it works fine.

[TheLumberjack]

I just went and looked at sdk-golang/curlz.go at main · openziti/sdk-golang · GitHub. It’s extremely short… The problem with reading the code is that you, as the reader, will be required to actually understand just what the golang APIs do at a relatively deep level of understanding. I wouldn’t expect a learner/novice using these APIs at first to have this sort of understanding. So in my opinion, this isn’t on you so much (as the novice/learner) as much as it is on us for providing two samples that operate differently without having really clear doc/comments as to how it works. At the same time, you do need to read the code, try to understand it, and try to follow it along/debug it. But mostly, I think this is on us…

So, curlz is going to be taking the url you provided by passing an argument to it in position 1 (the 1st arg), parse it, and then look for a service with the name of the host you provided. In your example this would be private-httpbin. So curlz needs a service to exist that is named exactly private-httpbin to work properly in your example. If you rename your service to that - I would expect curlz to work… So when you see the “Name or service not known” error from curlz - it’s no surprise (after looking at the code, and how it was implemented).

Assuming I understand what the Python SDK is doing, it doesn’t work this way at all. Instead, the python stuff is expecting a service to have been defined and available to the identity you are using which has an intercept.v1 config defined on it. Then when you issue an http request, the python SDK goes into all the intercepts and tries to find a matching service based on the hostname provided. See how that mechanism is different than what curlz does? It’s a TOTALLY different paradigm. You’re mixing two samples, expecting them to co-mingle and “just work” (which is not unreasonable) without understanding that nuance between the two. However, that’s a dangerous game you’re playing because as you learned, they do not work the same!

I hope that is enough deail and explains to you why curlz is not working for you when the python stuff does. Hopefully it’s a bit more clear. If not, holla back and I’ll see if I can make it more clear.

-Clint

[markamind]

Thanks for the explanation.. I find this quite fascinating.. as there is a big learning here for me at the end.. along with a mini celebration

It's a bit hard to explain what I am observing.. as its really strange.. and I will need to retest it on my desktop to confirm..

Curl example
I use an identity I know has been setup.. because its the same as the one I use for the Postgres JDBC example.
I setup the services for the curl example.. and test it... which results the httpbin result as expected.

so.. I can assume that the identity works and has the right levels of access to run the Python script.

tick to proceed

Python example
I include an intercept into the configuration.. I use the same identity.. and run the Python script.

For some reason.. the intercept does not get triggered.

note.. this only appears to happen when I run this script on a server

It does not happen if I run the script on my Desktop..

@gooseleggs provided some great insights.. as it could be linked somehow to how the nameserver works with a docker setup.. which is very likely.

I just don't know how to fix it.

more context..

To illustrate.. say I remove the docker-compose configuration out of the equation.. and just use the following command to setup the httpbin service

docker run -p 80:80 kennethreitz/httpbin

I can use another controller I have setup to create a ziti host.. and use this to test locally on my desktop.

This is where I started.. and confirmed that this works.

server side example

To make sure I can support a server implementation, I thought to run it on the server.. using the Docker Compose configuration

This is really the only thing that has changed.. everything else is pretty much the same.

I have made a lot of mistakes along the way.. though I am pretty sure there is something extra to configure..

it could be a permissions thing... so I disabled SELinux.. but this did not change anything either.

I have tried setting up a separate container to run the Python code.. from within Docker.. this did not work either

So.. I am out of bullets to fire at this problem..

[markamind]

I don't want to give up.. as it's not in my nature..

All I need is a few more pointers on what else I can test .. to work out what the specific cause is.

I tried adding the intercept address to the hosts file.. this resulted in an access denied error.. which was to be expected.

I also tried dig

This does not resolve outside the Docker container... but does inside the Docker container

dig @110.0.0.157 httpbin.ziti

However.. even though it resolves inside the container.. I the Python script cannot find the intercept.

well.. at least that is what I think is going on from what I can observe

[TheLumberjack]

The challenge I have is that your environment is totally opaque to me. I have no idea what you have done to get here nor how to help you and I’m still not sure what you’re trying to do that is not working.

Can you break it back to basics and give me the exact steps you have taken to reproduce your problem along with whatever logs you’re seeing?

It sounds like you’re starting with docker-compose - that’s perfect. Give me your compose file, give me the ziti cli commands to run locally, and then tell me what command/program/how you’re running your test and I could perhaps help you out.

I don’t know what other pointers to provide. We’re now talking about python and not curlz, and that has me more confused.

Are you trying to get “a python app” to work?

[markamind]

@TheLumberjack

Are you trying to get “a python app” to work?

Yes.. this is the main focus.. I will break it down step by step.. and include a recoding so that you can follow along.. I will upload this shortly

[markamind]

You will find all of the information in the link below.

1. yaml file
2. commands
3. video demonstration

The config problem can be narrowed down to these two specific commands that I run on the server

curl http://localhost:2000/json

this works.. but don't know why.. as it should be the intercept as I understand

curl http://httpbin.ziti:2000/json

this does not works.. but don't know why.

If I can find the answer..as two why the intercept is not working on the server.. I will have solved my problem

PS.. I have confirmed the following entries in the /etc/hosts file

127.0.0.1 ziti-edge-controller
127.0.0.1 ziti-edge-router

https://drive.google.com/drive/folders/1D_qtD4pp5rRB0jcF4bhla8m3pa2X81A2?usp=sharing

[gooseleggs]

Hello

I haven't set it up and tested, but how does this sound? I dont think the problem is with the docker-compose environment, but with the linux tunnel mode that you are using.

First of all, if you want a cut down docker-compose file, OpenZiti have one already - it achieves the same result, but thought that I would point it out anyway. What you have is fine. Just if you wanted to start with a simplier file, there is one: https://github.com/openziti/ziti/blob/release-next/quickstart/docker/simplified-docker-compose.yml

What I think your problem is, is that you are running in proxy mode and not tproxy mode. From the documentation here: https://openziti.github.io/ziti/clients/tunneler.html

PROXY
The proxy intercept mode creates a network listener for each Ziti service that is intercepted. The services to intercept, and the ports that they are intercepted on, are specified on the command line (as opposed to using the service definitions that are retrieved from the Ziti Edge Controller):

So, using the proxy mode, it creates a listener, in this case on port 2000, that will direct traffic to private-httpbin. From reading the manual, there is no reference that it is doing ANY DNS intercept, and you must specifically call out the services to pass through. This is why the curl http://localhost:2000/json works. The tunneller is listening on port 2000, and shoving it over the ziti network to the docker instance. curl http://httpbin.ziti:2000/json does not work, because it cannot resolve httpbin.ziti. The tunnel in proxy mode is not doing anything with DNS.

From your client which is where you run it that does work, is running in tproxy mode and it IS intercepting DNS and therefore sending it over the overlay network.

I think you are wanting to run the ziti-tunneller in tproxy where it is transparent on the linux host. For this, however, it (the tunneller) needs to receive ALL DNS requests, which is what is happening on your client device, so that it can intercept httpbin.ziti and do what it needs to do. If the DNS request is not a ziti service, it will pass onto another DNS to resolve.

from the manual...

TPROXY
Typically you will run ziti-tunnel tproxy. This is the transparent proxy mode that uses IPtables rules to intercept traffic intended for OpenZiti Services. In this mode ziti-tunnel will also serve as an OpenZiti nameserver. You must configure the OS with that nameserver as the primary resolver. The nameserver will only answer queries for which it is authoritative i.e. OpenZiti Services' domain names, and so you will also need a secondary, recursive resolver.

I wouldn't say what I have said is true or not, but reading the doc seems to suggest this. To work around your problem at the moment, if you added a hosts file entry for httpbin.ziti then it will achieve the same result.

@TheLumberjack - looks like stale documentation, as on the tunneller page referenced above, it is talking about windows only supporting proxy mode.

[markamind]

Awesome insights @gooseleggs… as I would say you are on the money.

I just tried it out… replacing proxy with tproxy… but I really have no idea of what I am doing

I think more changes are required for this to work… as it generated the following error

FATAL edge/tunnel/dns.NewDnsServer: dns server failed to start: listen udp 127.0.0.1:53: bind: permission denied

Whenever I get a new error… I always feel like I am getting closer to the answer. I have also always wondered what the difference between proxy and tproxy is… now I feel like I at least have a better conceptual understanding

I will look through the discourse topics to see if this has been solved before