Hey everyone, I am hosting some applications and proxying them both by tailscale and zrok. With tailscale I get the real ip and fail2ban works great but with zoom and caddy I get the wrong ip.
I have tried this:
X-Real-IP {remote_host}
Hi @aikooo7, welcome to the comunity and to OpenZiti!
I'm not exactly sure I understand what you're trying to do. Are you trying to intercept an ip on the 'client' side and have the intercepted IP from the client be the one that's connected to on the far side?
If that's what you want to do, you will want to look into the 'forwarding' configuration for host.v1 configs:
I think that's what you are looking for, but i'm not entirely sure.
Hey,
My use case is not on client but on the server, vaultwarden is logging the ips of who fails a login and fail2ban uses it. See about it here.
It works good and all in tailscale and gets my tailscale machine ip but when using zrok, it gets the zrok ip not my machine ip so if someone gets the authentication wrong x amounts of times the zrok will be banned, not who failed the login.
You're using the zrok service at zrok.io, correct? You're not self-hosting it?
I'll have to do a little digging, but you should be getting an HTTP header with the real IP address of the remote user. In a quick test on my end, it doesn't look like that's happening. It was... something may have changed in the production environment, so I will need to coordinate with that team.
Hey,
I can confirm I am using the zrok service at zrok.io and not self-hosting.
Anything I can help please let me know and keep me updated.
There was a fix applied to the production environment today. If you take a look at the X-Forwarded-For header in the inbound request to your service, the first address will be the remote address of the client accessing your public zrok share.
You can ignore the second address, for now. A follow-on change to the software will be forthcoming that will remove that second address.
But this should get you going for now.
Hey,
It is still not working, I searched and searched and found out that vaultwarden uses X-Real-Ip but I wasn't able to set it based on X-Forwarded-For, may I get some help here?
Also, I wasn't able to see X-Forwarded-For using the global directive debug, is that normal behavior?
The convention for proxies is to include an X-Forwarded-* set of headers, including X-Forwarded-For. And that is definitely working in production zrok now.
We might be able to include an additional header, X-Real-IP that is a mirror of X-Forwarded-For, in an upcoming release of zrok, but that might not happen for a week or so.
Hi,
You understood me wrong, I meant how can I make mirror the X-Forward as X-Real-Ip
I don't have an answer for you there. That depends on your setup and the software that you're using.
Currently, zrok does not have a facility that would allow you to mirror that header. It's something that I've added to the backlog and we will add as a new feature in an upcoming release, probably within a few weeks.
Hey @michael.quigley ,
I am still getting the wrong ip, is it still to be pushed to production or is this unexpected behavior?
Sorry for disturbing but I can't expose my services until this is fixed.
This is currently working correctly in production. The X-Forwarded-For header is currently returning the public IP address of the client, followed by an infrastructure IP address. The infrastructure address can be ignored, but the public IP is accurate.
We might remove the infrastructure address at some point in the future, but the functionality that is there now is working properly.
Hey, I can confirm is true but forgejo/vaultwarden are detecting in the logs the authentication as being done by the infrastructure, not by the real user, how soon could them being separated?
I'm not 100% sure we're going to change the behavior. For self-hosters and a number of other scenarios, what's there now is the most accurate implementation. It's not as simple as just stripping off that IP address...
Keep in mind, someone could also set an X-Forwarded-For in their request and push something to the front of that list. Preserving that addtiional address would actually be the most accurate behavior for a reverse proxy... if you were to put zrok behind another reverse proxy, you would want that behavior.
If you're going to put fail2ban behind a reverse proxy like this, then the most reliable thing you can do is to always take the next-to-last address from the X-Fowarded-For header and use that as your real IP address. zrok can't necessarily reliably do that for you because it doesn't know if it's behind another reverse proxy or not.
So, we may change this behavior in the future, but for now it's not really on the immediate roadmap.
If I were hosting fail2ban behind zrok, I would make sure that fail2ban is reliably getting the next-to-last address out of that list in X-Fowarded-For... I wouldn't delegate that responsibility to zrok.
For now, zrok is doing the most accurate thing it can.
Hey @michael.quigley,
Sorry to bother but with debug directive in caddy I am not seeing the header, any way I can see it?
{"level":"debug","ts":1716711026.88835,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"forgejo:3000","total_upstreams":1}
{"level":"debug","ts":1716711027.6467364,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"forgejo:3000","duration":0.757466899,"request":{"remote_ip":"ziti-edge-router connId=2147483649, logical=ziti-sdk[router=tls:ad2b1ffe-8cee-465d-b3fc-8d1100bb5f32.production.netfoundry.io:443]","remote_port":"","client_ip":"","proto":"HTTP/1.1","method":"GET","host":"forgejo.share.zrok.io","uri":"/","headers":{"Accept-Language":["en-US,pt-PT;q=0.5"],"Upgrade-Insecure-Requests":["1"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8"],"Priority":["u=1"],"X-Amzn-Trace-Id":["Root=1-6652ee72-690f6daf0e02aac701a14bcd"],"Accept-Encoding":["gzip, deflate, br, zstd"],"User-Agent":["Mozilla/5.0 (Android 14; Mobile; rv:126.0) Gecko/126.0 Firefox/126.0"],"Sec-Fetch-Mode":["navigate"],"X-Proxy":["zrok"],"Sec-Fetch-Dest":["document"],"Sec-Fetch-Site":["cross-site"],"Cookie":[],"X-Forwarded-Port":["443"]}},"headers":{"Cache-Control":["max-age=0, private, must-revalidate, no-transform"],"Content-Type":["text/html; charset=utf-8"],"Set-Cookie":[],"X-Frame-Options":["SAMEORIGIN"],"Date":["Sun, 26 May 2024 08:10:27 GMT"]},"status":200}
Run a zrok test endpoint in one terminal window... and then share it using zrok share public 9090. If you access that share, you'll see the headers that are sent by zrok.
1 Like
I am using a variation of zrok's public docker share exanple, can that same be done? I am using the caddy backend.
Yes, you can run any command in any container.
If you're using Docker Compose and your Compose Service is named "zrok-share" then you can run any command like:
docker compose exec zrok-share whoami
Hey,
I am getting some problems running this command in the docker container so could someone please help me be able to only send the client ip to the service i.e forgejo so in the logs the correct ip appears?
Here's a way to use Michael's advice with the Docker public share example. I assume you downloaded the compose.yml file from that guide. It contains the zrok test service Michael mentioned, so you only need to share it by setting these variables in the compose file's environment section.
ZROK_BACKEND_MODE: proxy
ZROK_TARGET: http://zrok-test:9090
Of course, you must have already set ZROK_ENABLE_TOKEN for the zrok-enable service.
When you run docker compose up the share will be reserved and you will see it in the web console. When you visit the public share URL you will see the request information received by the share backend running in Docker. This will tell you precisely which HTTP headers were sent to the backend by the reverse proxy.