Tips to optimise router configuration

Ziti Overlay
1

I am currently troubleshooting the use of two Golang reverse proxy servers.

#1. redirects from underlay to overlay
#2. redirects from overlay to underlay

I have this working end to end, though the performance is a bit slow and there are some issues with closed connections (I think this relates to Golang).

I did find some tweaks that you can make at the transport layer to improve the performance of the reverse proxy Golang server

zitiTransport.MaxIdleConns = 1000
zitiTransport.IdleConnTimeout = 90 * time.Second
zitiTransport.MaxConnsPerHost = 1000
zitiTransport.MaxIdleConnsPerHost = 1000

Unfortunately, this did not make much of a difference.. so I am starting to look upstream.

Specifically, I want to look at some of the edge router options to speed things up. My assumption is that the "QuickStart" install has some settings that maybe I could change.

I think the bottle neck is the rapid increase in connections being made which is the result of the Golang reverse proxy..

To investigate further, I am considering increasing outQueueSize.. but are not sure what this does

If you change this, do you need to restart / reinstall the router for the changes to take effect?

link:
dialers:
- binding: transport
listeners:
- binding: transport
bind: tls:0.0.0.0:10080
advertise: tls:instance-20220416-1603:10080
options:
outQueueSize: 10

are their other settings that would be more suitable to deal with a rapid influx of connections being made?

2

Unfortunately I think the key here is going to be reproducibility... Are you able to replicate the 'slowness' when using a tunneling app on the "client" side and a tunneling app/router on the "server" side? I'd be interested to know if you can express the same slowness with that sort of setup. That'll bring you back to 'baseline' for me and it might be something to look into on our side.

I also would need to define "a bit slow". For example I know establishing the very first connection is "a bit slow" when compared to using direct underlay because ziti is finding the fastest secure path to the target, doing the dialing etc but it's usually not something i end up 'feeling' as a human. It's measurable on computer scale for sure but after that first connection, I don't feel it myself. (Hopefully that makes sense)

We do not document all the options one can set on all the things really well right now. That's a task on the backlog to do. Generally speaking, I'd say you don't need that though. I think it's been a really rare event to change those defaults for any instances of openziti. I don't know how much it'll end up making a difference. :frowning:

Yes. any config changes currently require the router to restart

I don't know if I'll be able to give any good guidance here. I'm interested to konw if you observe the same behavior with a tunneller to tunneller/router setup.

3

Thanks @TheLumberjack ... as I am making progress..

One big news is I have now addressed the http / https in the browser over a desktop tunneller.

I was serving http when I was launching the zitified reverse proxy.

I did not much idea of what I was doing.. it was only after @PhilipGriffiths kept asking me a few questions... along with.. more ideas on how to program in Golang that the penny dropped

thanks so much for helping me resolve this issue..:slight_smile:

4

In relation to the performance.. there is clearly something happening.

Reverse Proxy Scenario #1
origin: my website
address: httpzitified.address

this works really well using a desktop tunneller, returning the traffic over https

Reverse Proxy Scenario #2
origin: httpzitified.address
address: localhost

this is takes over 3-4 times the time to load

While the second scenario may appear a little unusual.. as it is basically replicating the use of a tunneller.. I am developing this for another purpose.. so I want to understand how to retrieve traffic from a ziti DNS into a host server using the golang SDK

I can definitely say..

  1. redirecting traffic into the network overlay works well

Its just that I need to understand what is happening when I take the traffic in the opposite direction.. over the same controller, edge-router, and host.

I will keep playing around and update when I find out more what is happening.

5

I have been playing around with this most of the day now.

The main issue that keeps repeating that causes the slow response is a lot of retries..

and.. the context connection appears to also close

What I am thinking is that there is something wrong with my code.. so maybe defaulting back to the tunneller is a safer option.

Though.. I am keen to investigate further and learn more about this behaviour.

6

As a quick update… I have been tinkering around with this for a while now… and I have come to believe its a CPU bottleneck

In the specific configuration that I have… the following is setup on a single micro compute instance

  1. ziti controller
  2. ziti edge router
  3. ORDS server (this is what I believe is memory hungry)

I narrowed this down through the following approach.

  1. when I point the reverse proxy to my website… it works really quickly.

  2. when I point the reverse proxy to the ORDS server… its very slow.

I then thought… maybe I should check out the CPU performance trends… and noticed some big spikes in IO…

So… I will aim to move the ORDS server onto another compute… and try again…

This seems to make sense now…