If we deploy a public ziti controller or router what options do we have to mitigate DOS/DDOS attacks (both intentional and unintentional) against them?
Currently there don't seem to be any configuration settings that would assist with:
- blacklisting specific IPs
- limiting the max number of connections from clients
- throttling the max bandwidth that a client can use after establishing a session
Are there internal (non-configurable) limits set for these numbers?
Are there any 3rd party options that folks have used with Openziti to mitigate such attacks? Generally speaking, it would seem difficult to insert a WAF in front of a controller or router due to mTLS requirements.
We have table topped this exact scenario a couple of times over the last few years, and there are a few different methods. Attacks against routers are fairly simple to mitigate by spinning up new routers and deleting the old ones. Users attached to the network will receive the new router information within their session and can migrate to using the new nodes in minutes. The other routers could be left to absorb the attack and not let the attacker know they are no longer in use or removed. Only if an attacker was an authorized user could they retrieve the information needed to migrate the attack to the new nodes.
In the case of the controller, distributed controllers are in development, and nearing beta. This would allow similar maneuvers in that case. Also, the current single controller has a migration feature to announce a new controller. Once endpoints are authenticated and authorized to the new controller, they receive the information inside the session and reconnect to the new controller. This would allow a new controller to be built, preferably from a snapshot of the old one, and traffic migrated while leaving the old controller in place if the user chooses. This should be planned in advance as the certificate material, configuration database, etc. must be available for the build of the new controller, which may not be available due to the attack.
Lastly, there is development ongoing to utilize eBPF in the zfw (Ziti firewall) that is available today in the NetFoundry, Inc · GitHub project. The new feature will collect a map of the currently valid connections from Edge Routers and those endpoints having successfully authenticated, and if activated, would only allow IP traffic from those systems, blocking all others at the earliest point. While this would prevent attachments from new IPs, it will provide a considerable amount of functionality while other steps are taken. Some development of a new feature to simplify the collection of the addresses is being done to enable that process, so it is not available yet, but has been proven, and is mostly done, pending the last work to collect and maintain the map.
All of these would need a runbook approach specific to the environment, and the zfw will only work in systems that support eBPF, but a very strong DOS/DDOS mitigation plan can be developed using what is already available, and what will be in the near future will make it significantly stronger.
1 Like
@gormami
Thanks a lot for your reply. This definitely helps.
Would you happen to have any thoughts on throttling existing sessions to a maximum limit? I am trying to understand options to prevent a single edge-client from usurping the bandwidth of an entire router.
Nothing I'm aware of, don't think it's a problem we've had to tackle. If it is important, you might put up a feature request issue in the repo. To date, networks havebeen scaled out to handle whatever traffic was necessary, as they have been org focused not "retail", so any bandwidth issues were resolved by adding resources, rather than constraining. zrok, a project built on OpenZiti, has limits, but they are volume based (X GB per month, for example) rather than real time, and those limits are enforced by the zrok application, not OpenZiti, using metrics from OpenZiti.