Unwanted traffic #
In my spare time I run an AdFilter(ing) public DNS service that is available via DoT/DoH. While I've had reasonably little trouble, occasionally when running public services you can get unwanted traffic for whatever reason.
Recently I got alerts for resource usage on one of my servers and when checking the traffic graph for the server it looked like this.
Over a period of a few hours Internet traffic to/from the server had increased to a level 10x the normal levels for this server. This traffic increase resulted in increased CPU and memory utilisation, which in turn caused alerts for me to investigate.
Using tcpdump
I determined that the traffic increase (and now the vast majority of traffic to this server) was DoH at ~6000 packets per second from an Internet provider in Turkmenistan.
Given DoH is TCP port 443, I assume the traffic is somewhat legitimate as opposed to say a UDP DDOS flood, however I did note that that traffic was concentrated to a spread of IP addresses across a /22 network subnet.
This particular server was a smaller virtual machine, as the traffic continued to increase it started to cause CPU/RAM resource exhaustion on the server which resulted in impact to other users of the service.
Rate limiting via UFW #
I use UFW to manage rules to only allow traffic in/out from my servers that I want. UFW does have the ability to add rules to rate limit connections so I thought I would give it a try, it was as simple as:
sudo ufw limit from 95.x.x.0/22 to any port https
After implementing this I found that the default limit rule for UFW is 6 connections over a 30 second window, given I was applying this rule to a network subnet it was far too restrictive.
Modifying the UFW default rate limit #
I tried editing the full firewall rules as defined in /etc/ufw/user.rules
to something more along the lines of what I wanted, however this appeared to be overwritten with the defaults when issuing a reload of UFW.
Given I wasn't using the UFW rate limit on this machine for anything else, I decided the best quick fix was to edit the script that builds the iptables rules for UFW.
I changed the values for seconds and hitcount to something more appropriate for what I needed and issued sudo ufw reload
, this rebuilt and applied the rules with the updated default values.
As the rate limit is tracking new connections (not packets per second or DNS queries), I had to iterate a few times to find the right set of values.
The desired outcome was to limit traffic enough to manage the server load at a point where the traffic was no longer impacting the server resources and other users.