Protect your public server!

During the week, a DDoS attack with over 384 IP addresses from all over the world was started on our public Oracle APEX server. As we do like people (and proper bots) to make use of our free services to provide information about Oracle APEX, the bots managed to request an APEX page around 1.000 times a minute! With that many page views, the connection pool of Oracle REST Data Services (ORDS) with maximum 30 DB connections got full and people were starting to see an error message from ORDS. Of course we could scale up the connection pool with the database, but that would mean we had to scale up our hardware too, as ORDS was already taking up 100% CPU. Instead, my colleague Moritz Klein quickly found a way to throttle-down the requests in our proxy server installed in front of ORDS. We used a module called "event MPM". The configuration looks like this:

# event MPM
# StartServers: initial number of server processes to start
# MinSpareThreads: minimum number of worker threads which are kept spare
# MaxSpareThreads: maximum number of worker threads which are kept spare
# ThreadsPerChild: constant number of worker threads in each server process
# MaxRequestWorkers: maximum number of worker threads
# MaxConnectionsPerChild: maximum number of requests a server process serves
<IfModule mpm_event_module>
StartServers 2
MinSpareThreads 25
MaxSpareThreads 75
ThreadLimit 64
ThreadsPerChild 25
MaxRequestWorkers 75
MaxConnectionsPerChild    0
</IfModule>

What I like about this module, is that it doesn't reject requests, but it puts them in a queue instead. The user accessing the site has to wait longer before the application responds, but doesn't receive an error. After enabling this, the ORDS errors in the Tomcat log went away and the CPU load went down to 70%.

Of course, this didn't solve the problem. The question now was how to get rid of these requests? Nice bots, like Yanbot, just need a file called robots.txt and will respect this, but bad bots just ignore this standard.

We decided to install fail2ban and let it catch all incoming requests somewhere in the weekend (when normally very little traffic is expected) in order to block these directly at the firewall. After that, we set it to allow x requests in a certain time span, so that most people can still visit the site, but most new bots will be rejected at the firewall. For those of you that are not aware of what fail2ban is, it analyzes log files and blocks IP addresses for a certain time in the firewall (using iptables) based on criteria you define. Here is a great blog post about fail2ban: https://pipo.blog/articles/20210915-fail2ban-apache-dos

If you are not into Linux, you might want to use Cloudfare or Akamai and put their service in front of your server. For all Linux experts out there, I can also recommend mod_security, as it will give you even more possibilities to defend you from DDoS attacks, but it does require a 2-day training to get the hang of it. :)

Hope this helps to protect your public server before it hits you as well.