For users who deploy IPFire in a data center, a new interesting feature has arrived: SYN Flood Protection. In this article, I would like to give an introduction to Denial-of-Service attacks and what old and new mitigations IPFire has in stock against them.
(Distributed) Denial-of-Service Attacks
Denial-of-Service attacks are a common thing on the internet. They are basically designed to take down a website - or even a whole data center - by overwhelming a system. There are many ways for this, either by sending a complicated request that keeps the system busy, or simply by sending an insanely large amount of requests a system cannot cope with in short time. There is a very generic way to archive the latter, just open as many connections as possible, but never send any real data. The server side might sooner or later run out of memory as each connection needs to have its state stored somewhere and will therefore refuse to accept legitimate connections. This is called a SYN Flood attack.
What is a SYN Flood?
To understand a SYN Flood attack, we have to understand first how a TCP connection works. It all starts with a Three-Way-Handshake. The client sends a SYN packet to the server which receives the packet, allocates some memory to store the new connection and responds with a SYN-ACK packet. The client then responds with a final ACK packet and the connection has now been completely established and is ready to transfer data.
Client Server
| |
| SYN -------------------> | [Creates a new socket]
| |
| <--------------- SYN-ACK |
| |
| ACK -------------------> |
| |
| [Connection Established] |
| |
| <-------- DATA --------> |
| <-------- DATA --------> |
| <-------- DATA --------> |
| |
Both, the server and client have exchanged some basic data about the connection. They know the IP addresses and ports to use to communicate with each other and have synchronised a sequence number - hence the packet is called SYN. Although this data is only a few kilobytes, with thousands or millions of connections, this can consume a significant amount of memory.
But this is not only expensive for the server, the client will have to store the state, too. To launch a proper attack, this will become too expensive and will have to be cheaper. But how to do this? How about just firing the SYN packet at the server and immediately forget that we sent it? That way, we won't have to use any resources apart from network bandwidth, but the server will still respond with a SYN-ACK packet and wait for the final ACK.
Attacker Server
| |
| SYN -------------------> | [Creates a new socket]
| x <------------- SYN-ACK |
| |
| SYN -------------------> | [Creates a new socket]
| x <------------- SYN-ACK |
| |
| SYN -------------------> | [Creates a new socket]
| x <------------- SYN-ACK |
| |
| SYN -------------------> | [Creates a new socket]
| x <------------- SYN-ACK |
| |
...
What is IPFire's role in this?
How is this related to the firewall? IPFire technically is a stateful-inspection firewall. It will track the state of connections to allow packets through that belong to a valid connection more efficiently. For that, it keeps a so called connection tracking table with all open or almost open connections. You can see that table if you click on "Status -> Connections" on the web UI. This also needs memory and can grow really large. The larger is becomes, the slower the firewall will also become in passing packets. That is not a problem at all in normal operation, but attackers might want to exploit this by opening a large number of connections to slow down the firewall - or deny users its service.
If you are running a service like a mail or web server and you have set up a port-forwarding for clients to reach that server, the firewall will keep one entry in the connection tracking table per connection. The question is now, which system is running out of memory first for the attack to work? The server or the firewall?
SYN Flood Protection to the rescue
The firewall is there to protect the network that is behind it. That obviously includes to filter any unwanted data traffic from the internet into that network. In case when packets arrive for a port that isn't open or they arrive from a blocked origin, this is an easy task. Simply check if any of the firewall rule matches and drop the packet if there hasn't been a match. But what to do when there is a valid firewall rule accepting connections that later on overwhelm the service the firewall is supposed to protect? We would need a way to detect what is a legitimate connection and what is just a loose SYN packet designed to take the service down.
This can be achieved using SYN cookies.
A SYN cookie is a specially crafted SYN-ACK packet that is generated by the server side. When the client sends a SYN packet, the server usually generates a random sequence number and sends it to the client. That changes when using SYN cookies, because instead of storing the state of the new TCP connection in memory, it is being stored in the especially chosen sequence number. Actually it is not the entire state, but just a cookie that the server later can decode and reconstruct the initial connection state.
The cookie mainly consists of a timestamp (to avoid any replay attacks) and a cryptographic hash over the source and destination IP addresses as well as ports. The server will send this to the client which will increment the sequence number by one when replying with the ACK packet. The server can then subtract one from the value and check against the original cookie. If this matches, the ACK packet belongs to a connection where the server has seen the initial SYN packet before without storing any state. The connection is then opened and works as usual.
In a setup where the server uses SYN cookies like this, the firewall installed in front of the server is still susceptible to memory exhaustion. That is because it simply forwards the SYN, SYN-ACK and ACK packets for each connection.
Client Firewall Server
| | |
| SYN -------------------> | [Creates a CT entry ] |
| | -----------------------> | [Creates a new socket]
| | |
| | <--------------- SYN-ACK |
| <------------------------| |
| | |
| ACK -------------------> | |
| | -----------------------> |
| [Connection Established] |
We would need the firewall to deploy the SYN cookie instead of the server behind it. But that isn't as easy, because it is supposed to forward packets and not send any on its own. A possible solution could be using a reverse proxy like HAProxy, but that means that there is an extra piece of software running on the firewall and there has to be a proxy that actually supports the protocol that the client and server speak. It would be a lot easier if we simply have to care about TCP and not about any higher layers. For that, we have a new feature called a SYN proxy, which works as follows:
Client Firewall Server
| | |
| SYN -------------------> | |
| | |
| <--------------- SYN-ACK | |
| | |
| ACK -------------------> | [Creates a CT entry ] |
| | SYN -------------------> | [Creates a new socket]
| | |
| | <--------------- SYN-ACK |
| | |
| | ACK -------------------> |
| | |
| [Connection Established] |
The firewall performs the entire three-way handshake on behalf of the server and only forwards the connection once it has been fully established. That is amazing, because we can use this to mitigate Denial-of-Service attacks! If an attacker sends a large number of SYN packets, the firewall will respond with SYN-ACK packets that contain a SYN cookie. The attacker won't ever respond to those packets, but a legitimate client will and therefore the firewall can easily open the connection to the server. The server itself is entirely oblivious that an attack is happening because the firewall is smart enough to filter it all away.
Attacker Firewall Server
| | |
| SYN -------------------> | |
| x <------------- SYN-ACK | |
| | |
| SYN -------------------> | |
| x <------------- SYN-ACK | |
| | |
| SYN -------------------> | |
| x <------------- SYN-ACK | |
| | |
Legitimate Client
| SYN -------------------> | |
| <--------------- SYN-ACK | |
| ACK -------------------> | [Creates a CT entry ] |
| | SYN -------------------> | [Creates a new socket]
| | <--------------- SYN-ACK |
| | ACK -------------------> |
| [Connection Established] |
(Distributed) Denial-of-Service Attacks In The Real World
We are living in times where bandwidth is cheap. Assuming that the most common uplink in a data center is one Gigabit, an attacker only needs to generate one Gigabit of SYN packets a second to saturate your uplink and your service is offline. Since every cloud server comes with one Gigabit or more, this is neither difficult nor expensive to do. Once they send more packets than your internet connection can handle, there is nothing IPFire can do, because the legitimate packets have been dropped long before they reach the firewall.
So, you will need a bigger pipe than your attackers. But where to get that from? It does not seem to be a good investment of money to upgrade your internet connection only to accept malicious traffic. The attacker side might also do the same, so you are very quickly back to square one. But there is one option where you can rent a large amount of resources for any time you need: The Cloud!
Deploying DoS Protection in the Cloud
IPFire supports Amazon's Graviton Instances which - in the current network-optimised generation - support a network bandwidth of up to a whopping 200 GBit/s (C7gn). With IPFire's support for Amazon's Elastic Network adapter, that kind of high bandwidth isn't a problem to handle. It is the perfect pair to build your own DoS Protection in the Cloud! If that still isn't enough (and your budget allows) you can deploy many of these instances even in other regions instead of just the one.
The setup would work as follows: An instance of IPFire is being deployed in the cloud and you will direct all your users to it. With the SYN Flood protection enabled, you can forward all legitimate connections to your own infrastructure which may also be hosted on AWS or elsewhere (bonus points if you connect it using IPsec for extra security). Illegitimate connections might be blocked by the SYN Flood protection, but also by any of the other features that IPFire has to offer like our Intrusion Prevention System.
Clients Amazon Web Service Your Infrastructure
| | |
| Requests ------> |--------------------| Legitimate Requests ---> |
| ---------------> | | -----------------------> |
| ---------------> | IPFire | |
| | | |
Attackers | | |
| ---------------> | [Blocked] | |
| |--------------------| |
This is not even the end: Due to the large amount of features that IPFire offers, you could enable SYN Flood protection selectively by only using it for a couple of origins like bad actors that you have identified before, or by country. You could even direct traffic from certain places to one service that receives more legitimate requests and all the rest where you are not sure to another service. There are plenty of options and they will protect your infrastructure from those bad guys out there.
We Are Here To Help
As the landscape of cybersecurity threats becomes more complex, businesses have to build more robust defences against them. Our new SYN Flood protection feature offers Enterprise users a solution to mitigate the risks of downtime due to such attacks and brings you peace of mind. The Lightning Wire Labs Support Team is here to help you with the implementation and fine-tuning of IPFire's advanced features - whether you are a seasoned IT professional or just starting to fortify your network - don't hesitate to reach out to us. Together, we can tailor a solution to your specific needs and help safeguard your business. Get in touch today.