How deep is the pool of the chess industry, those who have engaged in it know. DDoS hits like a mad dog, but the most insidious are often the slow attacks that grind you to death - it's not as bombastic as a flood attack, but rather like a frog boiled in warm water, by the time you realize it, the server has long been paralyzed.
Last year, I helped a chess platform to do emergency response, their operation and maintenance at first also confident: “bandwidth is enough, firewall rules are stacked”. The result of a look at the monitor, the CPU did not explode, the traffic did not surge, but the user is stuck cursing. Finally, packet analysis only to find that there is a slow IP every 30 seconds to send a HTTP packet header, connection pinch in the hands of the dead do not let go, a server by hundreds of such connections to exhaust the resources.
The nasty thing about this Slow HTTP Attack is that it looks too “normal”. A traditional firewall might just let it go, because every request on its own conforms to the protocol specification. But the attacker is relying on this “slow” to exhaust your connection pool, thread pool, and ultimately prevent normal users from accessing.
Don't count on a hardware firewall to fix this.. I've actually tested that a certain brand of multi-million dollar hardware firewall with default policy is almost unresponsive to Slowloris attacks - because it detects traffic spikes, not connection durations. What can really carry it is also a chess high defense CDN combined with accurate timeout control and behavioral analysis.
Let's start with the request timeout setting. It's not just a matter of changing `keepalive_timeout` in Nginx. You have to break it down into several dimensions based on business characteristics: connection establishment timeout, header reception timeout, body transmission timeout, idle connection timeout. For example, in chess business, general HTTP request should be completed within 2 seconds, WS long connection may maintain minute level, but there must be a heartbeat mechanism under the hood.
Taking the Nginx configuration as an example, I usually layer it like this:
But that alone is not enough. Some attackers will intentionally send body slowly, for example, 1 byte per second. This time you need to set up a global flow control policy in the CDN edge node. Like CDN5 this home has done the breakdown: 10 seconds did not pass the default body size (such as 1KB) directly disconnected, and does not take up the back-end connection pool.
Even tougher is anomalous behavior interception. Slow attacks are hard to defend against because a single request looks harmless. But if you look at multiple requests in context, the devil is in the details.
I typically catch anomalies in three dimensions:connection speed(Normal users don't keep creating new connections without sending requests),transmission rate(Normal users don't take 10 minutes to pass a login package),Distribution of survival time(While chess user sessions usually fluctuate, attack connections tend to be unusually stable.)
CDN07“s algorithm team has developed a set of feature libraries specifically designed to catch this kind of ”fake slow" behavior. For example, an IP is detected to build a connection every 10 seconds, but each connection only sends 5 bytes and then remains silent, and if it is more than 10 times in a row, it will directly enter the challenge process. Measured kill rate of less than 0.1%, but can intercept 99% slow attack variants.
Another trick is to do a checksum on protocol integrity. Many slow attack tools actually have flawed protocol implementations (e.g., intentionally missing certain header fields). We can pre-build checksum logic on the CDN:
Of course, the above strategies should be utilized flexibly. For example, API interfaces and WS long connections must be treated differently. 08Host's chess-specific nodes do intelligent identification: strict timeout (10 seconds) is used for the /api/ path, while long connections are allowed for the /ws/ path but heartbeat packet intervals of no more than 25 seconds are required.
Never get carried away with a single solution. There was once a customer who bought a CDN from a major manufacturer and thought everything would be fine, but the result was that Slowloris was pierced. It was later found that the default timeout settings of the CDN were too loose (120 seconds of idle connections were allowed). Later, we worked with CDN5“s customized policy to reduce the timeout threshold to 15 seconds, and at the same time enabled its ”Slow Connection Learning Mode" - first release 24 hours to establish a baseline, and then automatically intercept IPs that deviate from the baseline behavior.
Lastly, I would like to mention a pitfall: too aggressive timeout settings may mistakenly hurt slow network users. My experience is to combine the geographical policy, for example, for domestic users with a uniform 10-second timeout, for Southeast Asian users to relax to 20 seconds, at the same time with the center of gravity algorithm to determine - if the user IP is very far away from the physical distance but the latency is very low, the probability that it is a forgery of cross-border traffic.
In practice I also like to bury some hidden stakes. For example, in the CDN configuration mixed into a few cold domains as bait, these domains will only be scanned by attackers, normal users will never visit. Once these domains are found to have slow connection attempts, directly block the entire ASN segment. This trick has helped me stop several waves of targeted attacks.
In the end, the prevention of slow attacks is like catching loaches, can not just rely on hard blocking, you have to learn to “listen to the sound of the position”.CDN's advantage is that it can see the global traffic, combined with timeout control and behavioral analysis, and can often be found in the early attack on the seedling. But remember, there is no once-and-for-all program, today's effective rules may be bypassed next month, so be sure to leave a channel for manual research and judgment - machine judgment 70% suspicious on the alarm push, so that people make the final decision.
If you are using CDN07 or 08Host, remember to go to the background to dig their slow attack protection module (some hidden in the “advanced security” or “custom rules”). Most customers do not even know these features exist, and it is a waste of the annual premiums. After all, these days, even the CDN have to “defense teammates” - not to prevent attackers, is to prevent your own will not use.
