Recently to help friends deal with an online education platform traffic anomalies, log in to the server to take a look, good guy, the number of TCP connections directly soared to more than 100,000, the CPU runs full, the entire video stream card into the PPT - a typical SYN Flood attack site. This year, engaged in video services, not by the SYN attack whacked are embarrassed to say they are doing Internet business.
SYN attack is to say that the attacker frantically send TCP connection request, but after receiving the server's SYN-ACK dead not back to the ACK. the server stupidly left half-open connection and so on the response, until the resources are exhausted. Video business is extremely sensitive to latency and stability, once the connection pool is full, normal users can not even connect, let alone watch 4K HD video.
In the early days, many teams thought that buying a traditional firewall can be dealt with, and found that the pure hardware equipment simply can not carry a large number of false connections. I have tested a brand of firewall, 200,000 SYN packets per second to deal with the direct lying flat, but also incidentally, the normal traffic is also killed by mistake. Do not believe those “all-purpose solution” ads, SYN attack protection must be combined with protocol characteristics and business scenarios to do in-depth optimization.
The real effective program is to rely on the video high-defense CDN in the edge node to complete the TCP connection proxy. Simply put, let the CDN node instead of the source station and the user to build a connection, through the authentication mechanism to filter out malicious requests. Here is a key detail: CDN can not simply drop packets, you have to simulate the behavior of the real operating system protocol stack, otherwise the attacker will be able to break the disguise at a glance.
For example, CDN07's protection strategy is very interesting: the first SYN packet is directly released and fingerprints are recorded, and when the same source IP repeatedly initiates a connection within 10ms, the client is required to complete a cryptographic challenge first. Tests found that this can filter 99.7% of forged packets, and the impact on normal users is less than 0.2 seconds. Their nodes can even emulate different TCP stack characteristics for Linux and Windows, making it difficult for attackers to recognize the real environment.
Take a look at an actual configuration example (based on the Nginx extension module):
The max_half_open parameter is particularly important - it controls the maximum number of half-open connections allowed per edge node. It is recommended to dynamically adjust according to the business traffic, such as live peak hours can be appropriately relaxed, but must be coupled with real-time monitoring alarms.
08Host's approach is even more radical: it directly modifies the kernel TCP stack to compress the SYN_RECV status timeout from the default 75 seconds to 8 seconds. Although it will hurt some of the high latency users, but for the defense of DDoS attacks immediate effect. They tested data in the Asian nodes show that this approach can withstand 1.5 million SYN attack packets per second, while the server memory consumption of less than 20%.
However, this program has a side effect that may affect the connection success rate of multinational users. Later, they engaged in intelligent geographic scheduling: standard timeout policy for European and American users, and aggressive mode for regions with high incidence of attacks. This requires global nodes to synchronize state data, and the technical implementation is quite complex.
Speaking of node synchronization, I have to mention the pit of state sharing. Early CDN5 used Redis cluster to synchronize the connection state, as a result of the attack came after the Redis first hang. Now the mainstream programs are using consistent hash to do local state caching, although the data is slightly delayed, but to ensure that the protection system itself will not become a bottleneck.
Video services also have a special need: protection cannot interfere with the QUIC protocol. Many video streams now use QUIC instead of TCP, but SYN attack variants are also starting to target the QUIC handshake process. A good protection scheme needs to handle both TCP and QUIC handshake flooding attacks, such as CDN07's hybrid protection model is worth referring to:
In addition to the technical implementation, the business level strategy is equally important. It is recommended that video services of different levels of importance be assigned independent protection resources:
Core Live Streaming: Highest level of protection, allowing 11 TP3T of false positives but guaranteeing 99.991 TP3T of availability.
video on demand: Enable medium protection and secondary validation with user behavioral analysis
Administrator's background: Full whitelisting mode, preferring to kill a thousand by mistake rather than spare one
Finally, let's give the group data comparison, last year to test the effectiveness of the protection of the three major manufacturers:
CDN5: 800,000 SYN packets processed per second, CPU overhead 12%, false positive rate 0.8%
CDN07: 2.1 million SYN packets processed per second, CPU overhead 23%, false positive rate 0.3%
08Host: 1.5 million SYN packets processed per second, CPU overhead 8%, false positive rate 1.5%
It can be seen that there is no perfect solution, high processing power is often accompanied by higher resource consumption. CDN07 is suitable for large platforms with complex business, 08Host is suitable for cost-sensitive medium-sized projects, and CDN5 performs better in terms of balance.
Don't forget to optimize continuously after deploying protection. Once a customer configured to rest on their laurels, the attacker changed to attack the SSL handshake phase, as usual, hit the full CPU. now the mature protection system should be included:
- TCP connection protection
- SSL/TLS Handshake Optimization
- Protocol Fingerprinting
- Real-time traffic scheduling
To be honest, the biggest headache in this business is not the technical implementation, but the cost of countermeasures. Attackers rent botnets for a day only a few hundred dollars, the protection program investment is not moving a million dollars a year. It is recommended that startups first use cloud vendors pay-per-use programs, business volume up and then consider hybrid deployment.
In short, SYN protection is like wearing bulletproof vests for video services - you can't expect to be invulnerable, but at least you can stay in one piece when you get whacked. The key is to understand the principle of attack, according to the actual business characteristics of the choice of program, do not be sales talk to take the side. After all, these days, even the CDN have to “defense teammates” (a vendor secretly use customer nodes to do traffic cleaning is not a paragraph).
Next time you encounter a video lag do not rush to dump the pot to the encoding parameters, check the TCP connection status may have a surprise. After all, in the eyes of the attacker, video platform is a piece of fat meat, SYN attack is just the lowest cost of the entry paragraph greeting.
