How to defend the server was CC attacked, I'll tell you how I do it!

I remember last summer, I was paralyzed on the sofa brushing drama, the phone suddenly like crazy buzzing vibrate non-stop - all the server alarm mail, CPU usage rate soared to 100%, the site opened as slow as a snail crawling. I cursed a mother, I understand: this eighty percent is another CC attack. To be honest, after so many years of operation and maintenance, I've seen a lot of such scenes, but every time you can be angry with the teeth, because the attackers always love to pick late at night, forcing you to stay up all night to put out the fire.

CC attack, to put it bluntly, is a group of “robots” disguised as normal users, crazy refresh your web page or submit a request, the purpose is not to hack into the system, but to live to the server resources to dry up, so that real users can not access. I found that the actual test, this thing is particularly disgusting, because it is not like traditional DDoS like a huge flow, but instead of simulating the operation with a real person like, many basic firewalls straight to a halt, mistakenly thinking that this is normal access.

Once, one of my e-commerce stations was screwed, the order page got stuck, the customer complaint phone almost went off the hook, and my boss rushed straight into the server room to ask me if I was mining.

These days, even CDNs have to be ‘teammate-proof', not to mention these underhanded tricks.

At that time I had a bare-bones server in my hands, and I relied on Nginx to carry it hard, and it was instantly washed out. I realized that it was impossible to fight alone, and I had to do a combination of things. Below I break open the crumbs to say, how I step by step to build up the defense system, you can refer to, but do not copy, because each business scenario is different.

First of all, don't just start messing around, you have to figure out what the attack looks like. I immediately logged into the monitoring system, staring at the real-time logs, and found that an IP in a few seconds sent thousands of requests to the login page, which is obviously not normal. I did a quick sift through the command line, and here's the command:

tail -f /var/log/nginx/access.log | awk '{print $1}' | sort | uniq -c | sort -nr | head -20

This line of code lists the top 20 most frequently requested IPs, pinpointing suspects at a glance. But it's too slow to do it manually. I've already set up an alert rule in Zabbix to automatically send me a text message if an IP is requested more than 50 times per second. Don't believe in those “monitoring is optional” nonsense, without it you are like a blind man fighting, only passive.

After pinpointing the source of the attack, my first reaction is to temporarily block the IP. use iptables to just pull the plug, for example:

iptables -A INPUT -s Malicious IP address -j DROP

But this trick treats the symptoms but not the root cause, and the attacker can keep messing around with a different IP segment. So I turned to configuring rate limiting in Nginx, which is the way to go. I added this paragraph to nginx.conf:

This setting means that each IP request up to 10 times per second, more than part of the first cache 20, and then over the direct return 429 error.

I have tested, this can block most of the low-ranking CC attacks, but meet the advanced point, IP pool large, or struggle. Once I adjusted too hard, mistakenly blocked a few search engine crawlers, SEO traffic dropped a lot, the boss and find me tea, so that the rules have to slowly adjust, do not cut across the board.

Next, I decided to go to a CDN service to filter the traffic to the edge nodes first. At that time, I compared several, and finally chose CDN07, why?

Because their intelligent protection is really not blowing, can automatically identify malicious requests based on behavioral analysis, but also with global acceleration, the price is also affordable. Configuration is also simple, I cut the domain name analysis to their CNAME, and then open the CC protection rules in the console, the threshold is set to 100 requests per second, more than the challenge of the verification code. After using, the server pressure immediately dropped eighty percent, customer service response is also fast, once in the middle of the night attack, they 5 minutes to help me adjust the rules. These days, a good CDN is like a personal bodyguard, saving too much worry.

CDN alone is not enough, I added a Web Application Firewall (WAF) at the server level, using the open source ModSecurity.

After installation, I customized some rules, such as brute force protection for the login page:

This rule checks for anomalous parameters in login requests and intercepts them if the cumulative score is high. I remind you that WAF rules should not be copied from the Internet, but should be slowly ground according to your own business. Once I added a strict rule, the result is that all the API requests of my own APP were blocked, and the users were blown up, so I had to roll back overnight. I had to roll back overnight. It's a lesson in blood and tears to run through the test environment before going online!

Also, I optimized the server resources.

For example, I cached the database queries and used Redis to store hotspot data to reduce the pressure of pressing directly to MySQL. I added some delayed responses to the code to return fake data for suspicious requests, draining the attacker's resources. It's a bit damaging, but effective, and I wrote a simple PHP script example:

This slows down the attacker's machine, while normal users are accelerated by the CDN and hardly feel the delay. However, this has to be used in conjunction with monitoring, or it would be a disaster to accidentally hurt a friendly.

Speaking of which, I have to complain, some tutorials always like to recommend “one-click protection tool”, blowing the sky high, I tried a few, the effect is uneven, and some even come with a backdoor.

There is no silver bullet to defend against CC attacks, the key is multi-layer depth: network layer with CDN to carry traffic, application layer with WAF and rate limit filtering, data layer to do a good job of caching to reduce the negative.

In my current architecture, I also added regular drills, such as simulating an attack once a month to check for vulnerabilities. One drill found that a certain API interface was not limited, and it almost became a breach, so I hurriedly patched it up.

Finally, let's talk about mindset. Defense is like a game of cat and mouse, the means of attack change every day, you can not set the rules and lay flat.

I've made it a habit to look at log summaries every day, keep an eye on new threat intelligence, and tune the rules as necessary. For example, last year, there was a wave of attacks on static resources, so I limited the speed of the image directory in Nginx, and the effect was immediate. In short, remain vigilant and continue to learn, which is fundamental.

After all that, my server is now steady as a rock and has never been crashed by a CC attack again.

If you have just entered the pit, remember these points: monitoring first, fast response; layer by layer defense, do not rely on a single point; tools to choose reliable, like CDN07 service providers can save a lot of effort; finally, practice makes perfect, more testing and more adjustments.

There's a long road to defense, but if you're willing to toss it around, you can always find a method that works for you.