When the game server is crashed, the most common pot-shaking posture in the technical group is: "This network fluctuation, the gods can't save ah". But the truth is often, your CDN may not have chosen the right one, or the configuration is wrong. I have tested no less than ten mainstream high-defense CDN, packet loss rate can be stabilized to 1% below, two hands count over.
Last year, one of our FPS games was in public beta, and there were almost 100,000 people online at the same time during the peak period. The server itself could handle it, but the players reported "unstable network connection". A check of the monitoring, edge nodes to the user side of the packet loss rate soared to 15%, which still play a hammer? The latency is low again, the packet loss is high, as usual, card into the PPT. later we spent two months tuning, the global packet loss rate to 0.8%, today, the actual test of the effective program pulled out to you.
Let's burst a few fantasies first:Don't believe in those CDN vendors that blow "BGP lines", BGP is just a routing protocol, it does not mean quality. I've encountered a vendor's "BGP multiline" node, the evening peak packet loss rate is higher than the second-tier ISP. What really affects the packet loss rate are three things: physical distance, line congestion, and the number of "handshake" carriers in between.
For example, the user from Heilongjiang connected to the Shanghai server room, the traffic may jump five or six times. Beijing shakes a bit, Jinan circles around, Nanjing and then make a card. Each link may lose packets. The core task of intelligent routing is to turn this process into a "point-to-point direct connection".
The scheme we use is dual-link probing + dynamic route switching. The client SDK sends a 4KB probe packet to the edge node every 10 seconds, and the node calculates the packet loss rate and delay. Once the threshold is exceeded (e.g., packet loss > 2% or latency > 80ms), it immediately cuts to the backup line. Here's a pitfall: don't use ICMP probing! Many carriers will limit the speed of ICMP, and the data is not allowed at all. You have to use TCP or UDP custom ports.
This is the core logic of our homegrown probe script (Python pseudo-code):
Detection is not enough, node quality is the foundation. We have compared three vendors: CDN5, CDN07, 08Host. the results are outrageous - the same area, CDN5 node packet loss rate can be lower than 08Host 40%. why? Because CDN5 self-built backbone network pop points, while 08Host rented second-hand bandwidth. These days, even CDN have to "defense teammates", you never know how many second-hand dealers behind the node you rent.
The core of node optimization is 'going to the middleman'.We later signed a deep contract directly with CDN5, requiring all nodes must be accessed by the three major carriers A class room, and the number of jumps does not exceed 3. By the way, pick a black material: a vendor called "global 800 nodes", in fact, 600 is a virtual node, the essence of the resale of AWS and Google Cloud. This node is usually okay, in case of DDoS directly lying flat.
Another pitfall of high defense CDNs is that the protection rules themselves can increase packet loss. For example, SYN Flood protection may accidentally kill normal TCP handshake. We have suffered a loss - after turning on the "super protection mode", normal players could not connect because the TCP handshake was forcibly reset three times. It was solved by changing to dynamic fingerprinting:
Finally, a metaphysical problem: cross-network scheduling. Mobile users go to telecom nodes, the packet loss rate is inherently high 3%. but many CDN vendors in order to save costs, only in a carrier pile of nodes. Our current strategy is "three nodes for three networks": each region deploys at least three nodes for telecom, mobile, and Unicom, and then uses Anycast IP as an entry point. Users will always be resolved to the same carrier node.
Comparison of measured data: before optimization, the average packet loss rate of cross-network access is 12.7%, and same-network access is 4.3%. after optimization, cross-network transmission is forced to be same-network through Anycast, and the global packet loss rate is pressed down to 0.8%. the cost has increased by 25%, but the players' complaints are less than 90%, which is a good deal of blood money.
One more tip: adjust the MTU value. Game packets are generally small, and an MTU of 1200 is less likely to be fragmented than 1500. Losing a slice can be a chain reaction - one slice lost and the whole packet retransmitted. We deployed MTU optimization on Linux nodes at full capacity:
To be honest, there are no more than five in China that can achieve a packet loss rate of less than 1%.CDN5 performs solidly as an old dog in East and South China, but CDN07 is recommended for Northeast China - they have a deep peering agreement with Unicom.08Host is cheap, but is only suitable for games that aren't sensitive to latency. If you are doing competitive games, don't save this money. Player churn could just double.
Finally throw a storm theory: see CDN vendors reliable or not, directly let him open the real-time monitoring background. Dare to show you 48 hours delay and packet loss curve, generally not too bad. Those who will only dump the "full network coverage schematic", basically a picture of the cake. We now have our own monitoring system, every 5 minutes from the country's 200 detection points to send packets, lower than 99.5% availability of nodes directly offline automatically.
Tossed for so long, the biggest feeling is: there is no silver bullet to reduce the packet loss rate, it's all about piling up the details. From the routing protocol to the cabinet temperature, each link can be overturned. But as long as you chew through the "intelligent routing" and "node optimization", the packet loss rate does not dare to be arrogant with you.
