How 360’s LVS FULLNAT Transforms Load Balancing and Boosts Security

Background

LVS (Linux Virtual Server) is a high‑performance, general‑purpose load‑balancing technology that handles the majority of traffic for 360, including core services such as 360 Search, Website Guard, and Huajiao Live. After more than six years of development, the platform has continuously evolved in deployment and new features to meet growing business demands.

Feature Highlights

2.1 FULLNAT Mode

Traditional LVS supports NAT, TUNNEL, and DR modes, each with limitations such as same‑VLAN requirements, complex configuration, or inability to cross network segments. To overcome these issues, 360 introduced a new forwarding mode called FULLNAT.

The main differences from NAT are:

During inbound packet processing, both DNAT and SNAT are performed, converting the destination IP to the real‑server IP and the source IP to an internal BIP (backend IP pool), while preserving session information.

During outbound processing, the session table is consulted to restore the source IP to the virtual IP (VIP) and the destination IP to the client IP.

Because it operates at layer 3, LVS and real servers can communicate across different VLANs. An internal BIP address is used for efficient multi‑queue NIC utilization.

The downside is that the real server cannot see the original client IP; to address this, the client IP is stored in a TCP option field and a ttm module is provided for real‑server side retrieval.

FULLNAT simplifies real‑server configuration, enables cross‑segment deployment, and has proven reliable and performant after extensive production use.

2.2 Security – Attack Mitigation

LVS originally lacked protection against flood‑type attacks, causing malicious traffic to reach backend real servers and consume CPU resources. To mitigate this, two measures were added:

Syn‑Proxy: a SYN‑cookie‑based mechanism that proxies the TCP three‑way handshake, establishing a connection with the real server only after the client handshake completes, thus defending against SYN‑flood attacks.

Session‑absence drop policy: packets with missing session entries are discarded, protecting against ACK/FIN/RST‑based attacks.

These strategies drop malicious packets before they reach the real servers, preserving performance and stability.

2.3 UDP Challenges

While FULLNAT can embed client IP information in TCP options, UDP lacks such a field. To support UDP source‑address transparency, a new LVS tunnel mode was designed. Unlike traditional tunnel mode, both inbound and outbound packets pass through the LVS, and the real server only needs an ipip module, eliminating the need for VIP binding and complex ARP configuration.

2.4 Deployment and Operations Architecture

Typical data‑center deployment consists of multiple LVS servers in a cluster, each advertising the VIP via OSPF to the external core and directly connecting to the internal core for real‑server communication, ensuring redundancy.

This model, however, suffers from two issues: lack of load balancing on dual external core links and insufficient high‑availability when the direct core link fails.

Collaborating with the NetOps team, a new redundant architecture was devised: LVS servers connect upstream via OSPF to two 10 Gbps TOR switches, and downstream via BGP to two internal cores. This improves both load balancing and redundancy.

Conclusion

LVS is a critical infrastructure component for 360. Over years of practice, the team has accumulated valuable experience in feature development, security hardening, stability, deployment, operations, and troubleshooting, successfully supporting the continuous evolution of business requirements.