How 360’s LVS Evolved: FULLNAT, Syn‑Proxy Security, and New Redundant Deployment

Background

LVS (Linux Virtual Server) is a versatile, high‑performance load‑balancing technology that handles the majority of traffic for Qihoo 360, including core services such as 360 Search, Website Guard, and Huajiao Live. As business volume grew, the platform required better performance, stability, and scalability, leading to continuous evolution over more than six years.

Features

1. FULLNAT Mode

Traditional LVS supports NAT, TUNNEL, and DR modes, each with drawbacks such as the need for real servers and the LVS to reside in the same VLAN, high deployment cost, and complex configuration. To overcome these issues, a new forwarding mode called FULLNAT was developed.

In FULLNAT, inbound packets undergo DNAT and SNAT: the destination IP is changed to the real‑server IP, and the source IP becomes an internal BIP (backend IP pool) while the session information is stored. Outbound packets look up the session table to restore the original VIP and client IP. This three‑layer communication enables cross‑VLAN deployment and leverages NIC multi‑queue features by assigning a dedicated internal BIP for LVS‑to‑real‑server traffic.

The mode masks the real client IP, so the original client address is saved in the TCP option field, and a custom ttm module allows real servers to retrieve it.

FULLNAT has proven reliable and performant after six years of production use.

2. Security – Attack Protection

LVS’s primary function is load balancing and failover, which does not natively defend against flood‑type attacks. Attack traffic would be forwarded to backend real servers, overwhelming them and consuming LVS CPU resources.

Two measures were introduced:

Integration of a Syn‑Proxy feature that implements a TCP SYN‑cookie‑like mechanism, completing the three‑way handshake with the client before establishing a connection to the real server, thereby mitigating SYN‑flood attacks.

Adoption of a “drop session if not found” policy, which discards packets with ACK/FIN/RST or other TCP flags when no corresponding session exists, protecting against additional TCP‑based attacks.

These strategies offload malicious traffic before it reaches real servers, improving overall system resilience.

3. UDP Limitation and New Tunnel Mode

While FULLNAT supports TCP source‑address transparency via the TCP option field, UDP lacks a comparable mechanism. To address this, a new LVS tunnel mode was designed. Unlike traditional tunnel mode, both inbound and outbound packets pass through the LVS, and real servers only need a custom ipip module without binding VIPs or configuring arp_filter, simplifying large‑scale deployments.

The model preserves client address information by encapsulating packets with ipip headers, enabling transparent UDP traffic.

4. Deployment and Operations Architecture

Typical data‑center deployment consists of multiple LVS servers in a cluster, each advertising the VIP via OSPF to the external core network and connecting directly to the internal core for communication with real servers, providing redundancy between LVS and real servers.

This model, however, suffers from two issues: lack of load balancing on dual external core links and insufficient high‑availability for the dual internal/external core structure.

To resolve these, a new redundant architecture was co‑designed with the NetOps team: LVS servers connect upstream to two 10 Gbps TOR switches via OSPF, and downstream to two internal cores via BGP. This design improves both load balancing and redundancy.

Conclusion

After years of iteration, the 360 LVS team has accumulated valuable experience in functionality, security, stability, deployment, operations, and troubleshooting, enabling the platform to meet continuously evolving business demands.