How Baidu Cloud Flow Log Boosts Network Visibility and Cuts Costs

1. Background and Problem

To improve problem‑location capability in ultra‑large scale networks and handle “10‑billion‑level IP pairs” scenarios, Baidu Intelligent Cloud released the Flow Log product.

The product offers basic network problem locating, traffic monitoring and visualization, allowing users to query and visualize dedicated line gateway traffic, enhancing diagnostic ability as bandwidth grows.

Compared with traditional self‑built flow statistics, Flow Log provides a standard product form, making cloud traffic more visual, plug‑and‑play, handling 10 TB‑level dedicated line traffic, reducing development and maintenance effort, and letting users focus on business.

2. Flow Log Introduction

Flow Log records network flow information sent and received by instances such as VPC cloud servers, dedicated line gateways, NAT gateways, etc. The data is aggregated and sent to Baidu’s log analysis service for storage and visualization.

Users can use Flow Log for fault diagnosis, business behavior analysis, security checks, and compliance audits, optimizing cloud network architecture and improving business security.

The product supports traffic mirroring analysis, fine‑grained statistics and security audit functions to meet diverse user needs.

3. Technical Architecture

The backend consists of several logical nodes:

Forwarding node: programmable hardware gateway with high throughput, low latency, low power consumption, and easy operation, supporting online 10 TB‑level dedicated line traffic.

Data node: aggregates mirrored traffic, supporting millions to billions of IP pairs.

Compute node: performs big‑data analysis on logs, processing billions of log entries per minute, handling storage and forwarding.

Log node: final destination for user data, currently supporting BOS and BLS.

4. Application Scenarios

4.1 Fault Diagnosis

Flow Log collects dedicated line traffic, helping users analyze historical or current traffic to understand service status, adjust ACL policies, locate packet loss points, and improve stability.

Classic case: a hybrid‑cloud user discovered ACL‑dropped traffic causing connection failures and quickly adjusted gateway policies.

4.2 Cost Allocation

Flow Log makes inter‑department traffic visible, simplifying cost sharing and bandwidth adjustment, aiding digital transformation.

Classic case: a hybrid‑cloud customer used Flow Log to allocate dedicated line costs across dozens of internal departments.

4.3 Elephant Flow Management

Large “elephant” flows can saturate bandwidth and cause congestion. Baidu provides TOP IP statistics for precise throttling.

Flow Log can deliver data via BLS or user‑managed Kafka.

Classic case: an e‑commerce customer used TOP IP data to detect and throttle elephant flows, reducing operational cost and improving stability.

4.4 Security Inspection

Real‑time visualization enables precise gateway traffic statistics, allowing detection and filtering of abnormal or dangerous traffic in million‑scale IP networks.

Classic case: a financial customer cleaned up unused IPs and abnormal traffic, strengthening security.

5. Summary

No performance impact: does not affect throughput or latency.

Full‑traffic capture: handles 10 TB‑level dedicated line traffic, giving complete network visibility.

Strong real‑time capability: minute‑level aggregation for rapid analysis and decision making.

Easy management: seconds to enable, replacing self‑built flow statistics, improving operational efficiency.

Flow Log is a powerful cloud monitoring tool that records, visualizes, and retains traffic data, enhancing stability. The product is in public beta; apply via the provided link.