36 Proven Strategies for Scalable and Efficient Big Data Operations

Preface: With the rapid growth of the Internet, big data is being created and collected at an astonishing speed. Companies such as Google and Alibaba recognize data as a strategic resource, leading many to build large‑scale big‑data platforms to extract commercial value. Big‑data operations have emerged in this context, sharing many commonalities with traditional operations while also possessing distinct characteristics.

First characteristic: large scale

In the big‑data domain, a single cluster typically consists of hundreds to tens of thousands of physical machines, and multiple clusters are often deployed across regions for disaster recovery. The large scale makes various anomalies—hardware failures, network issues—common, leading to several requirements:

The architecture must tolerate single‑machine failures and even single‑cluster failures.

An automated operations platform is needed to handle routine tasks such as hardware repair and service deployment, otherwise operational costs become prohibitive.

Deep attention to IDC architecture is required, considering regional distribution, scalability, power supply, inter‑rack bandwidth, and QoS policies.

Operators must also use big‑data analytics to monitor logs and events, discover platform risks, locate problems quickly, improve stability, and manage resources more precisely, enabling reasonable procurement.

Second characteristic: layered

Big‑data platforms essentially provide PaaS services for big data. Numerous applications—offline reports, machine learning, OLAP, real‑time analytics—run on top of the platform. Big‑data operators are generally responsible only for the platform, while business‑level applications have their own operation teams. Therefore, operators need the ability to quickly distinguish platform issues from business issues; mixing the two leads to constant overload.

Big Data Operations – 36 Strategies: Stability

1. Data is the foundation of big data; better to stop service than lose data.

2. Never disable the platform’s recycle‑bin; all deletions must go through it with a silent period before final removal.

3. Critical data must have off‑site disaster recovery; intra‑city backup is insufficient.

4. Encrypt all keys in configurations and focus on platform security.

5. Control services must support data‑center switching to accelerate fault recovery.

6. Large‑scale system releases must use staged gray‑release strategies.

7. Multi‑tenant quota limits and isolation are key to avoid interference.

8. Maintain real‑time TOP‑N resource usage analysis across dimensions.

9. Platform SLA should be transparent to users to prevent doubt.

10. Announce platform issues to users immediately to avoid overwhelming support.

11. Storage bottlenecks include not only capacity but also file count.

12. Offline jobs need baseline‑based critical‑path time‑prediction and early warning.

13. Real‑time platforms are latency‑sensitive; place resources close to data sources.

14. Real‑time processing chains are long and delay‑sensitive; detailed metrics at each stage are essential for troubleshooting.

15. Hotspot machines can slow down real‑time jobs; monitor and mitigate hotspots.

16. Critical real‑time services should have dual‑link disaster recovery for continuous availability.

17. Large‑scale data platforms must tolerate single‑machine failures; otherwise, they should not go live.

18. Platforms need service migration capability because data‑centers eventually run out of space.

19. Shared networks must have QoS isolation to prevent one workload’s traffic from affecting others.

20. Treat the platform as an "electric tiger"—plan rack density and estimate power consumption early.

21. Business planning must consider data‑center layout; otherwise, IDC construction and supply chains suffer.

22. Anticipate sudden user demand spikes to avoid resource shortages.

23. Multi‑master physical distribution must satisfy rack and switch requirements.

Big Data Operations – 36 Strategies: Cost

24. Closely monitor cluster water‑level utilization; optimizing even a single point can save substantial money.

25. Use peak‑off‑peak pricing to guide users toward reasonable task submissions.

26. Build job and storage health analysis models to encourage resource optimization.

27. Run system tasks such as merge and archive during business low‑peak periods.

28. Mixed batch‑stream workloads can save resources, but isolation capability is critical.

29. Separate storage‑compute architecture expands mixed‑workload possibilities and captures hardware benefits quickly.

30. Accurately predict storage needs; combine HDD and SSD to improve shuffle performance.

31. Reserve contingency plans for swapping compute with storage or vice versa to address temporary resource gaps.

32. Large scale and high pressure demand continuous monitoring of hardware and network advancements to capture technology benefits.

33. Anticipate hardware ratio changes; technology cycles outpace machine warranty periods.

Big Data Operations – 36 Strategies: Efficiency

34. Transitioning from small to large scale is a qualitative change; automation is essential.

35. Automation tools are the lifeline but also a risk source; enforce strict testing.

36. Continuously leverage big‑data analytics on operational data; data accumulation and analysis are crucial.