How UPYUN Scaled Cloud Operations: Automation, Monitoring, and Performance Visualization

Founded in 2005, UPYUN started with servers that required custom kernel and driver compilation on Marvell NICs. Over a decade, its operations transformed dramatically, prompting a systematic review of the company’s operational growth, pitfalls, and current state.

The Art of Operations

Operations are fundamentally about elasticity and scaling from zero to many. Early stages focus on establishing reliable services; later stages emphasize rapid growth through disciplined automation and monitoring.

Key Operational Pillars

UPYUN identifies three core pillars:

Automation & Process Standardization – Use scripts and tools to eliminate manual errors.

Continuous Monitoring – Real‑time alerts and isolation mechanisms, including custom scripts that detect abnormal NIC traffic to pre‑empt DDoS attacks.

Performance Visualization – Generate health reports and dashboards to justify resource allocation and communicate with non‑technical stakeholders.

Deployment Automation

The automation journey progressed through three phases: initial use of awk, sed, and bash; then adoption of Ansible with playbooks; and finally integration of a CMDB with Ansible for systematic releases. Scheduling regular release windows (e.g., testing on Tuesdays, deployment on Wednesdays) helped enforce consistency.

Network upgrades accompanied operational scaling: from early H3C equipment to multi‑layer fiber links, 10 GbE switches, and BFD for inter‑data‑center connectivity. UPYUN now operates multiple data centers with redundant paths, enabling rapid system swaps using USB‑based boot media within minutes.

Monitoring Normalization

Monitoring evolved from manual observation to a tiered system:

~100 servers – visual checks.

100–1,000 servers – Zabbix with first‑person alert scripts.

3,000+ servers – custom aggregation, bandwidth quality graphs, ELK analytics, and a proprietary “Dog Eye” system that color‑codes node health.

Daily checks of the three slowest data centers and ELK‑driven log analysis now allow UPYUN to detect issues before customers notice.

Performance Visualization

UPYUN leverages Nginx + Lua for caching, dual‑stack SSD/ATS clusters with LVS load balancing, and rapid ATS restarts. The team has also contributed to kernel‑level congestion algorithms, migrating from Linux 3.18 to 4.1 and experimenting with hybla to improve CDN latency.

Operational Challenges

Operations staff often act as “firefighters” and “scapegoats,” handling resource requests, frequent OpenStack/Docker migrations, and ensuring high availability through LVS, HAProxy, dual power supplies, multi‑carrier links, and multi‑site redundancy.

Guiding Philosophy

UPYUN’s operational mindset emphasizes:

Machine‑centric load balancing independent of human intervention.

Delegating tasks to empower sub‑teams while continuously learning.

Stateless, scalable services to reduce reliance on stateful code.

Maintaining constant deployment costs despite rapid business growth, thereby increasing operational influence.

Operational Mastery

Continuous skill development, active participation in industry events, and a habit of questioning “why” underpin UPYUN’s operational excellence.

UPYUN, a cloud CDN provider, accelerates kernel upgrades to a three‑month cycle, operates with a team of just over a hundred engineers, and aims to expand its market share while maintaining high‑performance, automated operations.