Guidelines for Evolving System Architecture and PaaS Operations

Overall Thinking – When evolving architecture, aim for rapid development (a testable hello‑world in five minutes), natural scalability, automation (auto‑provisioning resources and code generation), framework‑level services (logging, counters, tracing), quantitative metrics (percentiles, RPS), homogeneity of language, hardware virtualization, and versioned, gradual roll‑outs.

User – In mobile internet environments, distinguish users on good versus poor networks and provide reliable links and DNS for the latter.

Access Layer – Provide a client‑server suite requiring a team skilled in both mobile (Android, iOS) and server development, so that client developers need not handle network protocols.

Business Access Layer – Enable flexible traffic routing, typically using LVS, F5, or advanced traffic analysis devices for burst handling.

Business Layer – Implement business logic within a unified framework; prefer a single language (e.g., Java) to avoid duplicated foundations across heterogeneous modules, which can cause incompatibilities in RPC, configuration, monitoring, and tracing.

RPC Framework and Queues – Use a unified RPC framework for synchronous communication and message queues for asynchronous data transfer between data centers.

Configuration Management – Adopt Zookeeper as the de‑facto solution for service configuration at scale.

Logging System – Deploy a unified logging solution that is fast, tolerant of network errors, easy to deploy, stable, and horizontally scalable; candidates include Scribe and Kafka, with logs eventually stored in HDFS or HBase for Hive queries.

Monitoring and Alerting – Utilize open‑source tools such as Ganglia and Nagios, and ensure that performance counters defined in the business framework are automatically monitored and alerted.

Tracing System – Implement a tracing tool (e.g., Twitter’s Zipkin) to quickly debug bugs in a micro‑service environment.

PAAS Agent Daemon – A unified operations client that reports hardware/network metrics, starts/stops applications, and communicates status to monitoring and PAAS platforms.

Storage Platform – Include stateful services such as a memcached cluster with consistent‑hash client, a DBMS cluster with sharding and failover (e.g., MySQL‑proxy, Vitess), and an HBase cluster for high‑availability storage.

PAAS Resource Control Layer – Provide real‑time feedback on memory and CPU across IDC(s); address virtualization challenges using Docker, Warden, LXC, and control resources via cgroups and AUFS.

PAAS UI Layer – Offer both web and CLI interfaces for operations and developers to deploy services and manage machines.

Automatic Deployment Layer – Rely on CI tools like Hudson for automated builds, coupled with reliable test frameworks to avoid deployment failures.

Test Framework – Leverage advanced testing libraries such as JMockit and Spring‑Test to reduce boilerplate code.

Build Tools – Use Maven for Java builds and Nexus as a binary repository.

Code Generation – Automate repetitive code via Maven plugins or similar generators, continuously enhancing the tool to cover more scenarios.

Code Quality – Perform static analysis (e.g., Sonar) as part of regular CI pipelines.

Code and Common Systems – Manage source with GitLab, conduct code reviews via merge requests or ReviewBoard, maintain knowledge in MediaWiki, track issues in Jira, and record post‑mortems for incident analysis.

PAAS for DEV & TEST – Provide CLI tools for developers to spin up personal environments and ensure that test environments are highly available and stable, avoiding reliance on development setups for production testing.

Source: Translated from 五四陈科学院 (http://www.54chen.com).