What I Learned Building a Hybrid Cloud System: Architecture, Tools, and Best Practices

Preface

In the past two months I was deeply involved in developing a distributed system. Although I never aimed to become an architect, the experience forced me to take responsibility for both design decisions and more than 70% of the coding work.

I realized my strengths lay in software concepts such as design patterns, layered architecture, and API design, but I lacked knowledge in hardware and networking.

Characteristics of different operating systems

Network port management and distribution

Choosing network protocols for monitoring virtual machines

Hardware support for distribution and .NET/C# compatibility

When to use multithreading and how to handle thread exceptions

The importance of a logging system in a distributed environment

When to use relational databases versus NoSQL

Choosing between MSMQ and RabbitMQ

Effective communication with other departments

Scheduling across systems written in different languages

The role of test cases in evolving a large system

How coding standards and system standards aid maintenance and extension

Project Overview

The project, which cannot be described in detail, involved building a coordination system for a large domestic vendor's private cloud and purchased public cloud—a hybrid cloud solution.

We used RESTful APIs extensively, discovering that JSON and REST are universally loved by developers worldwide.

Why Software Configuration Failed After Installation

After configuring open‑source tools, the system would not start because the company’s IT only opened common ports and blocked others. Obtaining additional ports required a formal process, highlighting the need for an external development VM.

Using RabbitMQ

I prefer message queues; RabbitMQ offers a richer protocol than MSMQ and better supports the many open‑source languages we integrate with.

Message queues decouple components, simplify data exchange, and require asynchronous handling of unconsumed messages.

Redis

We employed MySQL, SQL Server, and Redis. Redis stores data that lacks clear relational structure, such as large JSON payloads, and we use it for read‑heavy operations while writes go to MySQL with triggers for replication.

Log System

In a deep, multi‑threaded system, comprehensive logging is essential for diagnosing issues that are hard to trace with traditional debugging.

Component‑Based Development

We adopted a branch‑per‑developer workflow, committing code weekly to reduce merge conflicts in a large, modular codebase.

Test Cases

Initially, test cases were seen as unnecessary, but for this project they became crucial for validating APIs that provide monitoring data (CPU, I/O, network) across multiple servers.

Stable APIs and thorough test cases help isolate logic and coordinate components effectively.

Coding Principles

Weekly code reviews address inconsistencies such as naming, variable placement, and method length. We also separate concerns, ensuring that operations like VM creation and database logging are not mixed within the same method.

Communication

Since the project spans multiple teams, understanding each team’s technology stack and business logic is vital. Coordinating message formats and exchange definitions for RabbitMQ required detailed email discussions to avoid misunderstandings.

Key Takeaways

The experience taught me many lessons about technology selection, architecture, and cross‑team collaboration that will guide future projects.

Technologies used:

Languages: C#, Java, Python

Data stores: MySQL, SQL Server, Redis, XML files

Data exchange: REST, JSON, RabbitMQ

Operating systems: Ubuntu, Windows

VM monitoring: Zabbix

Search: Solr

Concepts: multithreading, layered architecture, modular and component‑based development