What I Learned Building a Hybrid Cloud System: Restful APIs, RabbitMQ, Redis & More

Preface

During the past two months I was deeply involved in developing a distributed system. Although I never aimed to become an architect, the experience forced me to take responsibility for more than 70% of the coding and gave me a voice in architectural decisions.

I realized my knowledge of software design patterns, layered architecture, and API design was solid, but I lacked experience in hardware and networking.

Characteristics of different operating systems.

Network port management and distribution.

Which network protocols help us work better, and whether to add agents on VMs or control via protocols.

Whether hardware supports distribution and .NET/C# compatibility during scaling.

When to use multithreading and how to control and capture thread exceptions.

Importance of a logging system for a distributed system.

When to use relational databases versus NoSQL.

Choosing between MSMQ and RabbitMQ for message queues.

Effective communication with colleagues from other departments.

How to schedule systems developed in different languages.

Importance of test cases for large systems from fragmented to complete.

System standards and coding principles for later maintenance and expansion.

Project Overview

The project, which cannot be described in detail, involved building a hybrid‑cloud system for a large domestic vendor, integrating their private cloud with a public cloud.

Using Restful

The biggest gain was realizing that RESTful APIs are not only the way our system interacts with external services, but also the basis of many open‑source tools. This shows how much developers worldwide love JSON and REST.

Why the Software Couldn’t Run After Configuration

After adopting open‑source tools, the application failed to start because the company’s IT only opened common ports and blocked the rest. Obtaining additional ports required a formal process, highlighting the need for an external development VM.

Using RabbitMQ

I prefer MQ for decoupling services. RabbitMQ’s protocol is more advanced than MSMQ and enjoys broader language support, which fits our need to integrate many open‑source components.

Message queues enable asynchronous processing of messages that would otherwise be missed by consumers.

Redis

We used MySQL, SQL Server, and Redis. Redis handled data that lacked clear relationships, allowing us to store large JSON payloads and later deserialize them. Reads were directed to Redis, writes to MySQL, with MySQL triggers handling master‑slave replication.

Logging System

In a deep, multi‑threaded system, logs become essential for pinpointing issues quickly, as debugging the whole stack is impractical.

Component‑Based Development

When a system grows large, modular development and separate branches for each team reduce merge conflicts and improve productivity.

Test Cases

Test cases are crucial for large systems to ensure API stability and to verify monitoring data collection across multiple servers.

Coding Principles

Weekly code reviews help enforce consistent naming, variable declaration, and method length. Responsibilities should be clearly separated; for example, creating a VM and logging to the database must not be mixed in the same method.

Communication

Coordinating multiple teams requires understanding each other’s tech stacks, defining message formats (exchange, queue name, routing key), and confirming details via email to avoid miscommunication.

Technology Stack Used

Development languages: C#, Java, Python. Data storage: cache, XML files, MS SQL, MySQL, Redis. Data exchange: REST, JSON, RabbitMQ. Operating systems: Ubuntu, Windows. VM monitoring: Zabbix. Search: Solr. Features: multithreading, multi‑layer architecture, modular and component‑based development.