How to Build, Test, and Scale Microservices – From Service Discovery to Chaos Engineering

Outline

Build a microservice application using Go and the github.com/koding/kite framework

Service registration and discovery mechanisms

Testing strategies for microservices

Typical microservice design patterns

Practical implementation at Qiniu

1. Building a Microservice Application

A minimal Go service can be created with the Kite framework. The steps are:

Import the Kite package.

Define a service name (e.g., first).

Register an operation, such as square, that returns the square of an input integer.

Start the service, which registers its address and protocol with the framework.

import "github.com/koding/kite"

type Service struct{}

func (s *Service) Square(req struct{ X int }) (int, error) {
    return req.X * req.X, nil
}

func main() {
    k := kite.New("first")
    k.HandleFunc("square", (&Service{}).Square)
    k.Run()
}

2. Service Registration and Discovery

Microservices must announce their presence to a registration center so that callers can locate them. Two common patterns are:

Client‑side discovery: The client queries the registration center for all instances of a service (e.g., name first), selects one (often with load‑balancing logic), and calls it directly. This keeps the registration center out of the request path but requires language‑specific client code.

Server‑side discovery (router): Service instances register themselves, and a routing layer or load balancer proxies client requests to the appropriate backend instance. This centralises routing at the cost of an extra hop.

Both approaches handle dynamic scaling (instances registering and deregistering) and avoid calling stale endpoints.

3. Microservice Testing Pyramid

A testing pyramid for distributed systems consists of:

Unit tests: Test the smallest code units without network calls.

Integration tests: Verify interactions between services or between a service and its database.

Component tests: Isolate a single microservice, mocking external dependencies.

Contract tests: Ensure that a service’s input‑output contract holds, including response latency.

End‑to‑end tests: Validate the complete user journey across UI and backend; run less frequently due to cost.

4. Chaos (Monkey) Testing

Inspired by Netflix’s Chaos Monkey, three failure‑injection tools are commonly used:

Chaos Monkey – randomly kills service instances.

Latency Monkey – injects artificial latency.

Chaos Gorilla – simulates an entire availability‑zone outage.

These tests verify that the system remains resilient when components fail.

5. Traffic Replay

Realistic load for end‑to‑end tests can be generated by capturing live traffic and replaying it in a test environment. The open‑source Go tool Gor (https://github.com/buger/gor) records HTTP traffic and can replay it against a test deployment.

6. Typical Microservice Design Patterns

API Gateway: A single entry point that routes requests to backend services, performs authentication, and aggregates APIs.

Asynchronous communication: Message queues decouple services and handle long‑running tasks (e.g., video transcoding).

Three‑tier service deployment: Each microservice runs its own process, a cache layer (often in‑memory), and a persistence layer (database).

High‑availability components such as the gateway, queues, caches, and databases should be deployed across multiple zones.

7. Qiniu’s Microservice Practice

Qiniu’s data‑processing platform processes billions of requests daily (image resizing, video transcoding). Architecture highlights:

All requests enter through a unified gateway and load balancer.

Backend processing instances are single‑purpose services; each instance runs one service type.

An agent reads/writes storage on behalf of the service.

State is kept in an in‑memory cache; persistence is performed by the agent after processing.

8. Selected Q&A (Technical Highlights)

Registration & discovery solutions: Zookeeper and etcd are widely used.

Container monitoring: Google’s cAdvisor is a common choice.

Stubbing tools for testing: mbtest (http://www.mbtest.org/).

Distributed locks: Possible but often conflict with microservice principles; asynchronous messaging is preferred.

Deployment strategy: Qiniu prefers gray‑release (canary) deployments.

Service bus vs. API Gateway vs. OpenAPI: Service bus handles data transport, API Gateway handles request routing, authentication, and aggregation, while OpenAPI describes the contract for external developers.

Docker networking: Use Docker’s native networking features or Kubernetes for dynamic scaling.

Event‑sourcing for data consistency: Publish data changes as events and let other services react (see http://martinfowler.com/eaaDev/EventSourcing.html).

Service granularity: Align with the smallest business unit; keep the owning team size manageable.