How Pupu Optimized Development and Testing Environments with Kubernetes and Service Mesh

Development & Testing Environment Pain Points

Pupu's microservice stack uses the popular Spring Boot + Dubbo combination. Early on, dozens of services (product, store, account, order, notification, statistics, etc.) were split out, requiring developers to start multiple services together with Zookeeper, MySQL, Redis, Elasticsearch, and other components. Ten services alone consumed about 512 MB each, plus the supporting components, leading to roughly 6 GB of memory usage per developer machine. The need to run several environments simultaneously for front‑end and client integration further increased the overhead.

As business grew, more services were added, causing cross‑service development conflicts and making a single test environment insufficient. Test engineers often had to negotiate shared environments, highlighting the urgent need for a better solution.

First Optimization – Introducing Kubernetes

Only a subset of services is required for a given feature. Pupu proposed letting developers connect to a pre‑started full service cluster and launch only the relevant services locally. Kubernetes ( K8S) was a natural fit because the production environment was already planned to move to K8S for validation and operational experience.

Using K8S namespaces, each feature was given an isolated namespace. Developers could start the needed services and components within that namespace, accessing the cluster via the internal DNS provided by a self‑built DNS server. Access could be through VPN or by editing local /etc/hosts. For client‑side integration, developers used packet‑capture tools to forward traffic to the K8S environment.

This approach delivered a ready‑to‑use environment, freeing developers and testers from manual environment preparation.

Practical Issues and Solution Refinement

During the first rollout, several problems emerged:

Persistent storage for databases: Deploying databases inside K8S caused data loss due to pod drift. An initial NFS mount worked for a few environments but soon became a bottleneck, leading to database latency. The team moved persistent components out of K8S and deployed them separately.

Environment startup speed: Importing a full test dataset caused the first database start‑up to take nearly an hour. The team trimmed the data to essential stores, products, and users, created a baseline database, and set up automatic synchronization of schema changes. Startup time dropped to about ten minutes.

Service startup ordering: Dependencies required sequential starts, which were time‑consuming. By discussing with owners, they introduced parallel start‑up for independent services, reducing overall launch time.

Log collection: The original ELK‑Filebeat setup used NFS‑mounted log directories, which failed under load. Switching to the standard K8S logging model, they deployed Filebeat as a DaemonSet to read container stdout directly, eliminating the NFS bottleneck.

After these refinements, the second‑generation deployment reduced startup times, improved component stability, and added a component‑sharing strategy: non‑isolated services could be shared across environments, while stateful components like MQ and Canal remained isolated.

Resource‑Use Optimization – Embracing Service Mesh

As the number of services grew, K8S resource consumption rose again. To reuse existing resources, the team adopted a Service Mesh, specifically a customized Istio that added Dubbo support. By deploying a full base service set once and using traffic‑splitting ("traffic tinting") with sidecar proxies, they could forward only the necessary traffic for a given feature, avoiding the need to start separate databases when not required.

This mesh‑based approach cut the number of deployed services dramatically, allowing a complete environment to start in seconds and further reducing resource usage.

Maintaining Hundreds of Environments

By the time of writing, Pupu managed over a hundred development and test environments, including pressure‑test and integration environments. Frequent releases and evolving configurations made manual maintenance impossible, so automation became critical.

Configuration changes originate mainly from the configuration center, databases, Elasticsearch indices, and Kafka topics. Developers submit required changes through a standardized form on the operations platform, which then automatically synchronizes configurations to the appropriate components. The process includes review steps, audit rules, and staged synchronization (development → integration → other environments). A set of rules governs when and how configurations are applied, preventing premature updates that could break environments.

Special handling is needed for component addresses that differ across environments; internal DNS works for K8S, while external environments use placeholders that the system replaces with actual values during synchronization.

Review & Summary

Through iterative optimization—first adopting Kubernetes namespaces, then refining storage, startup, and logging, and finally introducing a customized Istio service mesh—Pupu dramatically shortened environment provisioning from hours to minutes, reduced resource consumption, and streamlined configuration management across hundreds of environments. Developers can now launch a complete development environment with a few clicks, testers receive ready‑made environments quickly, and operations staff spend far less time maintaining the sprawling landscape.