How We Scaled Multi‑Cluster Management with KubeSphere and QKE

Background

QingJiao Classroom is a teaching platform launched by HongYa Technology that provides integrated theory‑practice training for IT majors. The project evolved from a monolithic application to a main site plus a resource‑scheduling service with multi‑availability‑zone and mixed‑customer deployment.

Initially, the team used a visual PaaS platform on a single Beijing cluster, which lowered the barrier to Kubernetes but later caused cumbersome application definition and configuration changes.

Now the project fully uses Kubernetes in development and production, managed by KubeSphere, with weekly releases and dozens of daily deployments triggered by automated CD.

Why Use KubeSphere (QKE)?

KubeSphere (QKE) was chosen to provide a centralized control plane for multiple clusters, simplifying management and reducing vendor lock‑in.

Selection Criteria

Avoid vendor lock‑in

Open‑source solution

Acceptable commercial subscription for support

Low deployment difficulty

Unified authentication

Ease of operation

The team also preferred third‑party managed services for Git, CI, artifact storage, and deployment, using Alibaba Cloud ACK for production and a self‑built K8s for development.

Platform Solution

The chosen architecture deploys KubeSphere (QKE) on QingCloud as the host cluster, with a self‑built K8s development cluster and an Alibaba Cloud ACK production cluster as member clusters, all managed centrally by KubeSphere.

Caption: QingJiao Classroom uses KubeSphere + QKE for easy multi‑cluster management.

Application Deployment Attempts

After selecting the platform, the team migrated services to the new clusters. The original visual PaaS required repetitive manual operations across regions, leading to errors.

Application Characteristics

Deploy services across multiple regions (Beijing, Guangdong, Shanghai).

Replicas operate independently without inter‑communication.

Provide an API server for the main business to schedule.

Manual updates across regions become cumbersome as cluster count grows, increasing the risk of inconsistencies.

Building Native K8s Applications

Using KubeSphere’s console wizard, the team generated Deployment, StatefulSet, Service, and Ingress resources, then switched to edit mode to obtain complete YAML files.

Deployment Tool Selection – Why Helm?

Helm, a widely adopted Kubernetes package manager, simplifies multi‑replica deployments and configuration differences, and KubeSphere’s app store is based on Helm charts.

Choosing a CD Tool

Because KubeSphere’s multi‑cluster application feature only supports custom apps, the team evaluated alternatives and settled on Spinnaker for continuous delivery.

Automation of Application Initialization

DNS automation: A custom tool reads Ingress and LoadBalancer info to automatically configure DNS records.

Database migration and init scripts: Encapsulated as Kubernetes Job resources executed during deployment.

Custom Monitoring

KubeSphere’s custom monitoring consolidates metrics that were previously scattered across Alibaba Cloud, allowing teams to view service health from a single pane.

The member clusters include the Prometheus operator; creating a ServiceMonitor adds monitoring items, and the definitions are packaged into Helm charts.

Unified Authentication

Without an internal LDAP, the team used DingTalk as a single sign‑on source and integrated Alibaba Cloud IDaaS as an OAuth provider, contributing a KubeSphere IDaaS plugin (PR #2997).

Full Migration to KubeSphere

Following the successful migration of the Region service, the team migrated the entire development, testing, and production environments to KubeSphere, using Helm packages and CD pipelines to create isolated test environments on demand.

Key steps for production migration included inventorying databases, co‑existing async queues, migrating scheduled jobs, and gradually shifting traffic via DNS.

Elastic Scaling with Keda

Native HPA based on CPU/memory cannot quickly react to business load changes. Keda enables scaling based on Prometheus metrics, Redis list length, and MySQL queries, allowing proactive scaling for high‑traffic classroom sessions.

Future Outlook

Further ChatOps integration aims to automate regression testing and streamline release management. The team plans to improve gray‑release capabilities beyond the current single‑rule Nginx Ingress limitation.