How Panda Live’s Rancho System Automates Secure, Scalable Deployments

Background

With the rapid growth of Panda Live, the number of business requirements, version updates, and release cycles increased, exposing four main pain points: a growing number of project releases, long and manual release cycles, efficiency and risk issues such as smooth roll‑out and load‑balancer switching, and lack of detailed audit functionality.

The team decided to build a unified release platform, Rancho, to standardize, automate, and streamline the release process.

Key Objectives

Unified project onboarding – Operations only need to add a new project and assign release permissions; developers can then release independently.

One‑click automated release – Developers release at any time within their permission scope, while the system records all actions and results.

Summary: Enable self‑service operations for developers.

Security Layers

User Login (First Wall)

Rancho integrates with an internal SSO system and a whitelist strategy, focusing on security, session management, and dual‑verification to ensure only authorized users can log in.

User Permissions (Second Wall)

Two roles are defined:

Administrator – Full super‑permissions for all projects and users.

Regular user – Can view and release only the projects granted by operations, without edit rights.

Project Permissions (Third Wall)

A “project group” bridges users and projects, simplifying permission management and avoiding complex many‑to‑many updates.

Project groups reduce administrative overhead when many users share a project. The system enforces isolation of permissions per project group.

Frontend Release Interface

The frontend allows developers to select release tags, environments, and target machines. It automatically fetches environment/host lists for the selected project.

Release Page

Supports single, multiple, and all‑machine selection, as well as temporary host addition.

Tag Types

Development tag (testable)

Online tag (gray/production)

Other tags (custom input when the desired tag is not listed)

Auto‑updated list of the latest ten Git tags for online/development environments.

Tag Version Selection

Tag Comparison

Shows differences between the selected tag, the main branch, and the last released tag.

Environment/Machine Selection

Three naming schemes are used: based on service function, based on environment, or a combination of both.

Service‑based example: crontab, sdk, console, front.

Environment‑based example: online, beta, gray.

Combined example: sdk_beta, beta_console.

Release Confirmation Checklist

Before publishing, users confirm project, tag, environment, restart option, and selected machines.

Backend Release Management

Administrators manage projects, define Git clone URLs, assign project groups, and add remarks. The backend handles success/failure notifications and delegates most logic to Rancho.

Release Process & Results

The system supports three scenarios: load‑balancer (4/7 layer) strategy, no load‑balancer, and optional service restart. It records detailed audit data for each step.

Project Lock

Prevents concurrent releases on the same project/environment to avoid conflicts.

Release History

Shows status (in progress, failed, succeeded) for each release.

Manual Termination

Operators can abort a release if issues arise; the system logs user, time, project, and reason.

Success & Failure Visuals

Success view:

Failure view:

Wiki Documentation

A standardized wiki is maintained for each feature, architecture diagram, and operational procedure.

Conclusion

Rancho has been in production for nearly five months, supporting project creation, strict login isolation, project locking, front‑ and back‑end management, load‑balancer strategies, smooth host binding, multi‑language parallel releases (Golang, PHP, Node.js, C++, Lua, Python, Shell), asynchronous task handling, real‑time result visualization, multi‑environment and multi‑data‑center releases, manual/automatic termination, and comprehensive audit logs.

Future enhancements include integration with an alert platform, scheduled releases, and automatic error‑fixing based on collected failure patterns.