Changing the Engine Mid-Flight: Isolated Verification Environments for Login System Migration

Background

The project required replacing a large-scale login system within two months while normal feature development continued. The login system is a core dependency involving login, token refresh, logout, and integrations with Redis and databases. Verifying failure scenarios (Redis/DB unavailability) in shared environments was impossible because disrupting those dependencies would block hundreds of developers. Additionally, developers lacked familiarity with Kubernetes concepts (Namespace, Deployment, Pod, ConfigMap) and had no permissions to modify shared resources. The pre-production verification cycle took 30–60 minutes per change due to branch merges, approvals, cloud builds, and pipeline deployments.

Tools and Models

AI Assistant : Invokes standardized build workflows via pre-defined Skills, avoiding ad-hoc Docker commands.

Makefile : Encapsulates image compilation, build, and push processes.

Docker : Builds service images.

CSGHub : Hosts images and other build artifacts.

Helm : Manages service deployment configurations.

Kubernetes : Runs isolated verification environments.

小 TKE (Mini TKE) : A team-built lightweight K8s management platform providing a web UI for image selection, workload deployment, Helm chart deployment, ConfigMap management, and multi-cluster access.

Approach: AI Lubricates the Verification Pipeline, Not Replaces It

Initially, the team considered letting AI execute Docker, Helm, and kubectl commands on demand. However, services had different Dockerfiles, build contexts, naming conventions, and some required multi-architecture images. AI-generated commands were unstable, hard to review, and not reusable. Kubernetes operations still required deep knowledge of many concepts. The team instead codified high-frequency, error-prone operations into constrained, reusable workflows.

Give AI a reusable, auditable, well-bounded entry point rather than letting it generate commands from scratch each time.

Step 1: Makefile Standardizes “Code to Image”

Image build and push logic was consolidated into a Makefile with targets like make login, make idsvc, make keycloak-bridge, make keycloak-clusterless. Each target:

Builds the image using a specified Dockerfile and fixed build context.

Generates image tags via a unified rule.

Pushes the image to CSGHub.

Outputs the deployable image address and tag.

Multi-architecture builds are also handled in the Makefile. An AI Skill was written to call these Makefile targets, so AI triggers standard actions and helps analyze build logs on failure, rather than guessing commands. The team deliberately avoided a full CI pipeline to avoid shared build machines, queueing, and maintenance overhead; local Makefile builds were sufficient and matched the development cadence.

However, AI response latency and instability, combined with vague Skill descriptions, caused issues: newcomers accidentally created multiple Namespaces or targeted wrong clusters because they couldn't understand the K8s terminology (Namespace, Deployment, Pod, ConfigMap) that AI exposed. This confirmed that Skills suit well-bounded standard actions, but a simple, intuitive UI is more reliable for high-frequency, deterministic deploy/verify operations.

Step 2: 小 TKE Lowers the K8s Barrier

After image build, deployment to a verification environment was needed. Requiring every developer to master K8s concepts was unrealistic given the two-month schedule. The team built 小 TKE on top of existing Docker, Helm, and K8s assets, using CSGHub and AI to accelerate development.

小 TKE Architecture: Unified Platform Entry, Resources Deployed to Personal DevCloud

小 TKE runs on a shared machine, providing a unified UI and deployment entry point. It does not host the actual business workloads. Instead, it calls the Kubernetes kubectl API of each developer's DevCloud environment to perform controlled queries, deployments, and updates. The resulting Workloads, Pods, Services, ConfigMaps, Redis, DB, etc., reside in each developer's isolated DevCloud machine.

Developer Browser
  ↓
Mini TKE on Shared Machine
  ↓ calls target DevCloud's Kubernetes kubectl API
Developer's Personal DevCloud Machine
  ↓
Personal isolated Namespace / Workload / Pod / Config / Dependencies

This combines a single platform entry with per-developer resource isolation. Developers don't need to write complex YAML or run kubectl commands; they simply select an image tag in the UI and deploy to their own environment.

The UI shows a workload list where developers pick a service and an image tag to update. Sensitive data (namespace, image registry) is masked in screenshots. The tool itself is not shared externally due to internal cluster, registry, and permission boundaries; the article shares the design pattern: combining Dockerfile, Helm, Makefile, and AI Skills to lower the barrier for isolated environment deployment and verification.

For newcomers, the UI is immediately valuable: no need to learn all K8s terms first, just choose environment → deploy image → curl the API → verify. This closes the most basic and critical test loop.

Step 3: Verify Login System Boundaries and Degradation in Isolated Environments

With isolated environments, verification no longer depends on shared environments. Verification scope includes:

Normal login

Token refresh

Logout

Redis unavailable degradation

Database unavailable degradation

Behavior after config changes or image upgrades

Redis and DB fault verification are critical. Previously, these could not run in shared environments because any dependency failure would affect hundreds of developers. Now, developers deploy the service and its dependencies in their own environment, then stop or misconfigure Redis/DB to verify degradation logic.

The verification loop becomes:

Code change
→ AI invokes Makefile via Skill to build and push image
→ Deploy new image to isolated env via 小 TKE
→ Verify normal login, token refresh, logout
→ Simulate Redis or DB unavailability
→ Check login system degradation behavior, logs, API responses
→ Fix, rebuild, redeploy, re-verify

This shifts work that previously required environment coordination, ops support, and shared-resource windows into a developer-controlled verification loop.

Step 4: Token Re-signing Tool for Rapid Auth Debugging

The author owned token functionality, originally backed by Keycloak. During migration, token claims and interface dependencies were undocumented. Starting from a minimal claim set, fields were added incrementally. When an interface returned 401 or 403, root cause could be missing/incorrect claims in token issuance, or the interface's own auth logic, role mapping, or config. The old path:

Modify token issuance code
→ Build and deploy service
→ Re-run login flow
→ Call interface, see 401/403
→ Guess whether token or auth logic is at fault

The author built a Token Re-signing Tool for controlled test environments. It imports an existing token, modifies a specific claim, and re-signs a test token using an authorized test key. The new path:

Import test token
→ Modify only the claim under test
→ Re-sign test token
→ Call target interface directly
→ Determine if issue is in token issuance or interface auth

The tool is not a general-purpose identity platform; it targets the current migration's token debugging needs, turning multi-round guesswork into a single controlled-variable test.

Step 5: Session Decryption Tool to End Guesswork

Session data in cookies is encrypted. Developers troubleshooting login state, session renewal, or permission errors could not inspect session contents, relying on symptoms, logs, and code paths—leading to misdirection. A Session Decryption Tool was built for controlled test environments, using authorized test config to decrypt the session cookie. The old path:

Interface error
→ Guess if session is the problem
→ Repeatedly modify code, redeploy, re-login
→ Observe result

New path:

Import session cookie from controlled test env
→ Decrypt and view session content
→ Compare expected fields, interface response, logs
→ Quickly pinpoint issue in session, token, or server auth logic

Like the token tool, it's a project-specific utility that replaces speculation with observable facts.

Results

1. Previously “Untestable” Failure Scenarios Become Testable

Redis/DB outage scenarios no longer risk shared environments. Developers actively inject dependency failures in isolation, verifying degradation logic before reaching shared or production pipelines.

2. Deployment Prep Time Reduced from 30–60 Minutes to ~5 Minutes

Old path: merge to shared branch → coordinate approvals → wait for cloud build and pipeline → deploy to pre-prod (30–60 min). New path: local Makefile build → push image → deploy via 小 TKE to isolated env (~5 min). Single verification prep time reduced by 83–92%. Developers fit more “modify–build–deploy–verify” cycles into a single session.

3. Ops Freed from Per-Verification Support

Previously, environment deployment, dependency tweaks, and fault simulation required coordinating shared environments, requesting windows, and often ops assistance. Now developers self-serve image deployment, config changes, and dependency fault injection. Ops only engaged for rare cross-region tests, not every build/deploy/verify cycle.

4. K8s Learning Curve Flattened, Verification Path Shortened

Developers no longer need to master image build, push, Helm, K8s resources, and config management. Makefile standardizes build; 小 TKE UI handles deployment. Even K8s-novices follow “select image → configure deploy → view resources → verify”. The team didn't wait for everyone to learn K8s before starting verification.

5. DevCloud Local Network Utilization Reaches 100%

Local build, image push, isolated deployment, and service integration all happen inside the DevCloud network, avoiding waits for external shared build/deploy pipelines. All developers on the migration use this closed loop, turning DevCloud from idle infrastructure into a daily resource for high-frequency verification, parallel debugging, and fault drills.

6. AI Turned from One-off Productivity Boost into Reusable Team Capability

AI's value wasn't just faster platform code; it helped converge repetitive, error-prone build ops into standard entry points. Reusable assets:

Multi-service Makefile for image build/push

Constrained AI Skills

Lightweight K8s management UI

Login system verification workflow for isolated envs

Token re-signing tool for controlled test envs

Session decryption tool for controlled test envs

These assets serve future service migrations, canary verifications, and fault drills.

7. Four Direct Benefits of the Toolset

Onboard newcomers, lower knowledge barrier : Start with basic HTTP verification (choose env → deploy image → curl → observe). Learn Namespace, Workload, Pod gradually while delivering.

Enable extreme scenario verification without affecting shared clusters : Redis/DB failures crafted in personal envs, no impact on others, no ops coordination or special windows needed.

Facilitate multi-party collaboration and cross-platform integration : Unified build assets and lightweight UI let teams compose services, configs, and clusters for joint debugging or verification, reducing cross-team coordination cost.

Local builds are faster, no shared build machines : Makefile builds locally, eliminating queueing, resource contention, and feedback wait. Prep time dropped from 30–60 min to ~5 min (83–92% reduction), enabling immediate “build–deploy–verify” iterations.

8. Token Debugging: From Guessing to Controlled-Variable Validation

Previously, 401 / 403 errors required multi-round “modify token code → build/deploy → re-login → call interface → guess”. The token tool enables changing one claim at a time and directly calling the interface, collapsing the loop into a single controlled-variable test. Teams can now quickly distinguish:

Missing/incorrect claim in token

Interface's own auth, role mapping, or config issue

The tool's value lies in replacing conjecture with observable, reproducible facts, eliminating the most time-consuming debugging segment.

9. Session Debugging: From Speculation to Observation

The session decryption tool gives direct visibility into encrypted session cookies in test environments. Instead of “error → guess session → modify code → redeploy → re-login → guess again”, developers now inspect actual session content first, then decide whether to investigate token, session state, or server auth logic. This cuts ineffective “code–deploy–guess” cycles and grounds root-cause analysis in verifiable data.

Lessons Learned

1. AI's Value Extends Beyond Code Generation

In high-risk system migrations, code generation is only the start. Delivery speed and quality hinge on fast, low-cost verification of real dependencies and edge cases. AI can connect build, deploy, and debug workflows, lubricating the last mile from “code done” to “verified”.

2. Don't Let AI Improvise; Give It Standard Entry Points

For critical ops like image build/push, ad-hoc AI-generated shell commands are unstable. Better: humans codify the flow into Makefiles, scripts, or Skills with clear inputs, outputs, and boundaries; then AI invokes them within constraints. This ensures efficiency, reusability, auditability, and maintainability.

3. Skills ≠ Product UI; Deterministic Ops Should Be Tool-First

AI/Skills excel at open-ended tasks (explaining build failures, drafting solutions, assisting debugging). But for infrastructure operations with clear boundaries (clusters, namespaces, deployments), vague natural-language instructions cause instability and misoperations. Newcomers can't judge which cluster/namespace AI touched. A UI that presents allowed choices, target objects, and immediate feedback lets users complete tasks with basic computer skills:

Select correct environment
→ Select image tag
→ Deploy workload
→ Use curl to call interface
→ Complete verification based on response

Division of labor: Commands (Makefile) standardize build entry points; AI handles explanation and debug assistance; Platform UI owns stable, intuitive, controlled deployment operations.

4. Lightweight Tools Need Not Be Perfect; Value = Shortening the Critical Path

Why not wait for a general platform or adopt mature open-source? With a two-month reverse schedule and parallel feature work, waiting for a generic platform was unrealistic. Generic tools rarely fit the project's specific deployment model, dependency graph, and verification path. The team needed a lightweight capability tailored to the current verification flow: letting K8s-novices deploy, configure, and verify quickly.

小 TKE, token re-signing, and session decryption are not general-purpose products. 小 TKE relies on pre-existing Dockerfile/Helm assets; the other two focus solely on the project's token/session/auth debugging. Yet within the project's few-month window, they solved acute bottlenecks. AI lowered the cost of building such “small, sharp” tools, so the team didn't wait for a perfect universal solution—they unblocked the most critical delivery constraint first.

A good tool doesn't need the most features or broadest applicability; if it reliably shortens the team's current most critical process, it's valuable.

The Makefile, 小 TKE, token re-signing, and session decryption tools all embody the same principle: Good enough—just make verification flow smoothly.

Reusable Assets

Image Build Makefile : Unified encapsulation of Docker build, tag generation, push, and multi-arch build for multiple services.

AI Skills : Constrain AI operations to standard Makefile workflows, reducing uncertainty from ad-hoc command assembly.

小 TKE Platform Design : Provides multi-cluster access, workload management, Helm deployment, image and ConfigMap management, enabling non-K8s-experts to spin up isolated environments. The tool itself isn't shared due to internal security boundaries; the reusable part is the workflow design and asset organization.

Project-Specific Debugging & Verification Micro-tools : Teams can codify recurring, hard-to-observe debugging problems into project-specific utilities. These don't aim for cross-project generality but shorten the debug loop for the current stack, test config, and real issues. Examples:

Token Re-signing Tool : For controlled test envs; adjust claims and re-sign to quickly isolate token issuance vs. interface auth issues.

Session Decryption Tool : For controlled test envs; decrypt session cookies with authorized test config to assist login state, renewal, and auth debugging.

Similar tools can be added continuously, giving each project tailored debugging and verification capabilities.