Smooth Upgrade Strategies for Cloud Native Services to Prevent 5xx Errors

Frequent business R&D iterations often cause a small number of 5xx requests during releases, leading to business loss; smooth upgrade techniques can avoid these errors and achieve lossless deployments.

The article first explains why 5xx errors appear in a typical KVM‑style upgrade where services are stopped before pending requests finish and new services receive traffic before they are ready.

Two main scenarios cause 5xx: the old service is terminated while still handling requests, and the new service receives requests before it is fully initialized.

Avoiding 5xx requires coordination with the traffic entry point: remove traffic from a pod before it begins shutdown, let the pod finish processing, then restore traffic after the new version is ready.

In the future cloud environment, the pod lifecycle is described, including traffic allocation/removal based on readiness probes, and the detailed termination sequence (deleting the pod, entering Terminating, preStop execution, SIGTERM, SIGKILL, and final deletion).

For web services, two approaches are recommended: configure a preStop script that sleeps for the estimated request‑processing time, or have the application listen for SIGTERM and gracefully finish processing before exiting; both require the shutdown window to be within the default 30‑second grace period.

Readiness probes should be used to ensure a new service is only added to the service endpoints after it can successfully handle traffic, covering all critical dependencies while allowing optional components to be excluded.

For microservices that register with external service‑discovery systems, the service itself must deregister before shutdown and handle SIGTERM; the container’s entrypoint should use exec to replace the shell so the main process receives termination signals.

The article concludes that by linking service state with traffic routing, using readiness probes, preStop scripts, and proper entrypoint handling, smooth upgrades can be achieved across different service types in the cloud native environment.