Operations 4 min read

Why GitHub’s 8‑Hour Outage Stemmed from Auto‑Scaling Failure and a VS Code Retry Storm

GitHub’s nearly eight‑hour outage was caused by a saturated load balancer in its US‑central data center, a mis‑configured auto‑scaling policy that ignored Istio sidecar concurrency limits, and an optimistic retry loop in VS Code that amplified traffic tenfold, prompting emergency mitigations and a staggered recovery.

21CTO
21CTO
21CTO
Why GitHub’s 8‑Hour Outage Stemmed from Auto‑Scaling Failure and a VS Code Retry Storm

GitHub reported a service interruption that began on 17 August at 13:28 UTC and was fully resolved at 21:15 UTC, lasting 7 hours 47 minutes and affecting Issues, Pull Requests, the API, Actions, and Copilot with massive errors.

The immediate cause was network saturation of the load balancer serving the US‑central data center, triggered when the Istio sidecar reached its concurrency limit.

Although the auto‑scaling system should have added capacity once limits were hit, a configuration error caused the policy to monitor only host services while ignoring the auxiliary service’s concurrency constraints, leading to a cascading failure.

GitHub identified that optimistic retry logic in Visual Studio Code intensified the problem by overloading the internal load balancer; engineers temporarily reduced gateway retry attempts and reconfigured the load balancer to reject inbound Copilot Token Service requests that returned HTTP 403, which helped alleviate the overload.

Most services recovered at 16:36 UTC, general operations at 18:03 UTC, and the Copilot Token service was finally restored at 21:02 UTC. GitHub also noted that repeated network attacks on code‑loading nodes complicated the recovery.

Microsoft said it will correct the auto‑scaling policy, review retry limits, audit Istio concurrency settings, and fix the VS Code behavior that “exacerbated Copilot token traffic.”

Industry observers, such as CloudBees CEO Moritz Plassnig, highlighted that competitors like Cursor, OpenAI and smaller startups are building competitive solutions, suggesting a more fragmented ecosystem and indicating that developers may seek alternatives to GitHub after this reliability incident.

Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Sign in to view source
Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactadmin@besthub.devand we will review it promptly.

load balancingIstioGitHubauto-scalingVS Codeincident analysisoutage
21CTO
Written by

21CTO

21CTO (21CTO.com) offers developers community, training, and services, making it your go‑to learning and service platform.

0 followers
Reader feedback

How this landed with the community

Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.