BestHub
Sign in
Tagged articles

production incident

12 articles · Page 1 of 1
Top Architect
Top Architect
Oct 13, 2026 · Backend Development

Snowflake ID Collision in Production: Why Custom ID Generators Fail

A production incident caused by a flawed custom Snowflake ID generator with a 31-bit timestamp (24-day cycle), IP-based BusinessId collisions, and unconfigured WorkerId/DataCenterId led to duplicate order IDs; the article analyzes the root causes and recommends mature libraries like Hutool and Baomidou with proper WorkerId assignment strategies.

backend developmentbaomidoudistributed-id-generation
0 likes · 8 min read
Snowflake ID Collision in Production: Why Custom ID Generators Fail
liandk
liandk
Oct 12, 2026 · Backend Development

AI-Generated SQL Locks Database, Exhausts Tomcat Connection Pool: Production Post-Mortem

A production post-mortem reveals how AI-generated SQL caused long-running queries that locked SQL Server, exhausted Tomcat's 300-connection pool, and repeatedly hung the service until restart; the article details root-cause analysis, missing SQL execution timeout, and fixes including connection pool abandonment recovery, query optimization, and monitoring alerts.

AI-generated SQLSQL ServerTomcat connection pool
0 likes · 8 min read
AI-Generated SQL Locks Database, Exhausts Tomcat Connection Pool: Production Post-Mortem
DevOps Coach
DevOps Coach
Jan 22, 2026 · Fundamentals

How a Six-Line C++ Lambda Cost $2.3M and Why Rust Saved the Day

A six‑line C++ lambda caused a $2.3 million loss due to a dangling reference, and after months of debugging the team rewrote the order‑matching engine in Rust, cutting latency by 19 % and eliminating the hidden ‘safety tax’ of memory‑unsafe code.

C++Rustmemory safety
0 likes · 7 min read
How a Six-Line C++ Lambda Cost $2.3M and Why Rust Saved the Day
Lakehouse Research Base
Lakehouse Research Base
Jan 2, 2026 · Databases

StarRocks Production Incident: Continuous Report Refresh Triggers Cluster Crash & Recovery

This article details a StarRocks cluster crash caused by users continuously refreshing reconciliation dashboards, the emergency response including query timeout reduction and compute node scaling, root cause analysis highlighting storage-compute separation benefits, and long-term preventive measures like business reporting processes, Multi-Warehouse isolation, and query governance.

Multi-WarehouseSQL optimizationStarRocks
0 likes · 17 min read
StarRocks Production Incident: Continuous Report Refresh Triggers Cluster Crash & Recovery
Ops Community
Ops Community
Oct 4, 2025 · Databases

How to Quickly Diagnose and Fix a Frozen MySQL in Production: 5 Proven Steps

Facing a MySQL that suddenly becomes unresponsive in production? This article walks through the exact five‑step investigative process—checking process status, examining connections, locating lock waits, analyzing slow queries and system bottlenecks, and applying emergency recovery—illustrated with real‑world examples and command‑line snippets.

Database TroubleshootingMySQLoperations
0 likes · 19 min read
How to Quickly Diagnose and Fix a Frozen MySQL in Production: 5 Proven Steps
Su San Talks Tech
Su San Talks Tech
Nov 25, 2024 · Backend Development

How a Snowflake ID Overflow Crashed Our System and What We Learned

A production outage on 2024‑11‑20 was traced to a Snowflake‑based UID generator whose timestamp bits ran out, prompting a detailed post‑mortem that explains the root cause, bit‑allocation analysis, and the steps taken to fix and prevent future ID overflow issues.

UID generatorbackend developmentdistributed ID
0 likes · 9 min read
How a Snowflake ID Overflow Crashed Our System and What We Learned
ITPUB
ITPUB
Jun 6, 2024 · Databases

Why MySQL CPU Spiked to 400%: A Real‑World SQL Optimization Case Study

A startup’s production MySQL server hit 400% CPU usage due to poorly written queries with temporary tables, filesorts, and join buffers, prompting a deep dive into execution plans, concrete optimization recommendations, and a rollback strategy to restore service.

CPU overloadMySQLproduction incident
0 likes · 8 min read
Why MySQL CPU Spiked to 400%: A Real‑World SQL Optimization Case Study
Architect's Journey
Architect's Journey
Jun 1, 2022 · Operations

How We Resolved a Kafka Consumer Production Outage Step by Step

The article recounts a production incident where a Kafka‑based consumer in a finance microservice hit thread‑pool exhaustion and slow‑query alerts, analyzes the root causes of async processing and bulk message bursts, and outlines a three‑phase remediation that includes data repair, switching to synchronous consumption, and request‑level batching to prevent future failures.

JavaKafkaMessage Queue
0 likes · 6 min read
How We Resolved a Kafka Consumer Production Outage Step by Step
ITPUB
ITPUB
Sep 8, 2020 · Databases

Why a Missing Index Parameter Crashed Our Production MySQL Database

A production MySQL server was overwhelmed by a high‑volume query that ignored a crucial composite index because the leftmost key was missing, leading to full‑table scans, CPU saturation, and a half‑hour outage, and the post explains how the issue was diagnosed and fixed.

Java ValidationLeftmost MatchingMySQL
0 likes · 9 min read
Why a Missing Index Parameter Crashed Our Production MySQL Database
dbaplus Community
dbaplus Community
Aug 11, 2020 · Operations

7 Real-World Production Failures and Fast Diagnosis Techniques

The article shares seven authentic production incident cases—from JVM Full GC spikes and memory leaks to cache avalanches, disk I/O blocks, database deadlocks, DNS hijacking, and bandwidth exhaustion—detailing root causes, step‑by‑step troubleshooting methods, code snippets, and practical mitigation strategies for engineers.

DNS hijackingDatabase DeadlockJVM
0 likes · 17 min read
7 Real-World Production Failures and Fast Diagnosis Techniques