How I Traced a Sudden Data Drop After a Feature Release: 14 Debugging Steps

After a recent feature rollout caused a sharp drop in data volume, I followed a systematic debugging process that can serve as a reference for similar incidents.

1. Verify the problem’s reality

The data team reported a severe drop, and I first compared request volume with actual landed volume to confirm the issue before blaming the code.

2. Review code and compare differences with experienced colleagues

Without clear evidence, I inspected the code, discovered a hidden bug that could cause the drop, and fixed it.

3. Sit with the DBA to monitor data changes

I asked the DBA to track data volume after the release, but the numbers showed little change, indicating the problem was not simply timing.

4. Perform local debugging

Assuming an online issue, I switched to local debugging via VPN, which ran without errors, suggesting the problem lay elsewhere.

5. Compare online and test environment configurations

I examined every configuration difference, even file timestamps, and moved the online setup to the test environment, which ran smoothly, proving configuration was not the cause.

6. Add comprehensive logging and retry online debugging

I enhanced logs to capture full request details, but the new logs still showed intermittent failures, hinting at possible machine‑specific issues.

7. Capture network traffic

Using tcpdump and lsof, I confirmed that client requests reached the server and long‑living connections were healthy, shifting suspicion to the server side.

8. Restart services

Restarting the service and Kafka did not resolve the intermittent failures.

9. Switch asynchronous calls to synchronous

Changing the async request to sync made the system slower but allowed Kafka requests to succeed, indicating the async path might be problematic.

10. Conduct load testing in the test environment

I wrote a shell script to simulate concurrent requests; the test showed no anomalies, suggesting high concurrency was not the root cause.

11. Re‑examine the code thoroughly

Multiple developers reviewed the code again, but no additional issues were found.

12. Issue requests directly via command line

Using both the application code and Kafka’s native client to send requests yielded different results: the native client responded instantly, while the application still failed.

13. Re‑inspect the data

After a period of no progress, the data unexpectedly returned to normal, coinciding with a Kafka restart.

14. Identify the root cause: Kafka partition bottleneck

The final analysis revealed that the topic’s request volume exceeded the capacity of its small number of partitions; increasing the partition count restored normal throughput.