What Real-World Performance Tuning Taught Us About Legacy Web Apps

Overview

Our company operates 15 web applications that deliver data‑driven services for real‑time decision making. The main legacy system consists of many services older than 15 years, many of which have been refactored several times, and the original developers have often left.

Incident and Diagnosis

During a traffic surge, users complained about severe slowness. Monitoring showed that 90 % of response time was spent acquiring a DB connection. Further investigation revealed that every pod exhausted the connection pool because the liveness probe performed a simple DB heartbeat without releasing the connection. Adding a release call to the probe instantly stabilized performance.

Failed Load Test

We had run a load test the day before, which incorrectly indicated the system was within normal limits, misleading us into thinking no issue existed. This highlighted the importance of realistic testing.

Key Takeaways

Takeaway 1: Do not rely on average latency; examine tail‑latency percentiles. Average wait time stayed flat because many fast requests pulled the mean down. Use 50 %, 90 %, 95 %, 99 % latency metrics to spot outliers.

Takeaway 2: Invest time, tools, and people in performance optimization.

Load testing and realistic load scenarios.

Application Performance Monitoring (APM) tools such as Dynatrace, AppDynamics, or Epsagon.

Effective logging that is clear and useful.

Log aggregation and analysis platforms like ELK, Grafana, or Splunk.

Professional staff (e.g., an SRE team) to operate and interpret the above.

Takeaway 3: Legacy systems will die unless they are actively maintained. Without ongoing development, knowledge of the old code erodes, increasing MTTR when incidents occur.

Takeaway 4: Every line of code matters. A single forgotten DB‑release call can degrade user experience dramatically.

Recommendations

Run load tests for every PR or release in CI/CD pipelines.

When performance issues appear, scrutinize every line of code.

Continuously invest in understanding and improving the legacy system.

Conclusion

The article shares the full set of lessons learned from our performance‑tuning journey, emphasizing that application performance should be the top priority, outweighing UI polish or flashy features.