Turning Online Incidents into Growth: From Firefighting to Real Technical Mastery

Handling Online Incidents

When an online fault occurs, apply a two‑step “firefighting” process:

Stop the bleeding – quickly isolate or mitigate the symptom to restore service.

Root‑cause analysis – investigate why the fault happened and record findings.

Typical root‑cause categories and concrete mitigation actions include:

NullPointerException in Java code – introduce stricter code‑review policies and run static analysis tools such as findbugs or spotbugs.

Incorrect configuration change – enforce a double‑approval workflow (e.g., pull‑request review + change‑approval gate) before applying production config.

Loss of in‑memory state after a restart – schedule periodic persistence jobs (e.g., write‑behind caches, snapshot to Redis or a database) to recover state.

Untested business logic – expand unit‑test coverage, add integration tests that exercise the missing path, and document the test case for future reference.

Document the incident, the mitigation steps, and the root cause in a post‑mortem to prevent recurrence.

What Constitutes Technical Ability

Technical ability is fundamentally the capacity to solve problems, not merely familiarity with frameworks or source code. It can be broken into three maturity levels:

Level 1 – Immediate resolution : Fix the symptom so the system works again.

Level 2 – Elegant, reusable solution : Refactor the fix to be clean, maintainable, and reusable (e.g., applying design patterns, extracting common utilities).

Level 3 – Future‑proof architecture : Design the solution so it remains valid for upcoming requirements (e.g., choosing extensible abstractions, avoiding hard‑coded constants, planning for scalability).

These levels manifest in concrete quality attributes:

Code safety – avoid unchecked exceptions, ensure thread‑safety, prevent memory leaks.

Design elegance – follow the Open‑Closed Principle, keep module boundaries clear, write reusable components.

Architectural foresight – select appropriate technologies, design data models that can evolve, and structure subsystems so they can be reused across projects.

Avoid Reinventing the Wheel

Before building a custom component, answer three questions:

Does an existing open‑source or commercial solution already address the problem?

If it exists, what unique value does your approach provide? Can a small adaptation of the existing tool satisfy the need?

If no solution exists, is the scenario truly novel, or has the problem simply not been explored?

Common, battle‑tested alternatives in the Java ecosystem include:

Database connection pools – dbcp, c3p0, druid Caching – local ehcache, distributed redis (or tail)

RPC frameworks – dubbo, cross‑language thrift Distributed scheduling – schedulex Search – elasticsearch, solr Media services – object storage (e.g., Qiniu), IM SDKs (e.g., RongCloud, Easemob), audio‑video SDKs (e.g., Agora)

If a custom implementation is still considered, evaluate development cost, maintenance burden, and the risk of hidden bugs before proceeding.

Elevating System Perspective

Viewing a component as an isolated “tree” limits understanding. Treat it as part of a “forest” by mapping upstream and downstream dependencies:

Identify which services call the component and which services it calls.

Document data flow, protocol contracts, and failure propagation paths.

Assess how changes in one subsystem affect others, enabling more informed architectural decisions.

Higher‑level thinking helps prioritize work that delivers broader business value rather than merely fixing local bugs.

Effective Project Retrospectives

After a project or major feature is delivered, conduct a structured summary covering:

Depth of business domain knowledge acquired.

Mastery and correct usage of the underlying technical stack.

Design shortcomings and opportunities for refactoring.

Development pitfalls (e.g., missed edge cases, performance regressions).

Project management aspects – timeline adherence, staffing adequacy, coordination efficiency.

Operational readiness – monitoring coverage, on‑call response effectiveness during high‑traffic events.

Recording these insights creates reusable knowledge, reduces repeat mistakes, and accelerates both individual and team growth.