Mastering Technical Problem Analysis: Practical Strategies for IT Professionals

Why Problem Analysis Matters

Effective software development increasingly depends on the ability to diagnose and solve technical problems rather than merely delivering new features. Without solid analysis skills engineers waste time on repetitive tasks and cannot translate complex business issues into workable technical solutions.

Key Points of Technical Problem Solving

Practical experience outweighs theory – personal incident logs and past fixes are irreplaceable.

Rapid hypothesis generation and validation shorten the path to a solution.

Accurate problem‑boundary definition is essential for targeted fixes.

Systematic post‑mortems turn each incident into reusable knowledge.

Problem Localization Techniques

Replace method : swap component A with a known‑good variant A1; if the issue disappears, the fault lies in A.

Breakpoint method : insert monitoring or logging between A and B to verify A’s output.

Hypothesis method : adjust parameters of the suspected stage and observe the effect.

Binary‑search (divide‑and‑conquer) is often the most efficient way to narrow the search space.

Case Study 1 – Too Many Open Files

A production server reported IO Exception: too many open files and socket timeouts. Initial CPU, memory, and connection‑pool checks were normal.

Checked JVM health with jstat; no anomalies.

Verified database connection pool and thread pool capacity; both were sufficient.

Analyzed error logs and identified two concurrent errors: file‑handle exhaustion and service timeout.

Reviewed recent code changes; no obvious leaks.

Collected lsof output hourly; observed a steady increase of ~60 identical file handles.

Matched inode numbers from lsof with filesystem listings to locate the exact files.

Source review revealed a SAXReader that opened files without guaranteed closure.

Modified the code to close streams in a finally block, redeployed, and the “too many files” error disappeared.

Case Study 2 – Service Call Timeout

OSB services intermittently timed out after 1500 seconds, exceeding the configured 600 second read timeout and 30 second connection timeout.

Confirmed OSB configuration: max retry = 0, read timeout = 600 s, connection timeout = 30 s.

Log analysis showed a thread stuck for 610 s, then a 60 s retry, followed by a connection reset after ~900 s.

Identified a 900 s pool‑shrink interval in WebLogic that finally closed the hanging connection.

Checked external F5 load balancers; idle timeout defaulted to 300 s, contributing to the extended wait.

Bypassing the load balancers and invoking the service directly with SOAPUI, responses were successful, confirming the timeout originated in the load‑balancer chain.

Resolution: increased the load‑balancer idle timeout beyond the OSB read timeout and disabled unnecessary pool‑shrink checks.

Case Study 3 – JVM Memory Overflow

Frequent OutOfMemoryError occurrences in an Oracle SOA Suite 12c environment could not be reproduced in a test lab.

Collected GC logs and heap dumps, then used standard JVM memory‑analysis tools (e.g., jmap, jhat) to identify hot‑spot classes and memory‑leaking code paths.

Followed the methodology described in “从表象到根源‑JVM内存溢出问题分析” to trace the root cause.

Adjusted GC parameters (e.g., -XX:+UseG1GC, heap size tuning) and removed the identified leaks, eliminating the OutOfMemory errors.

Practical Recommendations

Continuously accumulate hands‑on experience and document each incident.

When a new problem appears, quickly define its boundary, generate plausible hypotheses, and validate them with minimal changes.

Leverage search engines and official knowledge bases, but be prepared to craft custom hypotheses when exact matches are unavailable.

Coordinate early with all stakeholders (service owners, middleware teams, network and load‑balancer administrators) to avoid isolated troubleshooting dead‑ends.

Perform thorough post‑mortems to capture lessons learned and enrich the organization’s knowledge repository.