Root Cause Analysis and Mitigation of Sysmon High CPU and Memory Usage

Background

Recent data‑leak incidents and ransomware attacks have highlighted the importance of security monitoring, and Sysmon (a Sysinternals monitoring tool) is widely used for this purpose. However, some deployments experience high CPU and memory usage, where Sysmon occupies a full logical core and its memory consumption continuously rises.

Initial Observation

After evaluating several log‑collection tools, the project chose Sysmon. In production, Sysmon sometimes shows high CPU (e.g., 25% on a dual‑core, four‑thread machine) and steadily increasing memory usage, but the problem appears only on certain machines and seems random.

Problem Localization and Analysis

CPU analysis focuses on threads, while memory analysis looks at handles and other metrics. ProcessExplorer screenshots reveal many short‑lived threads and a sudden surge in handle count, especially Event handles.

Thread Activity

ProcessExplorer shows numerous thread creations and destructions; CPU time is largely spent on these operations.

Handle Explosion

Handle count spikes to over 70,000 within minutes, with Event handles accounting for 99% of the total, indicating that many Event objects are created but not released.

Debugger Verification

Using a debugger, the team prints stack traces for each thread. The command used is: bu ~*kb Stack traces point to the address sysmon+0x494a8, and further back‑tracing identifies the problematic function at sysmon+0x1820e.

Confirming the Issues

Breakpoints are set on both the Sysmon recursive function ( sysmon+0x1820e) and the system CreateEventW API ( bu kernel32!CreateEventW). The debugger shows alternating hits, confirming that each thread creation leads to an Event creation.

Root Cause

The recursive logic resides in sysmon+0x180E0, where a function sub_140018360 creates a new thread that eventually calls back into sub_1400180e0, which invokes OpenTraceW and creates many Event objects. This recursion causes unbounded thread creation and Event allocation.

Reproduction Steps

The issue can be reproduced on any machine by:

Installing Sysmon ( sysmon -i) and stopping its service.

Running the following xperf commands:

Restarting the Sysmon service, which then consumes a full CPU core and gradually increases memory usage.

Stopping xperf again returns Sysmon to normal usage.

Why Sysmon Restarts

Investigation shows that the QQ Browser background service TsService.exe loads PerfTool.dll, which calls OpenServiceW on Sysmon, causing it to stop and restart.

Partial Solutions

Solution 1: The recursive logic is gated by a registry value

HKLM\System\CurrentControlSet\Services\SysmonDrv\Parameters\Options

. If the value AND 1 equals 0 (i.e., the value is even), the recursion is disabled. Temporarily setting this key to 0 stops the high‑CPU/memory behavior.

Solution 2: Since TsService.exe hard‑codes the Sysmon service name, installing Sysmon under a different service name prevents the unwanted stop.

Summary

Monitoring tools like Sysmon are essential for operations, but their stability can be compromised by complex runtime environments. Understanding the recursive thread‑creation logic, the role of Event handles, and the interaction with other services enables operators to diagnose and mitigate performance issues effectively.