Why Ops Tools Are Far More Complex Than You Think

Before moving from R&D to operations, I always thought operational tools were relatively simple. After leading an operations‑tool team for more than a year, my perception was completely overturned, and I realized that building high‑quality ops tools requires extremely high technical standards.

Initially, I considered ops tools simple because, from the perspective of an online business system, they have low traffic and small data volumes, showing no obvious technical challenges.

After a year of leading an ops‑tool team, I saw that the technical requirements of ops‑tool systems are essentially the same as those of online business systems, just from a different angle. The main responsibilities of an ops‑tool system are:

Automation of operational tasks;

Execution of emergency actions when online failures occur.

These two responsibilities dictate that, besides functional implementation, we must consider several non‑functional characteristics.

1. Automation of operational tasks

True automation hinges on a core metric: success rate. Imagine an automated operation that only succeeds 60% of the time—users would experience four failures out of ten attempts and would likely abandon the tool in favor of manual processes. Therefore, ops tools must ensure a very high success rate, unlike many online services where a lower per‑operation success rate is acceptable. Online services also adopt a fail‑fast strategy to guarantee response times, which is not suitable for ops tools.

A complex operation, such as scaling an application, resembles an online transaction: it involves multiple systems and intricate business logic, constituting a massive distributed operation. To guarantee success, the system must decide how to handle exceptions when Service A calls Service B—whether to retry, skip, or perform asynchronous follow‑up actions. Consequently, ops‑tool systems need clear, robust exception‑handling strategies to maximize success rates.

Thus, when designing ops‑tool systems, we must prioritize ensuring the success rate of each operation and define precise handling strategies for every possible failure, which differs fundamentally from typical online service design.

2. Execution of emergency actions during incidents

During an online failure, teams heavily rely on ops tools for monitoring, releases, traffic switching, and other rescue actions. If the ops tool itself fails at that critical moment, the result is disastrous. Early discussions on rescue mechanisms considered several sophisticated solutions, but we ultimately chose a very simple approach: avoid any dependencies whenever possible because rescue actions must be absolutely reliable.

According to this requirement, any rescue operation in an ops‑tool system must guarantee absolute stability, regardless of whether the incident is minor or a full‑scale data‑center outage.

Although a rescue operation may appear as a simple button, it often triggers coordination across dozens of subsystems. Ensuring that this seemingly simple button is completely trustworthy involves a great deal of underlying work.

From the analysis of responsibilities, it is clear that ops‑tool systems face high technical demands: they must ensure high success rates and absolute stability for rescue‑type actions.

Regarding scale, many smaller business scenarios rarely encounter massive release volumes, but in our experience we have repeatedly faced situations where the release system itself could not handle the load, forcing us to devise temporary solutions. Therefore, ops‑tool systems must understand their current capacity limits and know how to scale horizontally.

In short, do not underestimate ops‑tool systems. If you are interested, feel free to join us in tackling these challenges!

Finally, I recommend a memorable quote from Huang Yishan’s engineering‑management talk six years ago: Tools Are Top Priority . You can read the original article for more details.