Why Embracing Risk Is Essential for Effective IT Operations

We must admit we live in an imperfect world, and the same applies to IT operations; imperfections are the norm.

Introduction

Because the IT world is imperfect, operations are necessary. In previous articles I discussed the goals of operations and risk; in the risk‑control article I warned against refusing to accept risk.

For example, demanding that a vendor guarantee no equipment failure or that submarine cables never break is unrealistic; it reflects a refusal to accept risk and the mistaken belief that risk will not affect us.

Both attitudes amount to self‑deception. Over‑designing for ultra‑high reliability without regard to goals or cost is also a form of risk denial.

1. Shifting Risk to Vendors?

Relying on a vendor’s reluctant promise of no risk is the simplest yet most ineffective approach, because the risk remains unaddressed and is merely transferred.

Even with post‑incident compensation, the business goal of risk control is not achieved; the correct approach is to understand all risks from the vendor and prevent concealment.

This is difficult during the commercial phase, when vendors may hide risks to secure contracts, but easier during equipment delivery. Vendors may claim 100% reliability, yet failures still occur.

2. Never Count on Luck

Optimism that risk will not affect you is the most dangerous factor in operations. A case: on a weekend an engine failed, but dual‑engine protection switched automatically, so business continued.

After the alarm, staff assumed the weekend meant no immediate action was needed and postponed replacement, yet the second engine failed that night, causing severe outage.

Identical components from the same batch under the same conditions have a much higher joint failure probability than theory suggests.

Operational policy requires immediate handling of such incidents, preventing many serious failures.

3. What Is Compromise in Operations?

Compromise means acknowledging the existence of risk and managing it. Technical and resource constraints make some risks unavoidable, so they must be accepted.

Passive acceptance after a risk materializes is not compromise but avoidance.

The correct posture is: when risk exists and cannot be avoided, employ measures to control loss and impact rather than waiting for it to become fact.

4. How to Compromise Effectively?

Technical staff often fail to compromise properly, either ignoring risks after notifying management or refusing to adapt when conditions are unmet, sometimes resorting to futile rituals.

Temporary arrangements can become permanent if not reviewed, leaving hidden hazards such as ad‑hoc policies or junk configurations.

In one incident, massive network packet loss was traced not to traffic shaping but to an old temporary policy limiting bandwidth on a core device; removing the policy resolved the issue.

Preparedness plans are vital risk‑mitigation tools.

A preparedness plan defines emergency actions when risk materializes, ensuring quick response; however, plans must be rehearsed, otherwise they are ineffective.

Case example: A data‑center core could not support dual‑machine hot‑standby, so a cold‑backup engine board and business board were prepared with a switch‑over plan. When the engine board failed, an unfamiliar night‑shift engineer mishandled the power‑off procedure, causing severe delay.

Conclusion

Compromise is essential and must be executed well in operations. It is not ignoring risk nor refusing it, but balancing risk acceptance with business goals to avoid chaos.