How to Quantify and Benchmark Your IT Operations Performance

Preface

This article is part of a series on measuring IT‑operation maturity. It explains how to objectively assess an enterprise's IT‑operation level in the cloud era, describes the evaluation metrics and algorithms used by leading internet companies, and provides sources for the required data.

Core Evaluation Elements

Four major categories are used to score IT operations, each weighted to sum to 100 points:

100 points = Availability 50% + TCO 20% + Efficiency 20% + Technological Innovation 10%

1. Availability

Availability = 1 - (downtime / total service time)

Baseline availability for most services is 99.5%; core services often target 99.9% or 99.99%.

Sub‑metrics include:

Program availability

Security availability

Network availability (own network, carrier network, load‑balancer)

Server availability (overall failure rate, brand‑specific failure rate, component failure rate)

Traditional reliability indicators such as MTTR, MTTF, and MTBF are rarely used in large‑scale internet operations.

2. Cost

Total Cost of Ownership (TCO) is the standard cost metric. A simplified TCO model is shown below:

In leading internet companies a single server’s TCO can be as low as 15,000 CNY per year.

Cost components include:

Server purchase price (average per unit)

Network equipment price (average per port)

Cabling price (average per port)

IDC rental price (average per server; e.g., a 16 A cabinet priced at 8,000 CNY housing 10 servers yields 800 CNY per server)

Bandwidth price (average per Gbps)

Software price (average per server)

Outsourcing service price (average per server)

3. Efficiency

The efficiency dimension aggregates launch efficiency, repair efficiency, and resource‑usage efficiency.

Launch Efficiency

Measured from demand submission to production launch, with sub‑metrics such as budget approval time, procurement time, arrival time, rack‑up time, installation time, and deployment time.

Repair Efficiency

Measured from fault occurrence to fault resolution, with sub‑metrics including fault reporting time, hand‑over time, fault‑location time, and fault‑fix time.

Resource‑Usage Efficiency

Focuses on CPU, I/O, and storage utilization; CPU peak utilization in large internet firms often exceeds 40%.

4. Technological Advancement

Metrics include:

Number of patents

Number of papers, especially at top international conferences

Open‑source contributions (e.g., Alibaba’s contributions)

Unique innovations (e.g., Baidu’s first ARM server)

Commercialization of server technologies (e.g., machine‑learning‑based disk‑failure prediction, modular data‑center designs)

Ecosystem collaboration (e.g., BAT Scorpion organization)

Recording and Evaluating the Core Elements

With dozens of sub‑metrics, manual data collection is impractical and error‑prone. An integrated IT‑management platform that combines monitoring, asset management, alarm handling, fault knowledge base, and statistical reporting is essential.

Example 1 – Repair‑efficiency dashboard provides real‑time scores for the repair process:

Example 2 – The system can generate an overall operational score, creating a closed‑loop view from fault detection to resolution: