When Punishment Hurts Ops: Building Safer, More Collaborative Teams

This article, originally shared by a high‑efficiency operations expert group, begins with an introduction of the guest speaker, Liang Yupeng, chief architect and IM technology director at Huanxin, formerly a communications technology specialist at Sina Weibo.

Guest Introduction

Liang Yupeng Current chief architect and IM technology director at Huanxin, responsible for the overall R&D and management of the instant‑messaging cloud platform. Previously a communications technology specialist at Sina Weibo, designing and developing its messaging system.

On January 18, 2016, after a quiet period following the New Year, the team was performing a cluster migration that required overnight work. The next day a critical database issue occurred: a synchronization process that started at 5 am overwrote the production database, restoring data to a previous state. Fortunately a backup existed, and the team quickly isolated the primary from the replica, restored the primary from backup, and used the replica for incremental comparison.

Instead of focusing on technical remediation, some team members called for punishment of the responsible operator, believing that penalizing the individual would reinforce service stability.

Why Punishment Fails

Punishment may deter negligence, but it does not create value and often leads to poorer performance, negative attitudes, and a divided team. Fear of punishment reduces efficiency, encourages shortcuts, and shifts focus from technical excellence to internal politics, which is especially harmful for fast‑growing startups.

Progress Without Punishment

True growth should stem from team honor and a shared sense of purpose, not from fear. Learning from mistakes and turning them into experience is essential; otherwise, the same incidents will repeat.

An illustrative case from October 20, 2015, involved a junior trader at Deutsche Bank who mistakenly entered a net value as a total amount, costing a hedge‑fund client $60 billion. This “fat‑finger” error is commonly prevented by the four‑eyes principle, requiring two people to approve critical actions.

To apply this principle in a remote work environment, the team uses Tmux session sharing, allowing two operators to control the same console and keyboard input.

Technical Measures to Prevent Human Error

Beyond the four‑eyes principle, the team introduced technical safeguards:

Mandatory double‑confirmation for data‑recovery commands.

Alias handling for dangerous commands.

Disabling default Tmux sessions and forcing users to operate through named sessions with enforced aliases.

These measures demonstrate that human errors can be mitigated with appropriate tooling.

Technology‑Driven vs. Technology‑Led

Continuous innovation and a technology‑driven mindset are crucial for staying competitive. Companies that rely on top‑down directives (“leadership‑driven”) often fall into a punishment culture that stifles technical initiative. Emphasizing technology as a driver, not just a tool, empowers engineers and improves overall performance.

In summary, building a resilient operations culture requires collaborative safeguards, technical controls, and a focus on collective honor rather than individual punishment.

How Can Operations Avoid the Pain?

Join the GOPS2016 Global Operations Conference in Shenzhen to learn from over 30 top‑level operations experts. Early‑bird tickets are discounted until February 29. Scan the QR code below or click the original article link for more details.