How a Simple Pagination Change Triggered a P0 Outage and What We Learned

In July 2025, a product manager requested pagination for the order query page to improve load times. The developer modified OrderService.java by changing the return type of queryOrders from List<Order> to a paginated PageResult<Order> and added offset logic, assuming the change was isolated.

// OrderService.java – before
@Service
public class OrderService {
    @Autowired
    private OrderMapper orderMapper;
    public List<Order> queryOrders(QueryOrderRequest request) {
        return orderMapper.selectOrders(request);
    }
    // other methods ...
    public void updateOrderStatus(Long orderId, Integer status) {
        QueryOrderRequest request = new QueryOrderRequest();
        request.setOrderId(orderId);
        List<Order> orders = queryOrders(request); // critical call
        if (orders.isEmpty()) {
            throw new BusinessException("订单不存在");
        }
        Order order = orders.get(0);
        order.setStatus(status);
        orderMapper.updateOrder(order);
    }
}

// OrderService.java – after
@Service
public class OrderService {
    @Autowired
    private OrderMapper orderMapper;
    // new paginated query
    public PageResult<Order> queryOrders(QueryOrderRequest request, int page, int size) {
        int offset = (page - 1) * size;
        List<Order> orders = orderMapper.selectOrdersWithPage(request, offset, size);
        int total = orderMapper.countOrders(request);
        return new PageResult<>(orders, total, page, size);
    }
    // other methods unchanged ...
    public void updateOrderStatus(Long orderId, Integer status) {
        QueryOrderRequest request = new QueryOrderRequest();
        request.setOrderId(orderId);
        // BUG: method signature mismatch
        List<Order> orders = queryOrders(request); // compile error ignored
        if (orders.isEmpty()) {
            throw new BusinessException("订单不存在");
        }
        Order order = orders.get(0);
        order.setStatus(status);
        orderMapper.updateOrder(order);
    }
}

A unit test was added only for the new pagination method, confirming the result size and total count, but it did not verify backward compatibility of the original queryOrders signature.

@Test
public void testQueryOrdersWithPage() {
    QueryOrderRequest request = new QueryOrderRequest();
    request.setUserId(12345L);
    PageResult<Order> result = orderService.queryOrders(request, 1, 10);
    assertNotNull(result);
    assertTrue(result.getData().size() <= 10);
    assertTrue(result.getTotal() >= 0);
}

After a routine deployment (order‑service v1.2.3) at 17:00 on Friday, the new endpoint appeared to work in a quick manual test. However, at 18:58 the monitoring system reported a spike: error rate jumped from 0 % to 67.8 %, QPS fell from 1200 to 234, and response times timed out. Multiple teams received alarm messages, and users reported that order placement was completely broken.

[Alarm] 18:58:03 错误率: 0% → 67.8% QPS: 1200 → 234 响应时间: 45ms → 超时 [Business] 18:58:15 产品：用户反馈下单功能完全不能用了！ [Tech] 18:58:32 监控：订单状态更新全部失败 [DBA] 18:58:32 数据库连接池正常，应该是应用问题 [SRE] 18:58:32 准备回滚！

Investigating the logs revealed a NoSuchMethodError for OrderService.queryOrders with the original signature, confirming that the runtime could not find the method expected by internal callers.

2025-07-19 19:01:23.445 ERROR [order-update-thread-1] com.company.service.OrderService - 订单状态更新失败
java.lang.NoSuchMethodError: com.company.service.OrderService.queryOrders(Lcom/company/dto/QueryOrderRequest;)Ljava/util/List;
    at com.company.service.OrderService.updateOrderStatus(OrderService.java:45)
    ...

Using the IDE, the author searched for all queryOrders call sites and discovered 17 usages: only the new controller endpoint had been updated, while 16 internal services (order cancellation, refund validation, statistics, etc.) still called the old method, causing a cascade of failures.

// Search result – 17 call sites
1. OrderController.queryOrderList() ✅ updated
2. OrderService.updateOrderStatus() ❌ not updated
3. OrderService.cancelOrder() ❌ not updated
4. OrderCallbackService.processCallback() ❌ not updated
5. OrderStatisticsService.calculateDaily() ❌ not updated
6. RefundService.validateOrder() ❌ not updated
... (11 more)

The impact analysis showed that 94 % of the call points failed, leading to 23,847 failed status updates, 5,672 failed payment callbacks, 156 user complaints, and an estimated GMV loss of about 650,000 CNY within 73 minutes.

┌─ 📊 业务损失统计 ──────────────────┐
│ 订单取消: ██████████ 250,000 (38.5%) │
│ 支付失败: ████████ 200,000 (30.8%) │
│ 状态异常: ██████ 150,000 (23.1%) │
│ 其他损失: ██ 50,000 (7.6%) │
│ 总计: 650,000 │
└─────────────────────────────────────┘

Emergency remediation involved a hot‑fix deployment (order‑service v1.2.4) with a gray‑release strategy, gradually increasing traffic from 10 % to full rollout. Post‑deployment metrics confirmed error rate < 0.1 %, all interfaces responding normally, and the new pagination working without affecting existing functionality.

✅ 所有接口响应正常
✅ 错误率 < 0.1%
✅ 新分页功能工作正常
✅ 原有业务功能不受影响

The post‑mortem identified four root causes:

Insufficient system thinking – only the new requirement was considered, ignoring existing callers.

Incomplete test coverage – only the new method was unit‑tested.

Superficial code review – reviewers focused on changed lines, missing global impact.

Flawed release process – no compatibility checks, integration tests, or gray‑deployment safeguards.

To address these, a set of concrete improvements were introduced:

Technical improvements

Introduce a CompatibilityChecker that scans method signature changes and generates compatibility reports.

@Component
public class CompatibilityChecker {
    public void checkMethodCompatibility(Class<?> clazz, String methodName) {
        // 检查方法签名变更对现有调用的影响
        // 自动扫描所有调用点
        // 生成兼容性报告
    }
}

Expand test strategy to include unit, integration, compatibility, and performance tests.

Revise the CI/CD pipeline to enforce code review checklists (functionality, code quality, compatibility, performance impact), automated compatibility checks, gray‑deployment, and final monitoring verification before full release.

# New pipeline definition
pipeline:
  - code_review
  - unit_tests
  - integration_tests
  - compatibility_check  # new
  - gray_deployment      # new
  - monitoring_check    # new
  - full_deployment

Process improvements

Code‑review checklist now includes a mandatory compatibility assessment.

Pre‑release gate requires all tests to pass, monitoring configuration ready, and a documented rollback plan.

Gray‑release steps: 10 % traffic → monitor → 50 % traffic → monitor → full rollout.

By institutionalizing these practices, the team aims to prevent similar incidents caused by non‑backward‑compatible changes.