Introducing PIBench: A Payment‑Scenario Benchmark for AI Coding Agents

Background

Recent rapid advances in AI coding agents have led to agents handling more development tasks. Integrating payment and transaction flows, which involve financial safety, has become a critical test of commercial viability.

Why Payment Integration Is Hard

Payment integration requires complex business processes, fund flows, and exception handling, imposing stricter requirements than typical coding benchmarks.

Alipay‑PIBench Overview

Alipay‑PIBench is an open‑source benchmark built from real Alipay payment integrations. It covers:

9 payment product categories

9 open‑source projects from GitHub (e.g., litemall, ez_tickets, laravel‑gymie, bookcars, saas‑starter, bill‑express, medical‑appointment, AI‑content‑payment)

18 project‑level tasks

459 scenario‑based Rubric items that evaluate product APIs, transaction creation, front‑back‑end interaction, payment confirmation, business fulfillment, exception handling, refund boundaries, and fund safety

Construction Methodology

The benchmark addresses three questions: how to create realistic payment‑integration tasks, how to turn “what counts as correct” into executable evaluation, and how to quantify the value of structured payment knowledge for agents. All steps share a Rubric Contract that defines scoring standards and drives test generation.

Step 1 – Encode Expert Experience into a Deterministic Contract

Domain experts translate product specifications, business processes, and known pitfalls into a Rubric that specifies correct integration criteria, trust boundaries, and prohibited exception states. The Rubric produces two artifacts: the task instruction for the agent and the evaluation contract for the tester.

Step 2 – Generate Test Cases from the Rubric (Agent‑Assisted, Expert‑Reviewed)

Each Rubric line is turned into a concrete test case. Because the same requirement may be verified differently across products (e.g., client‑side result handling for mini‑programs vs. web), experts review agent‑generated cases to ensure they target the intended Rubric and reflect realistic payment semantics. Rare failure modes are reproduced with a programmable mock gateway.

Step 3 – Layered Evidence Collection

Static Check : Verify presence of expected structures (SDK, keys, entry points) without executing code.

Unit & Integration Tests : Run the application to confirm creation, API calls, state transitions, and cross‑module interactions.

E2E Tests : Drive the full user‑facing flow from payment entry to final status, checking ticket issuance, membership activation, etc.

LLM‑Assisted Assessment : Use LLMs to judge semantic criteria that are hard to encode, such as correct product selection, signature placement, and risk handling.

Out of 459 Rubric items, 399 are deterministic checks and 60 rely on LLM assistance.

Step 4 – Simulate Exceptions and Conduct Red‑Team Attacks

A deterministic test harness launches each evaluation in a fresh Docker container with a mock Alipay gateway capable of injecting scenarios such as wrong amount, bad signature, unsigned notification, timeout, gateway error, replay attacks, and over‑refunds. Red‑team style attacks then verify that agents reject malformed inputs and leave no corrupt state in the database.

Step 5 – Fixed Denominator Scoring

Scores are computed as a weighted Rubric Pass Rate, giving integration and E2E checks double weight. Any Rubric that yields no evidence counts as a failure, preventing missing evidence from being ignored.

Example: Gym Membership Mini‑Program

The gym‑membership mini‑program lacks a payment step after order creation. The benchmark turns this gap into a task, derives static, integration, and LLM‑assisted checks from the Rubric, and evaluates the agent’s solution end‑to‑end.

Experimental Results

Across 18 tasks and multiple mainstream models:

Model performance varied widely; the strongest model achieved an average score of 91.37 %, while the weakest reached 68.58 %, a gap of over 22 percentage points.

Providing agents with the official Alipay payment‑integration skill increased average scores by 10.31 % (101 out of 108 paired experiments improved), confirming the practical value of structured expert guidance.

Conclusion

Alipay‑PIBench defines “what counts as a correct payment integration” and open‑sources the full Rubric, enabling the community to benchmark, calibrate, and improve the security and reliability of AI coding agents. The 459‑item rubric serves as a ready‑to‑use checklist for developers, and the Rubric‑to‑test pipeline can be adapted to other high‑risk domains such as government, healthcare, and industrial control.

Paper: https://arxiv.org/abs/2607.14573

Code repository: https://github.com/inclusionAI/PIBench