Improving Test Sufficiency for Business Finance Systems

Test sufficiency is a long‑standing challenge in software testing. This article explores how to increase the sufficiency of testing for business‑finance (settlement) systems to reduce online defects.

Understanding Test Sufficiency

Test sufficiency measures how thoroughly a system is exercised. Code coverage is only a starting point; true sufficiency requires covering all user scenarios, events, data variations, configurations, states, and exceptions. Even with exhaustive modeling, 100% coverage is rarely achievable.

Test Sufficiency Modeling

A formal model defines the set of variable factors S = {A, B, C, …}. Each factor (e.g., A) has a set of possible values (a1, a2, …). The Cartesian product of all factors forms the denominator of the sufficiency ratio. The process follows Requirement → Scenario → Test Case , emphasizing scenario‑driven case design.

System Analysis and Scenario Modeling

Business, test, and user scenarios are unified as “test scenarios”. Modeling these scenarios clarifies the scope of testing and isolates contract‑bound units, reducing test chain length and associated costs.

Combinatorial Coverage

Using t‑way combinatorial coverage (e.g., 2‑way, 3‑way) quantifies how many variable combinations are exercised. For a system with variables a, b, c, d each taking values 0/1, 1‑way coverage is 100%, 2‑way coverage is 33.3%, and higher‑order coverage drops sharply. The metric (p, t)‑completeness expresses the proportion of t‑way combinations whose coverage exceeds a threshold p.

Coverage Target Model

Empirical studies show that achieving high 2‑way coverage often discovers >99% of defects, making higher‑order coverage unnecessary in most cases.

Test Case Design

Test cases are derived from the scenario model and combinatorial analysis, aiming to maximize scenario coverage with minimal cases. Variable factors such as tax rate, fund direction, invoice direction, and amount direction are encoded (a, b, c, d) to simplify design.

Tool Support

Several tools automate combinatorial test generation, e.g., Hexawise , Kiwi TCMS , and the open‑source Python library allpairspy .

Practical Adoption

The team applied these methods to the settlement system, defining contracts, identifying key variable factors, and generating baseline automated test cases, which significantly reduced pre‑release risk.

References

• NIST, Combinatorial Coverage Measurement Concepts and Applications (https://csrc.nist.gov/CSRC/media/Presentations/Combinatorial-Coverage-Measurement-Concepts-and-Ap/images-media/kuhn-et-al-iwct13.pdf) • https://www.pairwise.org/tools.html • https://github.com/thombashi/allpairspy