Why 5M Fake Users Miss Real Bugs: 3 Data Generation Lessons
The article shares three hard-learned lessons on generating realistic test data: modeling business rules as state graphs for banking payments, matching production distributions with generative models for load testing, and building dialogue trees with LLM fine-tuning for AI intent recognition—proving that data fidelity, not volume, determines test validity.
Scenario 1: Strong-Rule Systems — Data Grows Through State Evolution, Not Random Assembly
A bank core payment system upgrade project saw 83% of automated test cases fail in the first run. Failures were not due to script bugs or environment issues but to three basic business rule violations: insufficient balance, illegal account status, and anti-money-laundering rule triggers. The randomly generated test data simply did not satisfy the business logic.
Root cause analysis revealed that test data in strong-rule systems is not a collection of isolated fields; it is constrained by an entire chain of business rules. A valid transfer record requires a complete state evolution: account opening → deposit → limit setting → transfer initiation. Any break in that chain renders the data useless.
The team restructured data generation in three steps:
Model the business process as a directed state graph; data generation follows state transitions, no shortcuts.
Embed real-time validation at each state node — e.g., transfer amount must be ≤ available balance × (1 − freeze ratio) — and regenerate if validation fails.
Generate-and-verify: every data record passes through the pre-risk-control gateway before release.
Result: 100,000 transaction records all passed risk-control validation; test case pass rate jumped from 41% to 99.2%. The lesson: in strong business systems, test data = rule engine + state evolution + real-time verification. Random functions only fool yourself.
Scenario 2: Load Testing — Authenticity Lies in Distribution Fidelity, Not Volume
A promotional load test required simulating 5 million concurrent users. Using Faker libraries, the team easily hit the volume target, but the test report showed distorted distributions: top 10 hot products accounted for 78% of orders (vs. 22% in production), and traffic was concentrated in tier-1 cities with almost none from lower-tier cities. Real performance risks — cache penetration, shard hotspots — were completely missed; the test was effectively wasted.
Collaborating with the data team, they extracted 12 features from 30 days of production logs: user active periods, category preferences, region-price sensitivity matrices. They then used a generative model (Wasserstein-distance-constrained GAN) to synthesize realistic user-item-behavior tuples that matched the true joint distribution.
Effect: cache hit rate deviation narrowed from ±37% to ±2.1%; shard load standard deviation dropped 64%. Conclusion: pressure test data realism depends on distribution fidelity, not scale. 50,000 realistic users beat 5 million fake ones.
Scenario 3: AI Interface Testing — The Challenge Is Dialogue Logic, Not Syntax
Before launching an intelligent customer service NLU module, the team needed to test intent recognition robustness against vague expressions, dialect variants, and multi-turn references. Traditional text generators produced grammatically correct but logically hollow sentences. Example: "Help me check yesterday's that package" — the words "yesterday" and "that" lack contextual anchors, making the utterance impossible in a real conversation.
They adopted a three-level generation approach:
Dialogue trees: clustered historical sessions into typical paths (inquiry → follow-up → confirmation → complaint) to give conversations a skeleton.
Template injection: inserted domain-specific terms (e.g., logistics "pickup code", "stuck depot") at tree nodes.
LLM fine-tuning: used LoRA to fine-tune a small-parameter model (Qwen-1.5B) for coreference resolution and ambiguity generation.
Daily output reached 23,000 test utterances, uncovering three classes of boundary cases completely missed by the original test set: cross-turn temporal reference confusion, homophone intent shift, and others that are extremely hard to conceive manually.
Conclusion
Three scenarios, three data types: financial data constrained by rules, load-test data constrained by distributions, AI test data constrained by semantics. Constraints differ, but the core insight is the same: You are not testing code; you are testing the real world that code must serve. Fake data only validates a non-existent world. The team's rule of thumb: spend 1 hour designing the data generation strategy to save 10 hours of futile debugging.
Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.
