JiTTesting: Just-in-Time Testing for AI-Speed Code Changes

Why Traditional Testing Fails with AI-Generated Code

When AI programming tools (Agent/Copilot) modify large codebases in minutes, traditional testing systems face three critical problems:

Maintenance cost explosion: AI-generated diffs are large and frequent; human-written unit tests become obsolete overnight. Treating every AI-generated test as a permanent asset multiplies maintenance effort.

Hardening tests only guard the future, not the present: Hardening tests are designed to pass at generation time, then stored for long-term regression prevention. They are insensitive to behavioral changes introduced by the current diff—a green suite does not guarantee the new code is bug-free.

Full static test suites are not cost-effective: Higher AI output makes static suites more brittle, while deep logic changes often escape conventional functional tests.

Code change velocity now far exceeds human test-writing speed. Traditional testing assumes code is correct and does not actively hunt for unexpected behavioral differences in the current diff.

JiTTesting: Splitting Tests into Two Tracks

JiTTesting (Just-in-Time Catching Test Generation) reframes testing by separating it into two independent lines that coexist without replacing each other:

Hardening Test vs. Catching Test

Purpose: Hardening tests prevent future regressions; catching tests detect behavioral differences in the current diff.

Lifecycle: Hardening tests are committed and maintained long-term; catching tests are ephemeral—used once and discarded by default.

Pass condition: Hardening tests must pass at generation time; catching tests are designed to fail on the diff (exposing a difference).

Weak Catch vs. Strong Catch

A test that passes on the old version ( parent) but fails on the new version ( diff) is a Weak Catch —it only proves a behavioral difference exists. Applying a general oracle (e.g., exception, assertion violation) determines whether the failure is a real bug ( Strong Catch / true positive) or an expected change/flaky test (false positive).

Harvesting Passing Tests as Hardening Candidates

During the catching flow, LLM-generated tests run on both parent and diff. Tests that pass on both versions enter a passing pool as Hardening Candidates . They are not auto-committed; human review evaluates stability, long-term value, and maintenance cost before promotion to permanent hardening tests. This avoids both discarding all generated tests and blindly adding them.

Core Components

Dodgy Diff Pipeline (Broad Coverage)

Treats the diff directly as a "suspect mutant" without inferring intent. The LLM generates tests to distinguish parent vs. diff behavior. Simple, wide coverage, but higher false-positive rate.

Intent-Aware Pipeline (Precision)

LLM first reads the diff title, description, and code to infer business intent, then lists risk points (nulls, boundaries, state machines). It injects mutants on parent (e.g., changing >= 500 to > 500, injecting return null, negating conditions—six mutation categories) to produce tests that pass on parent but fail on the mutant. These tests are then run against the real diff.

Meta experimental data: Intent pipeline generated 811 tests, yielding 52 weak catches (6.4% rate) covering 7.9% of diffs; Dodgy generated 1,621 tests with 41 weak catches (2.5%) covering 4.0%. Intent's catch rate is approximately 2.6× Dodgy's.

RubFake Rule Evaluator (Deterministic Signal Layer)

Identifies 14 false-positive patterns (boolean flips, mock errors, reflection breaks, etc.) and 26 risk tags by scanning execution results and code snippets. It flags suspiciousness but does not make final judgments.

LLM-as-Judge Integration (Semantic Scoring Layer)

Multiple models output a risk probability score (-1 to +1) with a severity level and textual rationale. Scores are aggregated and ranked to reduce manual review load.

Human-in-the-Loop (Final Gate)

Engineers receive a simple yes/no question: "Is this behavioral change expected?" (e.g., "Boolean changed from true to false —expected?"). Expected changes are dismissed; unexpected ones trigger code fixes. Reviewers never read the generated test code.

Execution Steps (Triggered on PR Submission)

Environment isolation: CI checks out parent (master baseline) and diff (proposed version) into separate workspaces; tests run in isolated subprocesses to avoid APM probes or custom ClassLoader pollution.

Diff parsing: Extract changed functions/classes, filter irrelevant files, output structured diff summary for downstream pipelines.

Parallel pipeline generation: Dodgy writes tests targeting diff behavioral differences; Intent infers intent → lists risks → creates mutants → generates tests that pass on parent and fail on mutants.

Dual-version acceptance: All candidate tests run on parent first—must all pass (failures indicate test bugs, discarded). Same tests then run on diff. Only parent pass + diff fail become weak catches; tests passing both enter the passing pool (hardening candidates).

Noise reduction and prioritization: RubFake emits deterministic signals → LLM-as-Judge adds semantic scores → combined ranking surfaces high-confidence weak catches for human review.

Human review: Engineers answer the expected-change question in seconds to minutes. Catching tests are not committed by default.

Selective harvesting: From the passing pool, tests with stable assertions, valuable coverage, and manageable maintenance cost are promoted to hardening tests after human review. Current implementation archives them without auto-push.

Rollout Phases

Adopt incrementally: (1) Pilot Dodgy pipeline on one project to measure weak-catch rate and per-run cost; (2) Calibrate project-specific RubFake rules to tune false-positive suppression; (3) Add single-model semantic scoring, verify complementarity, then upgrade to multi-model ensemble; (4) Deploy Intent pipeline with its three-way matrix ( parent / mutant / diff) and complete the passing-test harvesting loop. Avoid pursuing full auto-commit from day one.

Essence

JiTTesting adds a dynamic, diff-tailored defensive line to traditional testing. The faster AI writes code, the more valuable this line becomes—yet it does not replace permanent test assets. Instead, it lets "temporary probes" and "permanent hardening" each do their job.