Why Random Splits Fail for Enterprise Data: Preventing Time, Entity & Business Leakage

Random 8:1:1 splitting assumes each row is independent, but enterprise data violates this: a single customer generates multiple orders, affiliated companies share contracts and transaction patterns, one template produces thousands of similar documents, one event leaves records across order, invoice, payment, and collection tables, adjacent time windows share business context, and labels are followed by disposition statuses, reason codes, and manual conclusions generated after the fact. When rows are shuffled, training and test files differ but the underlying business objects and information chains remain connected, producing fake generalization.

Define the Prediction Timepoint Before Deciding Usable Fields

Many leaks stem from the task definition lacking a time boundary. "Predict whether a customer will default" requires answering: at what timepoint is the prediction made, how far into the future is the outcome predicted, which historical window of facts may be used, when did each field actually occur and when did the system acquire it, and when does the label become final. For example, a project may define: predict 7 days before a receivable's due date, using only information already in the business system at that moment, to judge whether the receivable remains unpaid 3 days after the due date. This reveals three distinct times: fact occurrence time ≠ system acquisition time ≠ model-available-at-prediction-time. A contract may be signed early but its scan enters the warehouse days later; a payment may happen but the bank receipt syncs the next day; manual collection reason codes only appear after default. A field is usable only if it was truly available at the prediction timepoint, not merely because it exists in today's data warehouse.

Six Leakage Types That Inflate Test Results

The article identifies six leakage categories, each with typical manifestation and what the model actually borrows:

Row-level duplication : Raw records, cleaned results, or augmented samples appear across splits. The model borrows original or near-identical answers.

Entity leakage : Same customer, group, device, or contract entity appears on both sides. The model borrows object identity and historical behavior.

Template leakage : Same contract template, document format, or device configuration split randomly. The model borrows fixed structure and wording.

Time travel : Future records, future aggregates, or future knowledge enter training. The model borrows information that did not exist at prediction time.

Label leakage (hindsight) : Collection results, closure status, manual reason codes enter features. The model borrows post-label disposition traces.

Processing & knowledge leakage : Preprocessing parameters fitted on full data, or knowledge index contains post-prediction content, test labels, or gold answers. The model borrows test distribution, future knowledge, and answer clues.

These six collapse into three main threads: Entity leakage — test objects share identity, group, template, or event links with training objects; Time leakage — model uses information generated or obtainable only after the prediction timepoint; Business leakage — field names look ordinary but are actually derived from the label, disposition process, or human judgment. Row-level duplication is a cross-cutting prerequisite check; processing/knowledge leakage can span both time and business boundaries. The real question is whether the model learned generalizable business rules or merely recognized object identities, fixed templates, and future traces.

Splitting Unit Is Dictated by the Deployment Question, Not Stricter-Is-Better

Whether the same customer may appear in both training and test depends on the future use case. If the system predicts the next order of existing customers, historical customers can be in training but test records must fall after the training time window; if the goal is to assess generalization to never-seen customers, then neither the customer nor its affiliated group should cross splits. Different capabilities demand different splitting constraints:

Predict future behavior of existing customers : Primary constraint — time as primary axis; same customer only allowed in training from the past. Recommended observation slices — existing customers' future orders.

Generalize to never-seen new customers : Primary constraint — group by customer or corporate group. Recommended observation slices — new customers, new groups.

Generalize to new contracts or document templates : Primary constraint — group by template family. Recommended observation slices — unseen templates, rewritten templates.

Generalize to new organizations or regions : Primary constraint — hold out by organization, region, or regulatory scope. Recommended observation slices — new regions, new business units.

Predict subsequent states of a continuous event chain : Primary constraint — group by full event chain and truncate by time. Recommended observation slices — new event chains, later chain segments.

Grouping keys cannot rely solely on database primary keys. Multiple customers under the same group, supplementary agreements under a master contract, multiple alerts from the same device may all need to roll up to a higher-level business group. Whether entity overlap is allowed depends on whether production faces old objects' new behavior or entirely new objects. Overlap that matches the deployment assumption is not leakage; undocumented and unverified overlap is the risk.

Time Splitting and Entity Grouping Usually Need to Be Combined

Enterprise projects rarely rely on a single splitting strategy. For customer default prediction, first split by business timepoints into training, validation, and test windows, then impose customer-group, contract-template, and event-chain constraints:

Historical training window → buffer window → validation window → buffer window → independent test window

Buffer windows isolate cross-window continuing orders, payments, and collection events, preventing the first half of a business process from landing in training and the second half in test. Buffer length cannot be set mechanically; it must be based on the longest business cycle, label maturation time, and data sync latency. The combination follows this order:

First define prediction timepoint and time windows : clarify which information was available at that moment.

Then define business grouping keys : customer, group, contract, device, template, or event chain.

Set cross-window isolation rules : handle unfinished events, delayed labels, and data sync.

Balance distributions within legal partitions : then consider label ratios, industry, region, risk level.

Form independent challenge slices : reserve new templates, new organizations, and extreme time windows to validate extrapolation ability.

For data with business dependencies, randomization should occur only inside partitions that already satisfy business constraints, not as a first step followed by patching.

Four Data Partitions Must Isolate More Than File Directories

After splitting, training set, offline evaluation set, business acceptance set, and online replay partition need different visibility and flow rules. These four partitions are not the four AI data product types from the previous article; they are internal data zones in the training/evaluation pipeline separated by development visibility and lifecycle.

Training set : Main purpose — learn model capabilities. Visibility & isolation — usable by training and sample design, but must not contain independent evaluation answers.

Offline evaluation set : Main purpose — compare candidate versions. Visibility & isolation — isolated from training and targeted tuning; only controlled results returned.

Business acceptance set : Main purpose — decide fitness for real process. Visibility & isolation — maintained by business and quality roles; dev team should not per-item tune.

Online replay partition : Main purpose — validate real distribution and system integration. Visibility & isolation — de-identified, versioned, and subject to production data permissions.

Isolation objects also include: label generation rules and gold answers; prompt examples and human debugging materials; RAG knowledge versions and index snapshots; feature engineering, scaling parameters, and vocabularies; object identities, grouping keys, and event-chain mappings; detailed failure reports that could reverse-engineer test content. Preprocessing must respect boundaries: imputation, scaling, feature selection, vocabulary building, and dimensionality reduction parameters must be fitted only on training data then applied to validation and test. scikit-learn's leakage guide lists "split first, fit preprocessing only on training set" as a basic principle.

Leakage Checks Cannot Stop at One Text Deduplication Pass

Hash deduplication only catches exact duplicates. After format conversion, field concatenation, text rewriting, or sample augmentation, business-identical data can have completely different files and hashes. Before release, at least five layers of checks are required:

Exact duplication check : hashes of raw values, normalized text, and key field combinations.

Near-duplicate check : semantic similarity, template fingerprints, local fragments, and attachment reuse.

Business entity check : whether customers, groups, contracts, devices, organizations overlap across splits.

Time & lineage check : field generation time, acquisition time, processing lineage, and label maturation time — any crossing boundaries?

Manual business spot-check : domain-familiar reviewers inspect associations and hindsight fields that machine rules miss.

A leakage check report must answer: which rules ran and at what versions; how many exact duplicates, near duplicates, and entity overlaps were found; which overlaps match deployment assumptions and which must be blocked; which fields may originate after the prediction timepoint; who approved exceptions and on what basis. Results cannot be just "pass" or "fail". Allowed entity overlaps, time crossings, and template reuse must be documented with reasons, otherwise the next version cannot tell if boundaries were quietly shifted.

When New Data Arrives, Don't Just Reshuffle Everything

This section maintains the splitting baseline composed of sample ownership, time windows, and isolation relationships; evaluation cases, gold answers, and scoring rules versioning were covered in the previous article. If a dataset upgrade reshuffles all old and new data, objects previously reserved for independent evaluation may enter training, and historical metrics lose comparability. A safer practice is to freeze a Splitting Strategy Card :

Task time boundaries : prediction timepoint, observation window, outcome window, label maturation time.

Data snapshots : source systems, extraction time, query or processing version.

Grouping rules : primary grouping keys, associated grouping keys, and priority.

Time rules : windows for each collection, buffer periods, cross-period event handling.

Isolation rules : entities, templates, fields, and knowledge versions forbidden to cross collections.

Reproducible parameters : splitting algorithm, random seed, code version, output manifest.

Exception approvals : exception reason, impact scope, approver, expiry condition.

When new data arrives, first use the strategy card to decide which time window and business group it belongs to, then route it to training candidate pool, evaluation candidate pool, or online replay partition. Do not move existing objects back and forth across versions to chase a fixed ratio. If business objectives, label definitions, or grouping rules change, establish a new splitting baseline and explain why it cannot be directly compared with the old one, rather than silently overwriting history. Responsibility-wise: business owners define prediction timepoint, label, and forbidden information; data engineering maintains sources, lineage, and business grouping; algorithm team implements splitting and preprocessing; quality/evaluation lead independently runs leakage checks and signs the report.

When Random Splitting Is Still Valid

Random splitting is not a wrong method; it simply does not fit all business data. When samples are approximately independent — no shared entities, templates, or event chains; data has no clear temporal direction; preprocessing is fitted only on training set; and the goal is merely to verify statistical generalization on same-distribution data — random or stratified random splitting remains reasonable. For example, strictly deduplicated, independently collected static image classification samples may not need complex time windows and event-chain isolation. But as soon as the same photographed object, consecutive video frames, same-source augmented images, or the same collection batch are highly correlated, grouping by object or batch is still required. The criterion is not "is random splitting advanced" but "does it faithfully simulate the unknown data the model will face after deployment."

Final Deliverable Should Not Be Just Three Data Directories

An auditable, reproducible splitting outcome must include at least:

Splitting Strategy Card : task timepoints, windows, grouping, isolation, and version rules.

Leakage Type Checklist : itemized risks, detection methods, and blocking conditions.

Entity Grouping Rules : definitions of customer, group, template, and event-chain ownership.

Time Window Definitions : explicit observation, buffer, prediction, and outcome windows.

Pre-release Check Report : records of exact duplicate, near duplicate, entity, time, lineage, and manual spot-check results.

Only when these rules and evidence ship together with the data version do training, validation, and test sets become more than three directories — they become an engineering baseline that proves independence.

Summary

Row-level random splitting only separates records; it cannot automatically isolate the same customer, group, template, event chain, and future information. Enterprise data splitting should first define the prediction timepoint and available information, then choose time windows and business grouping keys based on the deployment question, balance label and sample distributions within legal partitions, and verify independence through hash, similarity, entity, lineage, and manual spot-checks. Random splitting still has applicable scenarios, but the prerequisite is approximate sample independence and that such splitting can simulate the truly unknown data the system will encounter in production. A trustworthy data splitting scheme proves not that "training and test files differ" but that the model at test time did not borrow object identities, future information, and business outcomes that it should not know in reality.