Stream.toMap() Duplicate Key Crash: Why the Common Fix Silently Loses Data

The Incident

A batch import of 5000+ employee records from Excel parsed and validated successfully, but after ten seconds the API returned 500 with a single log entry:

java.lang.IllegalStateException: Duplicate key 13800138000

The culprit was a one-line Stream collector that assumed phone numbers were unique:

Map<String, EmployeeImportRow> employeeMap = rows.stream().collect(Collectors.toMap(EmployeeImportRow::getMobile, Function.identity()));

The code had run for a long time because normal files had no duplicates. One duplicate row among 5000 caused the entire import to abort.

Why the Quick Fix Is Dangerous

The common fix adds a merge function to keep the first value:

Map<String, EmployeeImportRow> employeeMap = rows.stream().collect(Collectors.toMap(EmployeeImportRow::getMobile, Function.identity(), (oldValue, newValue) -> oldValue));

This suppresses the exception but silently drops the second row. The API then reports "Import successful: 4999 rows" and the user never knows which record was lost. For employee contacts this may be tolerable; for customer imports, permission configs, payroll, or settlement data, silent data loss is unacceptable.

The original system said: "Data has a problem, I can't handle it." The fixed system says: "Data has a problem, but I'll quietly throw one away."

Duplicate Keys Are a Business Rule Problem, Not a Stream Problem

Given two rows with the same phone number but different departments:

Row 12: Zhang San, 13800138000, R&D Dept
Row 328: Zhang San, 13800138000, Tech Center

Four distinct business rules could apply:

Reject the import and report the exact duplicate rows.

Keep the later row (overwrite).

Treat the file as update instructions — last occurrence wins.

Allow multiple values per key — then a Map is wrong; use Map<String, List<EmployeeImportRow>> via groupingBy.

The real issue is the unverified assumption that the key is unique. When the key comes from external input, that assumption is risky.

Detect Duplicates Early with groupingBy and Row Numbers

For fields that must be unique (e.g., phone numbers), first group by the key:

Map<String, List<EmployeeImportRow>> grouped = rows.stream().collect(Collectors.groupingBy(EmployeeImportRow::getMobile));

Then find keys with more than one entry:

List<String> duplicateMobiles = grouped.entrySet().stream().filter(entry -> entry.getValue().size() > 1).map(Map.Entry::getKey).toList();

If duplicates exist, throw a validation exception with the duplicate keys. But returning only the key (e.g., "Duplicate phone: 13800138000") forces the user to search 5000 rows. Better: store the original row number during parsing:

public record EmployeeImportRow(int rowNumber, String name, String mobile, String departmentCode) {}

Then report: Phone 13800138000 duplicated: rows 12, 328 Or return a structured error payload:

{ "code": "IMPORT_VALIDATION_FAILED", "errors": [ { "field": "mobile", "value": "13800138000", "rows": [12, 328], "message": "Phone number duplicated in file" } ] }

Normalization Before Uniqueness Checks

Two emails that differ only in case ( [email protected] vs [email protected]) or an employee code with a trailing space ( A001 vs A001) are technically different strings but business-wise identical. Uniqueness checks must use normalized keys:

private String normalizeEmail(String email) { if (email == null) return null; return email.trim().toLowerCase(Locale.ROOT); }

Phone numbers may need space/hyphen removal and country-code handling; employee codes may need trim and uppercase. Then group by the normalized key:

Map<String, List<EmployeeImportRow>> grouped = rows.stream().collect(Collectors.groupingBy(row -> normalizeEmail(row.email())));

Many "database has unique index but Java didn't catch it" issues stem from Java and the database disagreeing on equality.

Batch Database Checks Instead of Row-by-Row Queries

Even if the file has no internal duplicates, the database may already contain some keys. Querying per row (5000 SELECTs) kills performance. Instead, collect all normalized keys, batch-query, and compare:

Set<String> mobiles = rows.stream().map(EmployeeImportRow::mobile).map(this::normalizeMobile).collect(Collectors.toSet());
List<EmployeeEntity> existing = employeeRepository.findByMobileIn(mobiles);
Set<String> existingMobiles = existing.stream().map(EmployeeEntity::getMobile).collect(Collectors.toSet());
List<EmployeeImportRow> conflicts = rows.stream().filter(row -> existingMobiles.contains(normalizeMobile(row.mobile()))).toList();

For large datasets, split the IN clause into batches of 500–1000. This avoids the classic N+1 query pattern that only surfaces under real load.

Database Unique Constraints Remain the Final Guard

Application-level checks only provide friendly errors. Concurrent requests can both pass the check and then both insert. A database unique index ( UNIQUE KEY uk_employee_mobile (mobile)) is still mandatory. Java tells the user "Row 328 conflicts with existing employee"; the database guarantees "No two rows with the same phone can ever exist." Both layers are required.

When to Use the Merge Function

The three-argument toMap(keyMapper, valueMapper, mergeFunction) is valuable when the business explicitly defines how to merge duplicates:

Keep the latest log per user: (a, b) -> a.createdAt().isAfter(b.createdAt()) ? a : b Last write wins: (oldValue, newValue) -> newValue Sum quantities per SKU: Integer::sum Using (a, b) -> a merely to avoid an exception — without a business rule — hides the conflict instead of resolving it.

Don't Force One-to-Many into a Map

Mapping employees by department ID with toMap(Employee::getDepartmentId, Function.identity()) is wrong because a department has many employees. Adding a merge function that keeps the first employee discards the rest. The correct structure is Map<Long, List<Employee>> via groupingBy(Employee::getDepartmentId).

Handle Nulls Before Mapping

If a required field (e.g., phone) is null in rows 20 and 30, fail fast during initial validation with row-specific messages ("Row 20: phone cannot be empty"), not later inside toMap(). Structure the import pipeline in phases: 1) structural validation (required, format, length), 2) intra-file uniqueness, 3) database conflict check, 4) build Maps/indexes, 5) batch insert.

A Reusable "Must Be Unique" Collector

To make the uniqueness contract explicit in internal code, the author created a utility:

public static <T, K> Map<K, T> toUniqueMap(Collection<T> data, Function<T, K> keyMapper) {
    return data.stream().collect(Collectors.toMap(keyMapper, Function.identity(),
        (oldValue, newValue) -> { throw new DuplicateBusinessKeyException("Duplicate business key: " + keyMapper.apply(oldValue)); }
    ));
}

Usage:

Map<String, EmployeeImportRow> employeeMap = CollectionUtils.toUniqueMap(rows, EmployeeImportRow::mobile);

This signals that the key must be unique by business rule. For user-facing imports, groupingBy with row numbers remains preferable for precise error reporting.

Result: From 500 Error to Actionable Feedback

The final pipeline validates each phase and returns all errors at once:

Import failed, 3 issues found:
Rows 12, 328: Phone 13800138000 duplicated
Row 921: Phone format invalid
Row 1822: Phone 13900139000 already exists in system

The user fixes all problems in one round and re-uploads. This is the experience batch operations should deliver.

Three Questions Before Every toMap()

After this incident, the author asks before writing any Collectors.toMap():

Is this key truly unique in the business domain?

If not unique, should we error, overwrite, sum, or keep all?

If it should be unique, what should the user see when a duplicate appears?

The Stream API has no traps; the trap is hiding a business conflict inside a one-liner. In batch processing, the goal isn't just to process data — it's to tell the user exactly what's wrong when bad data arrives.