Milvus 3.0 Segment Explosion: 12% Seal Threshold, 24h TTL, 16 Compaction Slots

01 Who Decides a Segment Is Full: Twelve Signals, One-Minute Polling

Milvus 3.0's write path uses StreamingNode to write to WAL while maintaining a StatsManager per growing segment, tracking row count, byte count, creation time, and last write time. A background seal worker scans all growing segments every minute (default) performing three scans: capacity policy, time policy, and blocking-L0 policy. The author enumerated twelve seal signals from source code:

Lifecycle signals (6): partition_not_found, partition_removed, collection_removed, recover, fenced, force — trigger when segment, partition, or collection disappears or is forced.

Periodic polling signals (6): capacity, lifetime, idle, growing_bytes_hwm, node_memory, blocking_l0 — evaluated every scan cycle.

Besides the minute-level scan, two fast paths exist: a single allocation that crosses the soft target triggers immediate seal, and node memory usage exceeding 60% (default) forces continuous sealing until memory drops. A segment can be sealed by allocation-time crossing, periodic scan, or memory pressure — whichever arrives first.

02 Segments Don't Need to Be Full to Seal: 12%, 24 Hours, One Crossing

Three mechanisms operate simultaneously under defaults:

Soft target crossing: Each growing segment gets a soft limit at allocation. Source comment: "Every segment may accept exactly one allocation that crosses its soft assignment target. Once that crossing allocation has been accepted, all later allocations must move to another segment." A large insert batch is placed whole, then the segment seals — no splitting. Higher throughput increases frequency of this cross-and-seal behavior.

Seal proportion: dataCoord.segment.sealProportion defaults to 0.12 (12%). Seal triggers when row count reaches limit × 12%, with a 10% jitter ( sealProportionJitter default 0.1) making actual threshold 10.8%–12%. dataCoord.segment.maxSize defaults to 1024MB; row limit derives from maxSize divided by estimated record size. Thus a segment seals at roughly one-tenth of its configured capacity.

Lifetime and idle: dataCoord.segment.maxLife defaults to 86400 seconds (24 hours) — segment seals regardless of size. Idle seal: no writes for 10 minutes and segment ≥ 16MB ( minSizeFromIdleToSealed) triggers seal. L0 segments also have a 10-minute lifetime.

Datacoord runs parallel strategies: segment binlog files > 32 triggers seal; channel total growing segment size > 4096MB seals the largest segment. Both strategies aim to proactively seal segments early and repeatedly.

03 Three Forces Driving Segment Count Up: Sharding, Large Batches, Delete Chains

Sharding: Collection shard count determines parallel channels; each channel manages its own segments. Doubling shards doubles baseline segment count — a parallelism tax.

Large batch inserts: When a single insert exceeds per-segment limit, the system creates N new segments for the bulk, then places remainder in existing segments. Source comment notes this prevents oversized segments which hurt query performance; the system prefers more segments over fat ones. Imported segments ( isImporting) are excluded from all four compaction strategies.

Delete chains (blocking-L0): Deletes/updates generate L0 segments (delete logs). L0 compaction only processes L0 segments whose time range does not exceed the earliest growing segment's start. While growing segments exist, overlapping delete logs queue up. When a channel's L0 entries reach 5 million or size reaches 64MB, blocking-L0 seal triggers: the earliest growing segment seals to unblock L0 compaction. Source ASCII diagram shows G1, G2, G3 growing segments overlapping L0a/L0b — all three seal. High upsert/delete rates thus accelerate segment creation via both write and delete pressure.

04 Compaction Chases But Hits Four Hard Bottlenecks

Compaction merges small segments, removes delete marks, and folds L0 logs. Trigger cadences: L0 every 10s, mix every 60s, clustering every 600s. Per-segment triggers: delete ratio ≥ 20%, delete logs > 16MB, or delete log files > 200. Despite frequent checks, four scheduling bottlenecks stall compaction:

Queue capacity: Compaction task queue default 100,000; full queue skips task generation for that cycle.

Signal channel: Auto-trigger signal channel capacity 100; full channel logs "signal channel is full" and discards trigger.

Mutual exclusion: Same channel: L0 and mix exclude each other; clustering and same-label mix exclude each other. A segment participates in only one compaction at a time.

Execution slots: DataNode dataNode.slot.slotCap defaults to 16 slots shared by compaction and import; tasks wait for free slots.

Priority defaults to level: L0 highest, then mix, then clustering. Under delete storms, L0 floods queue and slots, starving mix compaction — fragments never merge, segment count only rises.

05 Two Manual Interventions, Both With Costs

Force merge solves standard compaction's inherent deadlock: standard compaction is m→1 (output one segment, cannot exceed size limit). Example: five 2MB segments with 3MB limit — any merge exceeds limit, standard compaction stalls forever. Force merge is m→n: can shatter and reassemble segments, output count unrestricted. However: (1) marked public preview — "Do not use in production." (2) Async, non-blocking but consumes I/O and memory; run during low traffic. (3) Target size constrained: request < maxSize rejected; request > safety value silently clamped. Safety value = min(min QueryNode memory, min DataNode memory) / 4 ( memoryFactor default 4.0); standalone non-pooled halves again. Example: smallest QueryNode 16GB → max target 4GB. Oversized requests silently merge to safety value.

Clustering compaction defaults off. Requires schema scalar field as clustering key; redistributes entities by key values, builds PartitionStats segment-to-key mapping for query pruning. Auto-trigger needs three switches true: global dataCoord.enableCompaction (default true), clustering.enable (default false), clustering.autoEnable (default false). Thus manual only: client.compact(collection_name=..., is_clustering=True). One task per collection at a time; triggers check incremental data > 512MB and > 24h since last run. Official internal benchmark: 20M 768-dim vectors, 99% pruning → up to 25× QPS vs no compaction (not externally reproduced). Author judges clustering compaction suits large collections with clear filter dimensions; small collections don't justify cost.

Author's advice: verify auto compaction queue and priority aren't blocked before manual intervention. Manual only temporarily relieves pressure; auto pipeline is the main battlefield.

06 When Segment Count Becomes a Problem: Cost, Curve, Decision Order

Segments carry indexes; more segments = more indexes to build/load. Query broadcasts to all shards; streaming nodes search growing, query nodes search sealed. Handoff distributes sealed segments across query nodes balancing memory, CPU, segment count. Object storage sees many small files → list/read amplification. Segment count curve shape matters more than absolute number:

Slow rise with data volume: sharding, sealing, compaction in stable balance → no action.

Stepwise rise without data growth: sealing rhythm broken or compaction pipeline blocked → investigate immediately.

Common causes: high-frequency upsert/batch delete triggering blocking-L0 cascade; compaction queue backlog, slots saturated, mix starved by L0 priority. Cost asymmetry: raising maxSize (e.g., to 8GB) reduces segment count but increases per-segment memory load and failure recovery cost. Force merge's 4GB safety cap limits manual merge size. maxSize governs write-time seal; adjusting segment size shifts problem from count to size.

Three-step decision order:

Check seal conditions: 12% seal proportion, 24h lifetime, 10min idle, blocking-L0 5M entries/64MB.

Check compaction health: queue length, running tasks, slot saturation, mix starvation by L0.

Manual intervention: force merge for medium-segment deadlock (m→n), clustering compaction for query-pattern reorganization; schedule off-peak, compute min node memory / 4 first.

Note on maxSize discrepancy: official docs force-merge example uses 512MB; source commit 1306ae0816 paramtable shows 1024MB. Article uses 1024MB; conclusion unchanged — 12% seal proportion means segments seal far below either limit.

Full reality: segments are preemptively sealed by defaults, split by shards and large batches, blocked by delete chains, then slowly cleaned by a capacity- and mutex-constrained compaction system. Knowing each decision point turns the segment-count curve from a black box into a checklist of parameters to verify. How many segments does your collection have now? Map it against the tables in sections 02 and 04 — where is it stuck?