Milvus 3.0 Delete Space Not Returned? The Queued Release Pipeline: Compaction Then GC

01 What Happens When Delete Returns Success

From the proxy side, a delete request only packages the target primary keys and a timestamp into a delete message and appends it to the WAL (Write-Ahead Log). Once the message is written, the request returns success. No data files, indexes, or segments are touched at this point.

The message structure determines its nature. The proxy hashes primary keys into virtual channels, each key carrying a timestamp. The batch size is limited by quotaAndLimits.limits.maxDeleteSize (default 16 MiB). The official config comment calls this a "tombstone" — a declaration that something should disappear, not the disappearance itself.

The returned delete_count is often mistaken for a confirmation receipt. For simple primary-key equality or IN expressions, it returns the number of keys in the expression; for complex conditions, it returns the number of keys matched by the query. Deleting a non-existent key still increments delete_count — it counts submitted tombstones, not confirmed deletions. The compactor's MissingDeleteCount metric tracks tombstones that find no matching row, confirming the disconnect.

02 Complex Condition Delete: Query First, Then Delete

After parsing the delete expression, the proxy takes two paths. Primary-key equality and IN expressions use a lightweight path that directly writes tombstones. All other conditions (scalar field filters, time-range cleanup) go through complexDelete: the condition is serialized into a query plan, sent to query nodes for full execution, matching primary keys are streamed back, and tombstones are written for those keys.

Proxy code comments note that a complex plan can reach tens of MiB; to avoid duplicating this memory in every concurrent shard callback, the plan is serialized once then distributed. A broad filter on a large collection costs a full query over the hit range and consumes delete rate-limiting quota.

03 Why Queries Stop Seeing Deleted Data

After tombstones enter the WAL, query nodes consume them in timestamp order and write them into each segment's internal DeletedRecord structure (in segcore): a timestamp-sorted skip list of (timestamp, row ID) plus a bitmap. During query execution, the bitmap is OR-ed into the result filter — rows with bitmap bit set to 1 are excluded from candidates. The data body remains in the segment, indexes still serve it, but the final result assembly blocks those rows.

The timestamp in the delete request is not a manual parameter; it is the GuaranteeTs derived from the query's consistency level. A tombstone takes effect only when its timestamp is earlier than this watermark. "Deleted but still queryable" is almost always because the watermark hasn't advanced past the deletion moment — wait a few seconds and re-query. A safeguard: tombstones with a timestamp not later than the insert timestamp are discarded, protecting against concurrent delete-then-write on the same key.

04 Tombstones Themselves Become Data

Delete records don't stay in the log forever; they are materialized into formal segments. WAL delete streams are batched into Level-0 (L0) segments, with a default threshold of 500,000 rows or 32 MiB per batch. L0 segments are first-class citizens in the segment system: they participate in failure recovery (their earliest position becomes the delete checkpoint), are organized by partition and channel, and have a full lifecycle. Their only storage difference from regular segments is content — they contain only tombstones.

This explains "I only delete, never write, yet capacity keeps growing": the growth is the delete records themselves. A tombstone is only a few dozen bytes at the message layer, but high-frequency deletes plus WAL retention and materialized copies make this non-trivial.

05 Compaction: Where Deleted Rows Physically Disappear

Physical elimination happens in two steps.

Step 1: L0 Compaction — A background task polls every 10 seconds (default). It takes accumulated L0 segments, uses Bloom filters of sealed segments to route tombstones to their target data segments, and rewrites a delete log for each segment. This step still does not touch the data body; it only archives scattered tombstones by "who they belong to."

Step 2: Mix Compaction — Triggered every 60 seconds (default). During segment merge, it reads all tombstones for the segment into a delete set, then scans the data row by row. Rows hitting the delete set are omitted from the new file, along with rows exceeding the collection TTL. After both L0 and mix compaction clocks complete, space still hasn't returned — that clock is the GC in the next stage.

Deleted rows are not erased in place; they remain in the old files, which are marked as dropped and never referenced by new versions. Logically the data is gone forever, but its bytes still occupy disk.

06 The Final Stage: GC's Two Clocks

Dropped files are cleaned by the Data Coordinator's garbage collector, which is deliberately slow. Two default tolerance windows: dropped segment files wait 3 hours ( dataCoord.gc.dropTolerance default 10800 seconds) before deletion; orphan files not found in metadata wait 24 hours ( missingTolerance default 86400 seconds). These windows are defensive — preventing accidental deletion of files still being created or referenced — at the cost of hour-scale space reclamation.

An additional bottleneck: an open community issue ( #49585) notes that the Data Coordinator's four GC sub-tasks run sequentially in a single goroutine. When dropped segments pile up, segment collection starves index collection; production observations show ~55% of index files become segment-index orphans (segment gone, index file remains). If your cluster underwent massive deletes, "space not returning" may also be caused by GC queueing, unrelated to config.

No disk movement half an hour after delete is normal — the clocks haven't ticked yet. If space hasn't returned after days, investigate GC queueing and segment backlog.

07 Per-Stage Troubleshooting: Mapping Phenomena to Stages

Deleted but still queryable : Visibility stage. The watermark is a timestamp derived from the query's consistency level; tombstones only take effect after the watermark passes the deletion moment. Re-query after a few seconds to distinguish "not effective yet" from "not deleted". delete_count does not answer this — it only counts submitted tombstones.

Not queryable, but space not freed : Compaction and reclamation stage. Check in order: is auto-compaction enabled for the collection (default on)? Has the deletion passed the three default clocks (L0 compaction 10s, mix compaction 60s, GC 3h)? Are dropped segments heavily backlogged (GC queueing issue)?

Only deletes, no writes, yet capacity rises : Delete records themselves consume storage. Verify L0 segment materialization and compaction are running normally, rather than searching for "undeleted" data.

The cost boundary is the flip side of this table: deletion is light when it's a few-dozen-byte tombstone message, heavy when it's a query-then-delete plus a growing write load, plus hour-scale space clocks. When scheduling data cleanup, factor in each stage's default clocks — don't just watch the delete return value.