Why Milvus 3.0 Can't Find Just-Inserted Data: The Hidden 5-Second Consistency Window

01 Water Level: The Hidden Parameter in Query Requests

The core concept is GuaranteeTs (called "water level" in the article). The official documentation states: "GuaranteeTs notifies query nodes not to execute search or query requests until they can see all data before GuaranteeTs." In plain terms, when you issue a search, you are saying "I only need to see data written before a certain moment." That moment is GuaranteeTs. The query node compares it with its own service water level (tSafe): if tSafe is higher, it executes immediately; if lower, it waits until tSafe catches up.

The documentation puts it bluntly: "When GuaranteeTs is greater than ServiceTime, the query node must wait until ServiceTime exceeds GuaranteeTs before executing the search operation." Therefore, "just inserted but not searchable" is not data loss — the data's timestamp simply falls outside the water level of this query. "Can it be found?" is a property of this query, not of the data.

02 Where Water Level Comes From: TSO and timetick

To understand why a water level is needed, look at the write side. Milvus is a distributed system; local clocks on different nodes are unreliable. Every insert operation gets its timestamp from the TSO (Timestamp Oracle) , a global sequencer producing a uint64: high 46 bits are millisecond-level UTC, low 18 bits are a logical sequence number.

Global timestamps alone are insufficient: data flows through message queues to query nodes, and messages from different proxies have no ordering guarantees. Milvus solves this with timetick : each proxy reports its latest message timestamp to Root coord every 200 ms by default. Root coord takes the minimum of all reported values and broadcasts it. Downstream consumers reading this timetick know that all messages with smaller timestamps have arrived. The query node's service water level tSafe advances every 200 ms in this way.

03 Your Consistency Level Is the Water-Level Knob

What value does GuaranteeTs take? You don't calculate it yourself: the consistency level you specify in the request is the knob the proxy uses to compute the water level. The conversion happens in internal/proxy/dql/task_search.go in PreExecute: if the request says "use collection default consistency," the collection's level is used; otherwise the request level is used; legacy SDKs that only send GuaranteeTimestamp follow a magic-number compatibility path ( strongTS=0, boundedTS=2, sentinel constants in util_dql.go).

After that, the result is compared with the collection schema's update timestamp and the larger value is taken — source-code comments explain this ensures schema-change views take effect before new read requests.

The conversion logic for the four levels ( ParseGuaranteeTsFromConsistency) is worth examining individually:

Strong : ts = tMax — the latest timestamp just assigned to the request. The query node must wait for the water level to reach here. You see the freshest world; the cost is waiting.

Bounded (default) : ts = tMax − GracefulTime. GracefulTime defaults to 5000 ms ( component_param.go 's DefaultGracefulTime = 5000 // ms, same in milvus.yaml). Under default config, your query's water level lags "now" by 5 seconds. Data written within those 5 seconds is not in the guaranteed view. This is by design, not a failure.

Eventually : ts = 1. A sentinel value; the water level is so low it effectively says "I don't check," and the query node executes immediately on its current view.

Session : Different — the switch has no branch for it; whatever timestamp is passed in is returned unchanged. The proxy does no conversion for Session level. The guarantee of reading your own writes relies on the client putting the latest write operation's timestamp into the request. Documentation concurs: "Client uses the latest write operation's timestamp as GuaranteeTs."

The mechanism's center of gravity is in the SDK, not the server you're querying. If the client fails to carry the timestamp correctly, the session guarantee silently fails, and server logs show no anomaly.

04 Query Node: Wait or Error

When the water level reaches the query node, handling logic resides in internal/querynodev2/delegator/delegator.go 's waitTSafe.

Fast path: if current tSafe is already above the water level, execute immediately. Slow path: standard condition-variable wait — if water level hasn't arrived, block waiting for timetick to advance.

It doesn't wait forever, and neither exit path "silently gives you stale data": before waiting, it computes lag (difference between required water level and current service time). If lag exceeds maxTimestampLag (default 86400 seconds ), it errors immediately ( TsLagTooLarge); if tSafe doesn't advance within waitTsafeStallTimeout (default 3 seconds , an unexported internal parameter), it's deemed stalled, fails fast, and lets the proxy retry on another replica.

So in strong-consistency scenarios, "can't find" normally manifests as slowness or errors. Actually seeing stale data under strong consistency would indicate a deeper problem.

The documentation only describes "Strong = wait until latest," but the source code contains an official acceleration path for strong consistency: speedupGuranteeTS (typo in source). Under non-iterator conditions, it can substitute the WAL's local latest MVCC timestamp for guaranteeTs. Code comments explicitly call this "make fast strong consistency search," with a TODO to merge the two timestamps later. Planning strong-consistency latency based solely on documentation may be inaccurate.

05 Triage When You "Can't Find"

Condense the mechanisms into a decision tree. When "insert succeeds but immediate query returns nothing," check three categories in order:

Consistency Semantics (most common) : Check the actual consistency level used for this request. Default Bounded means water level lags up to 5 seconds; newly written data not being in view is normal behavior.

Link Anomaly (Strong consistency scenarios) : If using Strong, focus on errors and latency: TsLagTooLarge, request timeouts, tSafe stalls. Direction is to investigate timetick link health, not to suspect the data view. A real community case: milvus-io/milvus issue #49435 (47 comments) — Strong + upsert concurrency triggered "Timestamp lag too large" errors.

Real Failure (last resort) : After ruling out the first two, examine write pipeline and replica status. Two other intuitive suspects can be dropped first: visibility is decided by timetick water level; query nodes serve both flushed batch data and unflushed streaming data simultaneously; "must wait for flush/index build to query" is not the primary explanation for "can't find."

Selection guidelines for three scenarios:

Post-write read-back verification logic and test assertions: use Session (and confirm your client actually carries the timestamp) or Strong.

High-frequency online recall: choose Bounded according to business tolerance for staleness.

Do not use Eventual consistency in delete-sensitive or write-verification scenarios: its water level is literally 1, not even promising "delete has taken effect." Reading deleted data is easy to misjudge as "delete failed," triggering a full round of write-pipeline investigation — the costliest misunderstanding under this semantic.

06 Boundaries

This article's criteria come from source code (commit 1306ae0816) and v3.0.x docs. Numeric anchors (5000 ms default grace window, 86400 s lag ceiling, 3 s stall judgment) are defaults; instances with modified configs don't apply.

Two points are deliberately not concluded: how each language SDK carries timestamps under Session level — client code not verified; quantitative relationship between timetick cadence and end-to-end visibility latency — not measured, so no "guaranteed visible within X seconds" promise is made. An external implementation special case: external tables' waitTSafe returns max timestamp directly to avoid low-water-level placeholders masking loaded deletes. These boundaries still evolve at the implementation layer; the same criteria against a new major version warrant re-checking the source code.