Vector Database Operations: Milvus vs Qdrant Deployment, Scaling & Troubleshooting

Introduction: Why Benchmarks Mislead

Vector database selection discussions often focus on QPS benchmark tables, but real operational challenges involve scaling to 200M vectors, disk-full incidents at 3 AM, index format changes during upgrades, query node failures, and cross‑product migration consistency. Milvus and Qdrant have fundamentally different architectures: Milvus uses storage‑compute separation with multiple microservices (etcd, MinIO/S3, Pulsar/Kafka/Woodpecker, proxy, querynode, datanode, indexnode), enabling billion‑scale but increasing operational complexity. Qdrant is a single Rust binary using Raft for consensus and manual shard management, offering simpler deployment and strong single‑node performance but requiring upfront shard planning and explicit data migration during scaling.

Version Baselines (as of Oct 2026)

Milvus v3.0.4 (2026‑10‑26) – storage layer rewritten to Loon manifest format (Avro, versioned, transactional).

Milvus v2.6.26 (2026‑10‑26) – LTS branch maintained in parallel with 3.0.

Qdrant v1.20.0 (2026‑10‑22) – latest stable.

Client libraries: pymilvus 2.6.x, qdrant‑client 1.20.x (match server major versions).

Critical: Milvus 3.0 is not a drop‑in upgrade; it changes on‑disk layout (new _metadata/manifest-{version}.avro, _data/, _delta/, _stats/, _index/ directories). Old code cannot read new format. Upgrade requires compaction‑driven migration with version gating. Recommendation: stay on 2.6.x LTS for existing clusters; evaluate 3.0 only for new clusters with a full migration plan.

Architecture & Failure Domains

Milvus Components

Nine component types, each a distinct failure domain (ordered by impact): etcd > object storage > message queue > coordinators (rootcoord, datacoord, querycoord) > worker nodes (querynode, datanode, indexnode) > proxy. High‑availability investment must follow this order: etcd 3‑node cluster, distributed object storage, multi‑broker MQ.

Qdrant Components

Single qdrant process. Cluster adds Raft consensus group (cluster topology + collection metadata), shards (fixed at creation via shard_number), and replicas (adjusted via replicate_shard / drop_replica). Failure domains: node‑level (replicas absorb loss) and Raft‑level (majority loss = read‑only/unavailable). Ports: 6333 (REST), 6334 (gRPC), 6335 (P2P – never expose publicly ).

Indexing, Quantization & Capacity Planning

Both support HNSW. Milvus offers FLAT, IVF_FLAT, IVF_SQ8, HNSW, DISKANN with varying memory/speed trade‑offs. Qdrant uses HNSW only; memory pressure addressed via quantization: scalar (int8, ~1/4 memory, <1% recall loss), binary (1/32 memory, needs oversampling/rescoring), and TurboQuant (v1.20, verify on own data).

Capacity formula: raw vectors = count × dim × 4 bytes. HNSW overhead ≈ 0.3‑0.5× raw. Total memory ≈ (raw + index) × replicas × 1.2 headroom. Example: 100M × 768‑dim = 307 GB raw + 123 GB index = 430 GB; ×1.2 = 516 GB. With int8 quantization: raw → 77 GB, total ≈ 240 GB (fits 256 GB node). Quantization is the line between single‑node and distributed deployment.

Consistency Models

Milvus: Strong (write visible after flush), Bounded (default, few seconds), Session (same client sees own writes), Eventually (fastest). Qdrant: write waits for WAL ( wait=true), read consistency via consistency param (all/majority/quorum). Operational tip: “Document just inserted but not searchable” is 90% a consistency level misconfiguration. Temporarily switch to Strong to confirm.

Selection Decision Matrix

Key questions: 3‑year data volume? Backup RTO? Scaling model (auto‑rebalance vs manual shard moves)? Upgrade path (irreversible storage migrations)? Team troubleshooting ability (9 components vs 3 failure points)?

Decision table highlights: <50M vectors → Qdrant single node; multi‑tenant SaaS 10M‑200M → Qdrant cluster with shard_key for tenant isolation; >500M → Milvus distributed; existing data lake → Milvus 3.0 External Collection (zero‑copy Iceberg); strong consistency → both support but throughput drops; edge/embedded → Qdrant single binary; minimal ops → Qdrant unless data exceeds single‑node capacity and manual shard moves are unacceptable.

Anti‑pattern: “Start with Qdrant, migrate to Milvus later.” Feasible <50M; >100M migration window becomes days, dual‑write complexity high. If 3‑year forecast hits 100M+, start with Milvus.

Milvus Deployment

Standalone (Docker Compose)

Services: etcd, MinIO, milvus‑standalone (all components in one process). Verify docker compose ps shows all Up (healthy). Health endpoint: curl http://localhost:9091/healthz → OK. Smoke test with Python SDK ( pymilvus>=2.6,<2.7) creating collection, inserting 100 vectors, searching.

Distributed (Kubernetes Helm)

Key values-prod.yaml settings: fixed image tag (e.g., v2.6.26), proxy replicas (stateless), querynode replicas (primary scaling target), datanode, indexnode, etcd (3 nodes, 50Gi persistence), MinIO distributed (4 nodes, 500Gi), Pulsar (3 brokers/bookies). Install with

helm upgrade --install milvus milvus/milvus -n milvus -f values-prod.yaml --timeout 20m

. Verify all pods Running and READY 1/1. Readiness probes hit :9091/healthz.

Critical config in milvus.yaml (via ConfigMap): etcd endpoints, MinIO address/bucket, MQ type, segment maxSize (default 1024MB) and sealProportion (0.12 → seal at ~123MB). Changing config requires rolling restart of affected components.

Authentication & Least Privilege

Enable authorizationEnabled: true. Create dedicated user/role with collection‑level privileges (Search, Query, Insert, Delete) – avoid object_name="*".

Qdrant Deployment

Single Node (Docker)

Mount /qdrant/storage (data+WAL) and /qdrant/snapshots (separate volume to avoid disk‑full crashes). Disable telemetry, pass API key via --env-file. Health: /healthz (liveness), /readyz (readiness – waits for WAL replay). Create collection with named vector, shard_number=6 (divisible by 1/2/3/6 nodes), replication_factor=2, scalar quantization ( int8, quantile=0.99, always_ram=true), indexing_threshold=20000. Write with wait=true for immediate visibility.

Three‑Node Cluster (Docker Compose / Kubernetes StatefulSet)

Constraints: first node uses --uri, others --bootstrap to existing peer; 6335 P2P port internal only; new nodes start empty – no auto‑rebalance. Compose example uses internal network qdrant-net for 6335. Verify GET /cluster shows 3 peers, valid leader, pending_operations=0. K8s StatefulSet: headless service, command builds --uri / --bootstrap from pod ordinal. Scaling down requires moving shards off the target node ( replicate_shard then drop_replica), removing peer via DELETE /cluster/peer/{id}, then reducing replicas. Direct kubectl scale risks permanent data loss.

Scaling Operations

Milvus: Add Querynode = Pure Horizontal Scale

Trigger when: query P99 > SLO with CPU <70%, querynode memory >80% causing load failures, or replica count <2 under high QPS. Steps: record state, increase queryNode.replicas in values, helm upgrade, watch new pods, check querycoord logs for balance segment messages. If new nodes stay empty: check labels, auto‑balance config, or insufficient segments. Force rebalance via release_collection + load_collection (causes downtime). Validate with concurrent search benchmark (P50/P95/P99).

Qdrant: Adding Nodes ≠ Adding Capacity

New node joins cluster but receives no data until new shards created or manual move_shard / replicate_shard. Procedure: scale StatefulSet, verify peer count, inspect shard distribution ( GET /collections/{name}/cluster), migrate shards using move_shard (relocate) or replicate_shard (add replica). Method choices: snapshot (fast, needs 2× disk), stream_records (slow, rebuilds index), wal_delta (default, strong ordering). Verify each shard appears replication_factor times across nodes. Shard number is immutable – plan for final node count (e.g., 12 shards for up to 12 nodes).

Backup & Restore

Milvus: milvus-backup to Object Storage

Tool reads etcd metadata and object storage segments, writes logical backup to separate bucket. Config: Milvus address + backup user (Strong consistency), MinIO source bucket + separate backup bucket. Commands: create -n full_20261030 (all) or -c kb_chunks (single). Verify: segment count matches, object count matches, test restore in isolated env . Restore:

restore -n kb_20261030 --restore_collection_name kb_chunks_restore --restore_index

. RTO for 100M vectors ~4‑8 hours. Risks: backup reads impact production (run off‑peak), no built‑in CDC for incremental (need dual‑write or MQ‑side), protect backup bucket with versioning/object lock.

Qdrant: Snapshots (Single File, Fast Restore)

Collection snapshot: POST /collections/kb_chunks/snapshots?wait=true → returns .snapshot file (contains vectors, payload, index, quantization). Full cluster snapshot: POST /snapshots?wait=true. Download to external storage, verify SHA256. Restore: PUT /collections/kb_chunks/snapshots/recover with location (file://, http://, s3://). Pre‑checks: target collection existence, disk space ≥ 1.5× snapshot size, cluster mode triggers shard redistribution. Cleanup script with dry‑run, retention, checksum verification.

Monitoring & Alerting

Milvus Metrics (Prometheus on :9091/metrics)

Key families: milvus_proxy_req_count (QPS/error rate), milvus_proxy_*_latency, milvus_querynode_* (query/load/memory), milvus_datanode_* (write/flush lag), milvus_rootcoord_*, milvus_querycoord_*. Always list actual metrics first ( curl .../metrics | grep -oE '^milvus_[a-z_]+' | sort -u). Use container memory ( container_memory_working_set_bytes) not process metrics for OOM prediction.

Qdrant Metrics (:6333/metrics)

Fewer built‑in latency histograms; rely on client‑side or gateway logs for P99. Key metrics: app_info, collections_total, collections_vector_total, rest_responses_total, cluster_enabled. Observability weaker than Milvus – supplement with client instrumentation.

Alert Rules (Prometheus)

Examples: memory >85% for 15m, Milvus load failures increase >0 in 10m, datanode message queue >100 for 10m, Qdrant no Raft leader for 3m, shard state != Active for 10m, disk avail <15%. Validate rules with promtool check rules and confirm expressions return data in Graph UI.

Troubleshooting Scenarios

1. Query P99 Spikes (20ms → 800ms), P50 Stable

Check querycoord balance logs, querynode memory skew (3× difference = hotspot), datacoord compaction activity (IO contention), index build progress (new segments brute‑force). Correlate with container_fs_reads_bytes_total spike, bimodal latency histogram, GC pauses. Fixes: throttle compaction concurrency, wait for index build or add indexnodes, trigger rebalance via release/load.

2. Collection Stuck in Loading

Querycoord logs show load segment failed retry. Querynode errors: object storage timeout (test MinIO latency), local disk full, OOM (memory stair‑step then drop). Root causes: too many small segments (compaction needed), insufficient querynode memory (add nodes), slow object storage. Fix: compact (off‑peak, ensure 1× free space), add querynodes, upgrade object storage.

3. Write Succeeds but Read Missing / Write Fails

Missing reads: consistency level issue – test with consistency_level=Strong or Session. Write failures: WAL unavailable – check Pulsar/BookKeeper under‑replication, disk full. Write latency spike: frequent segment sealing – increase sealProportion (restart datacoord).

4. Qdrant Cluster No Leader / Node Join Fails

No leader: majority loss (2/3 down) or network partition. Check each node’s /cluster for term/leader consistency. Causes: process crash, 6335 blocked, disk full (check /data/qdrant and snapshots). Recover network, restart failed nodes, do not force restart during partition. Join failures: DNS resolve, 6335 connection refused, API key mismatch (v1.20+ enforces internal auth).

Risk, Rollback & Change Control

High‑risk ops table: drop_collection (irreversible, rename first), release_collection (query outage, need replicas), querynode scale‑down (ensure replicas≥2), Qdrant node removal (move shards first), shard_number change (rebuild collection), Milvus 2.6→3.0 (storage format, dual‑write migration), Qdrant upgrade (no downgrade, snapshot rollback), config changes (segment size, replication factor).

Change windows aligned to low query QPS (<20% peak). Milvus gray‑release via dual‑cluster dual‑write; Qdrant rolling node upgrade if protocol compatible. Rollback: Helm rollback for config (data not rolled back), Qdrant full rebuild from snapshot (verify checksum, two‑person approval).

Production Hardening Checklist

Milvus object storage: never single‑node MinIO – use distributed MinIO or cloud S3.

etcd: odd nodes, dedicated SSD, separate from K8s etcd.

Qdrant disks: local NVMe only (local‑path PVC), no network FS.

Qdrant shard_number: set once, aim for 2‑4× final node count, divisible by common node counts.

Milvus segment size: balance write vs load granularity; default usually fine.

Limit limit / offset – deep paging degrades to linear scan; use cursor pagination.

Backup restore drills quarterly in isolated env – measure RTO.

Read release notes for breaking changes (Milvus useLoonFFI, Qdrant internal auth default).

Add “canary query” monitoring: fixed query asserting >0 results to catch silent index/corruption issues.

Capacity planning includes replicas: Milvus memory × replicas, Qdrant disk × replicas.

Conclusion

Choose by three dimensions: Data scale dictates architecture (<50M → Qdrant, >100M → Milvus). Team capability sets floor (Milvus needs multi‑component expertise). Scaling model determines long‑term cost (Milvus auto‑rebalance vs Qdrant manual shard moves). When dimensions conflict, prioritize scale → scaling model → team skill. Migration cost grows exponentially – if 3‑year plan includes 10× growth, design for it from day one.