From Monolith to Distributed: Spring AI Embedding Product Semantic Recall System

Users searching for "office-appropriate shoes comfortable for all-day walking" would miss relevant products like "soft-sole commuter derby shoes" or "lightweight business loafers" with pure keyword search. Embedding solves this by encoding queries and product descriptions into a shared vector space for nearest-neighbor semantic retrieval. However, calling embeddingModel.embed(text) in a demo is only the start; production challenges include product updates, model upgrades, duplicate messages, vector store failures, and traffic growth.

Embedding Is Not a Complete Recommender System

Embedding excels at semantic recall but does not replace ranking. For "office walking shoes" vector recall is valuable; for exact-model queries like WH-1000XM6 or iPhone 17 Pro 256GB Black, keyword, synonym, term query, and structured filtering are more reliable. A production query pipeline should combine:

┌────────────────┐
              │ Query Normalize│
              └───────┬────────┘
                      │
        ┌─────────────┼─────────────────────┐
        ▼             ▼                     ▼
 Keyword / BM25   Vector ANN            Rule Recall
        │             │                     │
        └─────────────┼─────────────────────┘
                      ▼
           Candidate Merge
                      ▼
         Rerank + Business Rules
                      ▼
                    Result

Structured conditions such as category=男鞋, price<600, status=ON_SALE must be enforced during retrieval and reranking; semantic understanding handles "comfortable for all-day walking, suitable for commuting".

1. Start with a Single-Node Demo, But Don't Stop There

Local validation can begin with SimpleVectorStore: pick a few products, write text templates, test a dozen real queries to verify the model understands business semantics. It validates "worth doing" but cannot sustain long-term indexing. Production evolution is not "data grew, switch to Milvus" but stepwise by bottleneck:

Validate semantic value
  SimpleVectorStore
       │
       ▼
Persistence & ANN
  pgvector
       │
       ▼
Reliable indexing
  Outbox + MQ + Worker + Reindex
       │
       ▼
Query quality & performance
  Hybrid Search + Cache + HNSW tuning
       │
       ▼
After clear bottleneck
  Milvus / OpenSearch / Qdrant

Migration triggers: at target recall, current infrastructure cannot meet P99/QPS, or rebuild window, write throughput, scaling, and HA become significant costs.

2. Design Product Text First, Then Choose Model

Product vector quality is primarily determined by input text. Avoid naive field concatenation like 男鞋 Example 软底 通勤 防滑. Instead, explicitly label fields so the model knows field semantics:

@Component
public class ProductEmbeddingTextBuilder {

public static final String TEMPLATE_VERSION = "product-template-v2";

public String build(Product product) {
return """
    商品名称：%s
    品类：%s
    品牌：%s
    核心特征：%s
    使用场景：%s
    商品描述：%s
    """.formatted(
  safe(product.getName()),
  safe(product.getCategoryName()),
  safe(product.getBrandName()),
  safeJoin(product.getFeatures()),
  safeJoin(product.getScenes()),
  safe(product.getDescription()));
}

private String safe(String value) {
return value == null ? "" : value;
}

private String safeJoin(Collection<String> values) {
return values == null ? "" : values.stream()
  .filter(Objects::nonNull)
  .map(String::trim)
  .filter(value -> !value.isEmpty())
  .collect(Collectors.joining("、"));
}
}

This isn't formatting obsession. Writing name + description today and adding "scenes, selling points, review summaries" tomorrow changes the vector space's business meaning even if the model stays the same. Therefore, production systems must version four items together:

index_version     = product-semantic-v2
model_version     = text-embedding-3-small
dimensions        = 1536
template_version  = product-template-v2
distance          = cosine

Any change equals a new index release; old vectors must not mix with new ones. Spring AI 2.0.x manages versions via BOM; key starters:

<dependency>
  <groupId>org.springframework.ai</groupId>
  <artifactId>spring-ai-starter-model-openai</artifactId>
</dependency>
<dependency>
  <groupId>org.springframework.ai</groupId>
  <artifactId>spring-ai-starter-vector-store-pgvector</artifactId>
</dependency>

Corresponding configuration:

spring:
  ai:
    openai:
      api-key: ${OPENAI_API_KEY}
      embedding:
        model: ${EMBEDDING_MODEL}
        dimensions: ${EMBEDDING_DIMENSIONS:1536}
    vectorstore:
      pgvector:
        initialize-schema: true
        index-type: HNSW
        distance-type: COSINE_DISTANCE
        dimensions: ${EMBEDDING_DIMENSIONS:1536}

initialize-schema

only runs once; it won't auto-migrate vector(1536) to vector(1024). On dimension change, create a new table/collection, full rebuild, verify, then cut traffic. When building HNSW/IVFFlat indexes with pgvector's vector type, note the 2,000-dimension index limit.

3. How to Reliably Refresh Vectors After Product Updates

The naive synchronous approach:

Save product → Call Embedding API → Write VectorStore → Return HTTP response

This couples the product write path to network timeouts, 429 errors, and vendor outages. The correct pattern is asynchronous indexing with the product database as the single source of truth:

Same database transaction
Product Service ─────────────────────────────┐
      │                                      │
      ▼                                      ▼
 product table update              product_outbox insert
      │                                      │
      └──────────────────┬───────────────────┘
                         ▼
              Outbox Publisher
                         ▼
                          MQ
                         ▼
                 Embedding Worker
                         ▼
         Embedding Model → Vector Store

The Outbox Publisher uses outbox_id as message deduplication key, acquires records via lease or FOR UPDATE SKIP LOCKED, and marks published after send. It accepts "send succeeded but mark failed → redeliver": the goal is at-least-once delivery + consumer idempotency , not end-to-end exactly-once.

Index status cannot use only product_id as primary key, otherwise v1/v2 dual-write overwrites previous state. Recommended minimal model:

CREATE TABLE product_vector_index (
  product_id      BIGINT NOT NULL,
  index_version   VARCHAR(64) NOT NULL,
  source_version  BIGINT NOT NULL,
  model_version   VARCHAR(128) NOT NULL,
  template_version VARCHAR(64) NOT NULL,
  content_hash    VARCHAR(64) NOT NULL,
  index_status    VARCHAR(32) NOT NULL,
  retry_count     INT NOT NULL DEFAULT 0,
  indexed_at      TIMESTAMP,
  updated_at      TIMESTAMP NOT NULL,
  PRIMARY KEY (product_id, index_version)
);

Workers re-read the product on event receipt rather than trusting the event payload. Messages may duplicate, delay, or reorder; the product master's source_version represents current state.

V12 arrives first, V11 arrives later
      │
Worker both read current version from product master
      │
Stable doc ID upsert + conditional status update
      ▼
Eventually converges to latest product

Conditional update prevents stale tasks overwriting newer state:

UPDATE product_vector_index
SET source_version = :sourceVersion,
    content_hash = :contentHash,
    index_status = 'SUCCESS',
    indexed_at = now()
WHERE product_id = :productId
AND index_version = :indexVersion
AND source_version <= :sourceVersion;

Skip embedding when content hash, model version, and template version are unchanged. 429, 503, and timeouts enter bounded exponential backoff; dimension mismatch, invalid fields, and config errors fail fast to a compensation queue. Off-shelf and deletion also emit events: ON_SALE upsert, OFF_SALE/DELETED delete or disable, preventing delisted products from appearing.

4. Query Time: Coexisting Vector Recall with Keyword Search

Minimal Spring AI semantic recall implementation with explicit structured filters and business status:

public List<Document> vectorRecall(String query, int topK, Long categoryId) {
  FilterExpressionBuilder f = new FilterExpressionBuilder();
  var filter = categoryId == null
    ? f.eq("status", "ON_SALE").build()
    : f.and(f.eq("status", "ON_SALE"),
            f.eq("categoryId", categoryId)).build();

  return vectorStore.similaritySearch(SearchRequest.builder()
    .query(query)
    .topK(topK)
    .similarityThreshold(0.0)
    .filterExpression(filter)
    .build());
}

similarityThreshold(0.0)

isn't "never filter"; it avoids arbitrary thresholds at launch. After preparing a labeled Query–Product set:

Evaluate recall and ranking with Recall@K, MRR, NDCG.

Use

threshold → Precision / Coverage / Zero-result rate / Conversion

curves to determine cutoff.

Evaluate categories (apparel, electronics, books) separately; don't assume same score means same relevance.

Keyword and vector each take Top 200, deduplicate, then fuse. The simplest production method is Reciprocal Rank Fusion (RRF):

finalScore = 1 / (k + bm25Rank) + 1 / (k + vectorRank)

RRF avoids forcing BM25 scores and cosine similarity onto a common scale. Later, sales, inventory, price, CTR, brand score, and user preference feed into the ranking model.

Cache in layers: cache query embeddings and shared candidate IDs, not cross-user final results. Cache key must include

modelVersion + templateVersion + indexVersion + normalizedQuery + filters

. Final results still require online reranking by inventory, region, membership, and user profile.

When vector store is unavailable, fall back to keyword and popular products; when model is unavailable, serve from query embedding cache, else fall back to keyword. Degradation is not an exception branch but part of the query path.

5. pgvector HNSW: The Real Difficulty Is Filtering and Trade-offs

HNSW parameters m, ef_construction, ef_search control graph connectivity, build quality, and query candidate width. Increasing ef_search typically improves recall but raises latency; no "universal parameters" exist.

BEGIN;
SET LOCAL hnsw.ef_search = 100;
SELECT ...;
COMMIT;

SET LOCAL

applies only to the current transaction and connection. If JDBC runs in auto-commit or the SET and SELECT execute separately, the parameter won't affect the query. For PostgreSQL-specific tuning, wrap native JDBC operations in a single transaction rather than assuming generic VectorStore passes the setting.

Another overlooked issue: filtering. Approximate indexes may scan ANN candidates first, then apply filters like categoryId, brandId, tenantId. When filter selectivity is 10%, ef_search=40 yields only ~4 candidates on average, causing insufficient TopK and recall drop.

Low filter selectivity   → filter column B-tree/multi-column index, exact search if needed
Few stable filter values → partial HNSW index
Many tenants/categories  → partitioning, separate tables or collections
ANN + filter           → iterative scan, monitor scan cost vs TopK completeness

Therefore, load testing must cover real query distribution: hot short queries, long natural language, model numbers, categories, multi-condition filters. For each ef_search record Recall@K, P50/P95/P99, CPU, IOPS, QPS, zero-result rate, and TopK completeness.

6. Migrating from pgvector to Milvus: Prove Need First, Then Switch Safely

Don't use "10 million vectors must migrate" as a rule. 20M vectors + 20 QPS and 3M vectors + 5000 QPS are completely different problems.

Once target-quality P99/QPS, write throughput, rebuild window, scaling, or fault domains prove PostgreSQL is the bottleneck, migrate via dual-version release, not downtime SDK swap:

┌───────────── pgvector (old) ─────────────┐
Product Event ──┤                                          ├── Shadow Read
                └───────────── Milvus (new) ───────────────┘
                                     │
                          Backfill + incremental dual-write
                                     │
                    1% → 10% → 50% → 100% read traffic
                                     │
                              keep rollback switch

Shadow Read does not return new-store results to users; it only logs latency, error rate, zero-result rate, filter-complete TopK rate, and TopK overlap. Overlap only diagnoses implementation differences, not quality proof. Before ramping traffic, verify:

Data coverage, content hash, delete status consistency
Fixed labeled set Recall@K, NDCG@K not below baseline
Real filter conditions produce complete results
Online P95/P99, error rate, fallback rate meet SLO
Old read path can be restored instantly

Model or template upgrades follow the identical process. Full reindex must be a routine capability: scan by product ID range, rate-limitable, pausable/resumable, with checkpoints and failure compensation — not a one-off dangerous manual script.

7. When Is It Production-Grade?

Validate with explicit targets, not component count:

Search availability            ≥ 99.95%
Vector-search latency P95      < 80 ms
Full recommendation P95        < 150 ms
Full recommendation P99        < 300 ms
Vector Recall@100              ≥ business quality floor
Vector empty rate              < 1%
Fallback rate                  < 0.5%

Simultaneously observe four signal categories:

Embedding  requests, latency, 429, errors, tokens & cost
Index      pending, failures, retries, backlog time, version coverage
Search     latency, zero-results, TopK completeness, fallback rate, cache hit rate
Quality    Recall@K, NDCG@K, CTR, CVR, zero-result rate

CTR/CVR must be interpreted with position bias, inventory, and experiment buckets — not equated directly to recall quality. Only when latency and resources persistently exceed budget and further lowering ANN parameters would breach quality guardrails should the next infrastructure upgrade be pursued.

Pre-Release Checklist

Index contract includes model, dimension, distance, template, and index version.

Index status table uses (product_id, index_version); supports dual-version parallelism.

Product transaction writes Outbox; Publisher replayable; Worker idempotent with version-conditional updates.

Covers create, update, off-shelf, delete, and recoverable full rebuild.

Keyword, vector, and rule recall unify into ranking and business rules.

Filter conditions have real load tests with correct ANN/exact search strategy chosen.

Cache keys carry index version; no sharing of personalized final results.

Degradation paths exist for vector store, model, and cache unavailability.

Model/vector store migration supports backfill, shadow, canary, and one-click rollback.

The value of embedding is not dropping an API call into a project, but turning semantic understanding into a governable, measurable, recoverable recall infrastructure. Only when models are upgradable, indexes rebuildable, messages repeatable, services degradable, and quality measurable does it truly possess production value.