This article explains the principles and practical implementation of vector retrieval in StarRocks, covering approximate nearest‑neighbor algorithms, index design, query planning, performance optimizations, real‑world case studies, and future challenges, showing how query latency dropped from 15 seconds to 2 seconds while cutting costs to a third.
Redis has unveiled a multi‑threaded query engine that dramatically increases query throughput and lowers latency for vector similarity searches, offering up to 16× performance gains and enabling real‑time Retrieval‑Augmented Generation (RAG) workloads in generative AI applications.
Bilibili’s large‑scale recall system separates online processing into a two‑tier merge service and an index service, supports multi‑channel text, item‑to‑item and vector indexes with real‑time updates, uses horizontal sharding, robust CI/CD, monitoring and degradation mechanisms, and is being extended toward model‑based recall and greater automation.
This article introduces Easysearch, a lightweight, secure, Elasticsearch‑compatible search database, outlines its core features and applicable scenarios, and provides a step‑by‑step CentOS 7 deployment guide together with practical verification commands and future development considerations.
This article explains how OpenSearch's GPU‑accelerated vector search leverages parallel graph algorithms to achieve up to tenfold speed improvements over CPU solutions, detailing ANNS techniques, performance benchmarks, and practical GPU specifications for high‑QPS AI applications.
Redis Labs announces the return of its founder Antirez, who shares his reasons for rejoining, discusses the recent licensing shift, reflects on his past projects, and outlines future plans for AI integration, vector indexing, and community engagement within the Redis ecosystem.
This article explores how modern data technologies—from traditional relational databases through NoSQL to vector‑based retrieval and RAG—demystify data, address the 4V challenges, and enable applications to adopt flexible, scalable, and AI‑driven data paradigms.
The article explains how Retrieval‑Augmented Generation (RAG) outperforms direct LLM inference by enabling real‑time knowledge updates and lower costs, and demonstrates a practical multi‑modal RAG pipeline that uses Chinese‑CLIP for vector encoding, various chunking strategies, and Redis Search for fast vector storage and retrieval.
This article explores how modern data technologies—from relational databases and NoSQL to vector databases and AI‑driven retrieval—address the 4V challenges of volume, velocity, variety, and value, enabling polyglot persistence, semantic embeddings, and retrieval‑augmented generation for next‑generation applications.
This article analyzes a GPU‑based approximate nearest neighbor (ANN) retrieval solution built on NVIDIA's RAFT library, detailing algorithm selection, offline indexing tricks, batch online search design, performance results on a 25‑million‑vector workload, and cost‑saving implications for large‑scale search services.
This article examines how Baidu and NVIDIA leveraged the open‑source RAFT library to build a GPU‑accelerated approximate nearest neighbor (ANN) retrieval system, detailing algorithm choices, offline indexing, online batch processing, performance results, and practical guidelines for deploying ANN on GPUs.
The article traces the progression from the human DIKW information hierarchy to its computer‑world counterpart, illustrates how a homestay platform’s data architecture evolves through relational, NoSQL, search, and data‑warehouse layers, and introduces the next‑generation distributed Data Warebase that unifies structured, semi‑structured, and vectorized knowledge to meet modern AI‑driven business demands.
This article surveys the evolution of modern data technologies—from the 4V characteristics of big data and the limitations of traditional relational databases, through the rise of NoSQL and polyglot persistence, to embedding‑driven vector search, hybrid retrieval and RAG, illustrating how each paradigm frees applications from data constraints.
Redis has launched an enhanced, multi‑threaded query engine that dramatically increases throughput and reduces latency for vector similarity searches, enabling vertical scaling and better support for real‑time RAG applications while maintaining sub‑10 ms response times.
This guide walks entry‑level developers through building a logistics‑focused QA bot by first embedding documents for vector similarity search, then adding retrieval‑augmented generation, fine‑tuning a small model, integrating hybrid checks, and optimizing deployment with feedback loops to achieve fast, accurate, out‑of‑scope‑aware answers.
MongoDB 8.0 launches with up to 36% higher read throughput, 56% faster batch writes, 200% faster time‑series processing, 50‑fold faster sharding distribution, quantized vector search, and queryable encryption, delivering significant performance, cost, and security improvements for modern workloads.
This article explains how Elasticsearch’s advanced search capabilities—including vector and semantic search, hardware acceleration, hybrid retrieval, model re‑ranking, multi‑vector support, and integrated security—enable robust RAG implementations and outlines future directions such as a new compute engine, stronger vector engines, and cloud‑native serverless deployment.
This article explores the evolution of data lakes for AI, discusses the challenges of AI-era data management, introduces Apache Iceberg and its architecture, demonstrates PyIceberg-based AI training and inference pipelines, and presents vector table designs with LSH indexing and performance optimizations.
Redis has launched a multi‑threaded query engine that vertically scales its in‑memory database, dramatically increasing query throughput and lowering latency for vector similarity searches, thereby addressing the performance demands of real‑time retrieval‑augmented generation in generative AI applications.
This guide explains how to query a Microsoft GraphRAG knowledge graph using the official CLI, API, and a custom Neo4j implementation, covering both local and global retrieval modes, vector index creation, Cypher query customization, and integration with LangChain for end‑to‑end RAG pipelines.
This article reviews two decades of database research, analyzing the rise and decline of various data models—from hierarchical and relational to NoSQL, vector, and graph databases—while highlighting how AI, cloud, and hardware advances are reshaping DBMS architecture and predicting which approaches will dominate tomorrow’s data landscape.
This case study details how ShiHuo, an e‑commerce recommendation platform, overcame rapid product growth, cluster instability, and high operational overhead by adopting Alibaba Cloud’s fully managed Milvus vector search service, achieving higher performance, better availability, and reduced management costs.
This article describes how to combine Elasticsearch’s keyword matching with Milvus’s vector‑based semantic search to build a scalable document search service, covering data preprocessing, architecture, query handling, custom scoring, DSL configuration, and result merging.
This article examines how the big‑data AI development paradigm has shifted from model‑centric to data‑centric workflows, outlines the challenges of integrating data and AI teams, and details Alibaba Cloud’s end‑to‑end, serverless big‑data platform—including MaxCompute, Hologres, MaxFrame, Object Table, and vector search—designed to accelerate large‑scale AI applications.
This article presents a comprehensive overview of modern Data+AI data lake challenges and solutions, covering the evolution of data lakes, an introduction to Apache Iceberg, practical use of PyIceberg for AI training and inference pipelines, and advanced vector table and indexing techniques for efficient similarity search.
This article explores two advanced retrieval paradigms—Fusion Retrieval, which merges results from multiple retrievers using re‑ranking, and Recursive Retrieval, which builds hierarchical chunk‑to‑chunk or chunk‑to‑retriever links—to boost the quality and flexibility of Retrieval‑Augmented Generation pipelines.
This guide explains how to combine traditional keyword search and vector‑based semantic search into a hybrid retrieval system using Volcengine Cloud Search, covering the underlying concepts, required components, workflow steps, and a complete Python implementation for an image search application.
This article examines how vector search using Milvus handles sentence and paragraph queries compared to traditional Elasticsearch, demonstrating the advantages of embedding‑based semantic matching for document search scenarios through practical experiments and visual results.
Oracle’s latest release, Database 23ai, upgrades its 23c platform with AI-driven vector search, RAG capabilities, and enhanced JSON and graph querying, positioning the database as a unified, secure, and scalable solution for handling structured, semi‑structured, and unstructured data across cloud and on‑premises environments.
With the rise of LLMs, vector search and vector databases have become essential for extending model memory, and this article explains the principles, algorithms, design choices, implementation details, and performance results of ByteHouse’s cloud‑native vector retrieval engine.
MyScaleDB is a high‑performance, cost‑effective SQL vector database built on ClickHouse that lets developers use familiar SQL to store, index, and search billions of vectors alongside structured data, offering fast, accurate AI retrieval and seamless integration with existing tools.
Meituan Waimai’s search team built a GPU‑accelerated, mixed vector‑and‑scalar retrieval engine that supports billions of items, achieving over 99% recall and up to 89% latency reduction by combining pre‑filtering, optimized data layouts, multi‑GPU parallelism, and FP16 precision.
This article explains how traditional keyword search with Elasticsearch often yields inaccurate or incomplete results for document retrieval, introduces vectorization and semantic search using NLP embeddings, and demonstrates a practical workflow that combines these techniques with the Milvus vector database to achieve more accurate and efficient document search.
DingoDB is a multi‑modal vector database that unifies storage and analysis of structured, semi‑structured and unstructured data through a Raft‑based distributed architecture, offering MySQL‑compatible SQL, high‑performance APIs, automatic sharding, real‑time index optimization, and hybrid scalar‑vector queries for enterprise knowledge bases, LLM memory, and real‑time decision‑making.
Google has introduced preview vector search to its Cloud SQL for MySQL service, positioning it ahead of Oracle, while industry analysts note the growing importance of vector capabilities for generative AI applications across major database platforms.
Sun Ke’s keynote at the 2023 Baidu Cloud Intelligence Conference explains how AI‑native development has shifted from model selection to building practical applications, and introduces Baidu Cloud AppBuilder—a three‑layer, low‑code‑and‑code platform that provides multimodal, LLM, and infrastructure services, enabling rapid prototyping of solutions such as automated resume screening and interview preparation.
This article explores the rise of generative AI, outlines popular models like ChatGPT, DALL‑E, and Google Bard, examines their limitations, and then delves into Elasticsearch’s Relevance Engine and vector capabilities, demonstrating how to store, index, and query dense embeddings with practical code examples.
The 7th Tongcheng Travel Big Data Technology Salon in Beijing featured a series of expert talks covering StarRocks architecture evolution, lake‑house solutions with Paimon, Iceberg real‑time upsert, Data+AI for travel recommendation, vector retrieval in AI, JD Logistics real‑time computing governance, and multi‑task hotel ranking modeling, providing deep technical insights and future roadmaps.
Learn how to generate text embeddings with OpenAI's ChatGPT API, store them in Redis' vector database, and perform efficient similarity searches using PHP, covering installation, API calls, index creation, vector storage, KNN querying, and practical code examples.
The article explains how an e‑commerce risk‑control team leverages OpenAI's CLIP model to generate image and text embeddings and stores them in the Milvus cloud‑native vector database to enable fast, scalable similarity searches for compliance verification and risk detection.
This guide explains how to implement reverse image search by extracting visual features with a multilingual CLIP model, storing the vectors in Elasticsearch 8.x, and using its k‑NN plugin to retrieve similar images, covering architecture, tools, code snippets, and results.
This article explains how Alibaba Cloud OpenSearch leverages vector retrieval, engineering and algorithmic optimizations, heterogeneous CPU‑GPU computing, and dense‑sparse hybrid memory to deliver billion‑scale, high‑throughput search performance and enable conversational AI use cases such as intelligent Q&A and SmartArXiv.
This article introduces Elasticsearch, explains its hot‑warm‑cold lifecycle management, demonstrates vector search and built‑in NLP capabilities, and describes how the 360 Zhihui Cloud Platform integrates these features with practical test cases and new visualization tools.
Alibaba Cloud unveiled AI-driven upgrades to its OpenSearch and Elasticsearch services, highlighting LLM‑based conversational search, three‑fold vector retrieval speed gains, and up to 70% cost reductions through serverless architectures and extensive performance optimizations.
This article explains the principles of vector databases, outlines the engineering practices of Baidu Intelligent Cloud BES for large‑scale vector retrieval, discusses optimization techniques such as HNSW, IVF and filter integration, and presents real‑world AI use cases and future development directions.
This guide details a complete end‑to‑end solution for constructing a large‑language‑model knowledge‑base chatbot on Alibaba Cloud, covering background, modular architecture, vector database selection, text preprocessing, embedding models, LLM fine‑tuning, prompt engineering, deployment with PAI‑EAS and BladeLLM, and real‑world results.
This article explains how vector search converts unstructured data such as speech, images, video, and text into high‑dimensional embeddings, explores common algorithms like Brute‑Force, ANN, and HNSW, and presents optimization techniques that dramatically improve recall and query‑per‑second performance for large‑scale AI retrieval systems.
This article introduces the evolution of embedding techniques, presents real‑world case studies of embedding‑based FAQ retrieval, explains evaluation metrics such as Recall and MRR, and compares the performance of a proprietary ZhongAn embedding model with OpenAI and Sentence‑BERT models on Chinese FAQ datasets.
Redis 7.2, the first Unified Redis Release, introduces extensive AI support, vector database capabilities, scalable search, server‑side Triggers and Functions, enhanced geospatial queries, performance‑boosted sorted sets, and the Redis Data Integration tool, while expanding client and protocol compatibility.
Redis 7.2, the first Unified Redis Release, introduces AI‑ready vector indexing, hybrid semantic search, scalable RAG support, server‑side Triggers and Functions, enhanced geospatial queries, and a preview of high‑performance searchable indexes, while expanding client library support and integrating Redis Data Integration for seamless enterprise data pipelines.
This article details Bilibili's transition from a purchased customer‑service platform to a self‑developed system, describing the background, architectural design, core modules such as intelligent QA, seat scheduling, workbench, permission management, the use of Faiss for vector search, and future explorations with large language models, highlighting the technical challenges and solutions across backend development and AI integration.
This article explains the limitations of large language models, introduces prompt engineering as a remedy, and provides a comprehensive guide to using the LangChain framework—including models, prompts, chains, agents, vector search, and practical Node.js code examples—to enable LLMs to interact with external tools and data sources.
This tutorial walks you through installing Elasticsearch 8.x, generating sentence embeddings with Python, creating a dense_vector index, bulk‑loading documents, and performing both exact and approximate k‑nearest‑neighbor queries using script_score and the built‑in knn API.
This article explains what embeddings are, how they capture semantic similarity, and how to use OpenAI's embedding API to transform a custom knowledge base into vector representations for efficient search, retrieval, and question‑answering.
This article explains how Cassandra’s newly added vector data type and ANN search capabilities empower AI developers to store, index, and query high‑dimensional embeddings at scale, enabling use cases such as image retrieval, recommendation, and large‑language‑model integration.
This article describes the design, algorithmic improvements, and engineering performance optimizations of Bilibili's massive video duplicate detection (collision) system, covering challenges of low‑edit‑degree reposts, two‑stage retrieval, self‑supervised feature extraction, GPU‑accelerated preprocessing, and the resulting gains in accuracy and throughput.
Xiaohongshu transformed its recommendation pipeline from daily to minute‑level updates by redesigning recall, ranking and feature‑joining components, deploying a base‑plus‑incremental training scheme, migrating Spark to Flink, rewriting services in C++, and optimizing RocksDB, which yielded over 10% longer dwell time, 15% more interactions and roughly 50% higher new‑note efficiency.
Alibaba’s advertising pipeline progressed from low‑dimensional quantization partitions to hierarchical tree indexes, then to graph‑based HNSW structures—including multi‑category, multi‑level graphs and a BlazeOp‑driven scoring service—dramatically boosting recall efficiency, scalability and maintainability while meeting strict latency constraints.
The Dolphin engine, built by Alibaba’s Data Engine team, combines Flink and Hologres to deliver ultra‑large‑scale OLAP, streaming, batch, and AI capabilities for real‑time advertising analytics, offering smart materialization, intelligent indexing, and vector recall while supporting millions of advertisers and petabyte‑level data.
This article describes the background, technical selection, architecture, deployment, performance tuning, and operational practices of a Milvus‑driven vector retrieval platform, including cloud‑native deployment, index choices, capacity planning, and real‑world application cases that improve recall latency and resource efficiency.
The author stress‑tested Milvus 2.1.4’s cloud‑native, micro‑service architecture—detailing its write and search paths, evaluating FLAT index performance across 100 K to 10 M 512‑dim vectors, uncovering scaling, scheduler, segment‑rebalance, and upgrade issues, and concluding the system is robust but benefits from graph‑based indexes and Helm‑driven scaling.
The gpdb‑faiss‑vector plugin embeds Facebook’s Faiss library into the Dolphin (Greenplum‑compatible) engine, exposing SQL functions for distributed, high‑performance approximate nearest‑neighbor vector retrieval with caching, parallel search, configurable indexes, and sub‑millisecond latency, enabling scalable recommendation and advertising workloads.
This article explains Huya Live's recommendation architecture, covering business background, system design, vector retrieval challenges and solutions with ScaNN, and the fine‑ranking pipeline, while highlighting performance optimizations, scalability, and future directions for their live‑streaming platform.
This article details the evolution of the ‘逛逛’ content community’s recommendation system, comparing the legacy rule‑based Hive workflow with a new algorithm‑driven architecture that leverages Elasticsearch, Redis, multi‑stage recall, coarse‑ and fine‑ranking, re‑ranking, exposure filtering, cold‑start handling, performance tuning, and future plans for vector‑based recall and platformization.
This tutorial explains what Faiss is, how to install it, construct various indexes such as IndexFlatL2, IndexIVFFlat, and IndexIVFPQ, and demonstrates code examples for building and querying vector similarity search pipelines while discussing speed‑accuracy trade‑offs.
This article explains Elasticsearch's vector search capabilities, detailing two approaches—script_score using dense_vector fields for exact similarity scoring and the experimental _knn_search for approximate nearest neighbor queries—along with data modeling examples, code snippets, performance considerations, and usage guidelines.
This article details NetEase Cloud Music's technical approach to building implicit user relationship chains—using SimHash, Item2Vec, and MetaPath2Vec embeddings, large‑scale vector search, and a unified service architecture—to address cold‑start challenges across multiple business scenarios.
Apache Lucene 9.1.0 introduces modular JAR organization, nearest‑neighbor vector search support, internal query syntax, a Spanish plural stem filter, significant throughput and speed gains for high‑dimensional vectors, faster count queries, and various other enhancements and test framework changes.
This article presents a comprehensive study of local vector retrieval optimization, covering memory‑based filtering techniques, Redis‑backed vector storage designs, and various sorting algorithms—including radix and heap‑based approaches—to achieve lower latency and higher throughput for large‑scale ad recommendation systems.
This article presents a comprehensive overview of Huya Live's recommendation system, covering business background, system architecture, vector retrieval techniques, ranking pipeline, technical challenges, implementation details, and future outlook, highlighting scalability and performance optimizations.
The inaugural BigANN competition, organized by NeurIPS, showcased large‑scale ANN research, and Baidu's self‑developed PUCK algorithm secured top scores across all four tracks by leveraging multi‑layer quantization, two‑level inverted indexing, and extensive system‑level optimizations.
This tutorial walks through constructing a movie recommendation engine by downloading a dataset, cleaning and de‑duplicating entries, encoding plot summaries into vectors with transformer models, and performing nearest‑neighbor searches using scikit‑learn, while handling misspellings with Levenshtein distance.
Apache Lucene 9.0, a high‑performance Java full‑text search library, introduces high‑dimensional vector indexing, new language analyzers, faster faceting and sorting, updated file formats, and several performance optimizations, providing developers with a richer, more efficient search toolkit.
The article describes the design, challenges, architecture, key features, algorithm optimizations, and future roadmap of Kuaishou's Starry vector retrieval platform, which delivers high‑performance, high‑reliability, and easy‑to‑use large‑scale ANN search for diverse business scenarios.
iQIYI’s engineering team built an online vector‑recall service on Milvus, wrapping it with a Dubbo‑gRPC interface to serve 6 M 64‑dimensional embeddings at roughly 3 k QPS and 20 ms p99 latency, integrating query‑embedding generation, simplifying recommendation pipelines, and demonstrating the performance and operational advantages of a platformized ANN‑based recall layer.
The article explains why traditional text‑based search engines like ElasticSearch struggle with modern multimodal data, introduces vector databases that store implicit semantic embeddings, and proposes a generalized search architecture that decouples data‑to‑vector mapping from the engine while leveraging clustering or graph indexes for similarity search.
This article describes how Dingxiangyuan's algorithm team adopted Milvus for distributed vector indexing to improve semantic search in their community forum, detailing the background, retrieval workflow, various embedding models—including Bi‑Encoder, Spherical Embedding, and Knowledge Embedding—and summarizing the benefits and future applications.
Baidu’s rich‑media retrieval system combines CNN‑based feature extraction with an Approximate Nearest Neighbor engine called GNOIMI, employing hierarchical clustering, product quantization, and optimized indexing to achieve sub‑millisecond search over billions of images, videos and audio, supporting anti‑spam, recommendation and risk‑control across dozens of services.
Alibaba’s Damo Academy unveils Proxima, a high‑performance vector search engine that powers e‑commerce, video, and payment services, detailing its core capabilities, large‑scale indexing, distributed construction, real‑time updates, and challenges such as algorithm diversity, scalability, and multi‑modal retrieval.
The article describes the design, architecture, and key components of a vector search platform built on Faiss that supports full‑index construction, incremental and distributed indexing, online retrieval, city‑level search, and vector update/delete operations to meet large‑scale AI application needs.
This article describes the two‑generation evolution of Youpai Image Manager's billion‑scale image search system, explaining the mathematical representation of images, the limitations of MD5, the first‑generation pHash‑ElasticSearch solution, and the second‑generation CNN‑Milvus approach for robust, large‑scale visual similarity search.
This article explains the fundamentals of building an image‑by‑image search system, covering image feature extraction methods such as hashing, traditional descriptors, CNN‑based vectors, and the use of vector search engines like Milvus for similarity retrieval.
This article details the 360 display advertising system's architecture and the progressive evolution of its recall module, covering business overview, overall pipeline, various recall strategies—including Boolean, vectorized, and deep‑tree approaches—and the performance optimizations applied to meet real‑time constraints.
This article details how the MX recommendation team rebuilt their Faiss‑based similarity search service using gRPC, covering service selection, multi‑type vector handling, dynamic index updates, deployment strategies, and performance gains that doubled QPS and cut latency by two‑thirds.
This article introduces the background of vector representations in machine learning, explains the K‑Nearest Neighbors algorithm and its key parameters, reviews traditional tree‑based and modern high‑performance search solutions, and demonstrates how FAISS can achieve microsecond‑level KNN queries on large‑scale high‑dimensional data.
The article explains how to replace keyword matching with deep‑learning document embeddings in Elasticsearch by applying PCA dimensionality reduction, indexing vectors using Lucene’s KD‑tree structures via a custom plugin, and leveraging FAISS‑style nearest‑neighbour techniques to achieve fast, semantically aware similarity search.
Xianyu’s personalized recommendation system struggles with tiny, fast‑turnover item pools because traditional X2I matrices provide insufficient recall, so the team introduced pool‑specific pre‑filtering, high‑dimensional vector search, and a real‑time search‑engine recall, the latter boosting clicks by 14 % and transactions by 0.14 %.
