This article explores the evolution of data lakes for AI, discusses the challenges of AI-era data management, introduces Apache Iceberg and its architecture, demonstrates PyIceberg-based AI training and inference pipelines, and presents vector table designs with LSH indexing and performance optimizations.
Redis has launched a multi‑threaded query engine that vertically scales its in‑memory database, dramatically increasing query throughput and lowering latency for vector similarity searches, thereby addressing the performance demands of real‑time retrieval‑augmented generation in generative AI applications.
This guide explains how to query a Microsoft GraphRAG knowledge graph using the official CLI, API, and a custom Neo4j implementation, covering both local and global retrieval modes, vector index creation, Cypher query customization, and integration with LangChain for end‑to‑end RAG pipelines.
This article reviews two decades of database research, analyzing the rise and decline of various data models—from hierarchical and relational to NoSQL, vector, and graph databases—while highlighting how AI, cloud, and hardware advances are reshaping DBMS architecture and predicting which approaches will dominate tomorrow’s data landscape.
This case study details how ShiHuo, an e‑commerce recommendation platform, overcame rapid product growth, cluster instability, and high operational overhead by adopting Alibaba Cloud’s fully managed Milvus vector search service, achieving higher performance, better availability, and reduced management costs.
This article describes how to combine Elasticsearch’s keyword matching with Milvus’s vector‑based semantic search to build a scalable document search service, covering data preprocessing, architecture, query handling, custom scoring, DSL configuration, and result merging.
This article examines how the big‑data AI development paradigm has shifted from model‑centric to data‑centric workflows, outlines the challenges of integrating data and AI teams, and details Alibaba Cloud’s end‑to‑end, serverless big‑data platform—including MaxCompute, Hologres, MaxFrame, Object Table, and vector search—designed to accelerate large‑scale AI applications.
This article presents a comprehensive overview of modern Data+AI data lake challenges and solutions, covering the evolution of data lakes, an introduction to Apache Iceberg, practical use of PyIceberg for AI training and inference pipelines, and advanced vector table and indexing techniques for efficient similarity search.
This article explores two advanced retrieval paradigms—Fusion Retrieval, which merges results from multiple retrievers using re‑ranking, and Recursive Retrieval, which builds hierarchical chunk‑to‑chunk or chunk‑to‑retriever links—to boost the quality and flexibility of Retrieval‑Augmented Generation pipelines.
This guide explains how to combine traditional keyword search and vector‑based semantic search into a hybrid retrieval system using Volcengine Cloud Search, covering the underlying concepts, required components, workflow steps, and a complete Python implementation for an image search application.
This article examines how vector search using Milvus handles sentence and paragraph queries compared to traditional Elasticsearch, demonstrating the advantages of embedding‑based semantic matching for document search scenarios through practical experiments and visual results.
Oracle’s latest release, Database 23ai, upgrades its 23c platform with AI-driven vector search, RAG capabilities, and enhanced JSON and graph querying, positioning the database as a unified, secure, and scalable solution for handling structured, semi‑structured, and unstructured data across cloud and on‑premises environments.
With the rise of LLMs, vector search and vector databases have become essential for extending model memory, and this article explains the principles, algorithms, design choices, implementation details, and performance results of ByteHouse’s cloud‑native vector retrieval engine.
MyScaleDB is a high‑performance, cost‑effective SQL vector database built on ClickHouse that lets developers use familiar SQL to store, index, and search billions of vectors alongside structured data, offering fast, accurate AI retrieval and seamless integration with existing tools.
Meituan Waimai’s search team built a GPU‑accelerated, mixed vector‑and‑scalar retrieval engine that supports billions of items, achieving over 99% recall and up to 89% latency reduction by combining pre‑filtering, optimized data layouts, multi‑GPU parallelism, and FP16 precision.
This article explains how traditional keyword search with Elasticsearch often yields inaccurate or incomplete results for document retrieval, introduces vectorization and semantic search using NLP embeddings, and demonstrates a practical workflow that combines these techniques with the Milvus vector database to achieve more accurate and efficient document search.
DingoDB is a multi‑modal vector database that unifies storage and analysis of structured, semi‑structured and unstructured data through a Raft‑based distributed architecture, offering MySQL‑compatible SQL, high‑performance APIs, automatic sharding, real‑time index optimization, and hybrid scalar‑vector queries for enterprise knowledge bases, LLM memory, and real‑time decision‑making.
Google has introduced preview vector search to its Cloud SQL for MySQL service, positioning it ahead of Oracle, while industry analysts note the growing importance of vector capabilities for generative AI applications across major database platforms.
Sun Ke’s keynote at the 2023 Baidu Cloud Intelligence Conference explains how AI‑native development has shifted from model selection to building practical applications, and introduces Baidu Cloud AppBuilder—a three‑layer, low‑code‑and‑code platform that provides multimodal, LLM, and infrastructure services, enabling rapid prototyping of solutions such as automated resume screening and interview preparation.
This article explores the rise of generative AI, outlines popular models like ChatGPT, DALL‑E, and Google Bard, examines their limitations, and then delves into Elasticsearch’s Relevance Engine and vector capabilities, demonstrating how to store, index, and query dense embeddings with practical code examples.
The 7th Tongcheng Travel Big Data Technology Salon in Beijing featured a series of expert talks covering StarRocks architecture evolution, lake‑house solutions with Paimon, Iceberg real‑time upsert, Data+AI for travel recommendation, vector retrieval in AI, JD Logistics real‑time computing governance, and multi‑task hotel ranking modeling, providing deep technical insights and future roadmaps.
Learn how to generate text embeddings with OpenAI's ChatGPT API, store them in Redis' vector database, and perform efficient similarity searches using PHP, covering installation, API calls, index creation, vector storage, KNN querying, and practical code examples.
The article explains how an e‑commerce risk‑control team leverages OpenAI's CLIP model to generate image and text embeddings and stores them in the Milvus cloud‑native vector database to enable fast, scalable similarity searches for compliance verification and risk detection.
This guide explains how to implement reverse image search by extracting visual features with a multilingual CLIP model, storing the vectors in Elasticsearch 8.x, and using its k‑NN plugin to retrieve similar images, covering architecture, tools, code snippets, and results.
This article explains how Alibaba Cloud OpenSearch leverages vector retrieval, engineering and algorithmic optimizations, heterogeneous CPU‑GPU computing, and dense‑sparse hybrid memory to deliver billion‑scale, high‑throughput search performance and enable conversational AI use cases such as intelligent Q&A and SmartArXiv.
This article introduces Elasticsearch, explains its hot‑warm‑cold lifecycle management, demonstrates vector search and built‑in NLP capabilities, and describes how the 360 Zhihui Cloud Platform integrates these features with practical test cases and new visualization tools.
Alibaba Cloud unveiled AI-driven upgrades to its OpenSearch and Elasticsearch services, highlighting LLM‑based conversational search, three‑fold vector retrieval speed gains, and up to 70% cost reductions through serverless architectures and extensive performance optimizations.
This article explains the principles of vector databases, outlines the engineering practices of Baidu Intelligent Cloud BES for large‑scale vector retrieval, discusses optimization techniques such as HNSW, IVF and filter integration, and presents real‑world AI use cases and future development directions.
This guide details a complete end‑to‑end solution for constructing a large‑language‑model knowledge‑base chatbot on Alibaba Cloud, covering background, modular architecture, vector database selection, text preprocessing, embedding models, LLM fine‑tuning, prompt engineering, deployment with PAI‑EAS and BladeLLM, and real‑world results.
This article explains how vector search converts unstructured data such as speech, images, video, and text into high‑dimensional embeddings, explores common algorithms like Brute‑Force, ANN, and HNSW, and presents optimization techniques that dramatically improve recall and query‑per‑second performance for large‑scale AI retrieval systems.
This article introduces the evolution of embedding techniques, presents real‑world case studies of embedding‑based FAQ retrieval, explains evaluation metrics such as Recall and MRR, and compares the performance of a proprietary ZhongAn embedding model with OpenAI and Sentence‑BERT models on Chinese FAQ datasets.
Redis 7.2, the first Unified Redis Release, introduces extensive AI support, vector database capabilities, scalable search, server‑side Triggers and Functions, enhanced geospatial queries, performance‑boosted sorted sets, and the Redis Data Integration tool, while expanding client and protocol compatibility.
Redis 7.2, the first Unified Redis Release, introduces AI‑ready vector indexing, hybrid semantic search, scalable RAG support, server‑side Triggers and Functions, enhanced geospatial queries, and a preview of high‑performance searchable indexes, while expanding client library support and integrating Redis Data Integration for seamless enterprise data pipelines.
This article details Bilibili's transition from a purchased customer‑service platform to a self‑developed system, describing the background, architectural design, core modules such as intelligent QA, seat scheduling, workbench, permission management, the use of Faiss for vector search, and future explorations with large language models, highlighting the technical challenges and solutions across backend development and AI integration.
This article explains the limitations of large language models, introduces prompt engineering as a remedy, and provides a comprehensive guide to using the LangChain framework—including models, prompts, chains, agents, vector search, and practical Node.js code examples—to enable LLMs to interact with external tools and data sources.
This tutorial walks you through installing Elasticsearch 8.x, generating sentence embeddings with Python, creating a dense_vector index, bulk‑loading documents, and performing both exact and approximate k‑nearest‑neighbor queries using script_score and the built‑in knn API.
This article explains what embeddings are, how they capture semantic similarity, and how to use OpenAI's embedding API to transform a custom knowledge base into vector representations for efficient search, retrieval, and question‑answering.
This article explains how Cassandra’s newly added vector data type and ANN search capabilities empower AI developers to store, index, and query high‑dimensional embeddings at scale, enabling use cases such as image retrieval, recommendation, and large‑language‑model integration.
This article describes the design, algorithmic improvements, and engineering performance optimizations of Bilibili's massive video duplicate detection (collision) system, covering challenges of low‑edit‑degree reposts, two‑stage retrieval, self‑supervised feature extraction, GPU‑accelerated preprocessing, and the resulting gains in accuracy and throughput.
Xiaohongshu transformed its recommendation pipeline from daily to minute‑level updates by redesigning recall, ranking and feature‑joining components, deploying a base‑plus‑incremental training scheme, migrating Spark to Flink, rewriting services in C++, and optimizing RocksDB, which yielded over 10% longer dwell time, 15% more interactions and roughly 50% higher new‑note efficiency.
Model ServingReal-time TrainingRecommendation System
Alibaba’s advertising pipeline progressed from low‑dimensional quantization partitions to hierarchical tree indexes, then to graph‑based HNSW structures—including multi‑category, multi‑level graphs and a BlazeOp‑driven scoring service—dramatically boosting recall efficiency, scalability and maintainability while meeting strict latency constraints.
The Dolphin engine, built by Alibaba’s Data Engine team, combines Flink and Hologres to deliver ultra‑large‑scale OLAP, streaming, batch, and AI capabilities for real‑time advertising analytics, offering smart materialization, intelligent indexing, and vector recall while supporting millions of advertisers and petabyte‑level data.
This article describes the background, technical selection, architecture, deployment, performance tuning, and operational practices of a Milvus‑driven vector retrieval platform, including cloud‑native deployment, index choices, capacity planning, and real‑world application cases that improve recall latency and resource efficiency.
The author stress‑tested Milvus 2.1.4’s cloud‑native, micro‑service architecture—detailing its write and search paths, evaluating FLAT index performance across 100 K to 10 M 512‑dim vectors, uncovering scaling, scheduler, segment‑rebalance, and upgrade issues, and concluding the system is robust but benefits from graph‑based indexes and Helm‑driven scaling.
This article explains Huya Live's recommendation architecture, covering business background, system design, vector retrieval challenges and solutions with ScaNN, and the fine‑ranking pipeline, while highlighting performance optimizations, scalability, and future directions for their live‑streaming platform.
This article details the evolution of the ‘逛逛’ content community’s recommendation system, comparing the legacy rule‑based Hive workflow with a new algorithm‑driven architecture that leverages Elasticsearch, Redis, multi‑stage recall, coarse‑ and fine‑ranking, re‑ranking, exposure filtering, cold‑start handling, performance tuning, and future plans for vector‑based recall and platformization.
This tutorial explains what Faiss is, how to install it, construct various indexes such as IndexFlatL2, IndexIVFFlat, and IndexIVFPQ, and demonstrates code examples for building and querying vector similarity search pipelines while discussing speed‑accuracy trade‑offs.
This article explains Elasticsearch's vector search capabilities, detailing two approaches—script_score using dense_vector fields for exact similarity scoring and the experimental _knn_search for approximate nearest neighbor queries—along with data modeling examples, code snippets, performance considerations, and usage guidelines.
This article details NetEase Cloud Music's technical approach to building implicit user relationship chains—using SimHash, Item2Vec, and MetaPath2Vec embeddings, large‑scale vector search, and a unified service architecture—to address cold‑start challenges across multiple business scenarios.
Apache Lucene 9.1.0 introduces modular JAR organization, nearest‑neighbor vector search support, internal query syntax, a Spanish plural stem filter, significant throughput and speed gains for high‑dimensional vectors, faster count queries, and various other enhancements and test framework changes.
This article presents a comprehensive study of local vector retrieval optimization, covering memory‑based filtering techniques, Redis‑backed vector storage designs, and various sorting algorithms—including radix and heap‑based approaches—to achieve lower latency and higher throughput for large‑scale ad recommendation systems.
This article presents a comprehensive overview of Huya Live's recommendation system, covering business background, system architecture, vector retrieval techniques, ranking pipeline, technical challenges, implementation details, and future outlook, highlighting scalability and performance optimizations.
The inaugural BigANN competition, organized by NeurIPS, showcased large‑scale ANN research, and Baidu's self‑developed PUCK algorithm secured top scores across all four tracks by leveraging multi‑layer quantization, two‑level inverted indexing, and extensive system‑level optimizations.
This tutorial walks through constructing a movie recommendation engine by downloading a dataset, cleaning and de‑duplicating entries, encoding plot summaries into vectors with transformer models, and performing nearest‑neighbor searches using scikit‑learn, while handling misspellings with Levenshtein distance.
Apache Lucene 9.0, a high‑performance Java full‑text search library, introduces high‑dimensional vector indexing, new language analyzers, faster faceting and sorting, updated file formats, and several performance optimizations, providing developers with a richer, more efficient search toolkit.
The article describes the design, challenges, architecture, key features, algorithm optimizations, and future roadmap of Kuaishou's Starry vector retrieval platform, which delivers high‑performance, high‑reliability, and easy‑to‑use large‑scale ANN search for diverse business scenarios.
iQIYI’s engineering team built an online vector‑recall service on Milvus, wrapping it with a Dubbo‑gRPC interface to serve 6 M 64‑dimensional embeddings at roughly 3 k QPS and 20 ms p99 latency, integrating query‑embedding generation, simplifying recommendation pipelines, and demonstrating the performance and operational advantages of a platformized ANN‑based recall layer.
The article explains why traditional text‑based search engines like ElasticSearch struggle with modern multimodal data, introduces vector databases that store implicit semantic embeddings, and proposes a generalized search architecture that decouples data‑to‑vector mapping from the engine while leveraging clustering or graph indexes for similarity search.
This article describes how Dingxiangyuan's algorithm team adopted Milvus for distributed vector indexing to improve semantic search in their community forum, detailing the background, retrieval workflow, various embedding models—including Bi‑Encoder, Spherical Embedding, and Knowledge Embedding—and summarizing the benefits and future applications.
Baidu’s rich‑media retrieval system combines CNN‑based feature extraction with an Approximate Nearest Neighbor engine called GNOIMI, employing hierarchical clustering, product quantization, and optimized indexing to achieve sub‑millisecond search over billions of images, videos and audio, supporting anti‑spam, recommendation and risk‑control across dozens of services.
Alibaba’s Damo Academy unveils Proxima, a high‑performance vector search engine that powers e‑commerce, video, and payment services, detailing its core capabilities, large‑scale indexing, distributed construction, real‑time updates, and challenges such as algorithm diversity, scalability, and multi‑modal retrieval.
The article describes the design, architecture, and key components of a vector search platform built on Faiss that supports full‑index construction, incremental and distributed indexing, online retrieval, city‑level search, and vector update/delete operations to meet large‑scale AI application needs.
This article describes the two‑generation evolution of Youpai Image Manager's billion‑scale image search system, explaining the mathematical representation of images, the limitations of MD5, the first‑generation pHash‑ElasticSearch solution, and the second‑generation CNN‑Milvus approach for robust, large‑scale visual similarity search.
This article explains the fundamentals of building an image‑by‑image search system, covering image feature extraction methods such as hashing, traditional descriptors, CNN‑based vectors, and the use of vector search engines like Milvus for similarity retrieval.
This article details the 360 display advertising system's architecture and the progressive evolution of its recall module, covering business overview, overall pipeline, various recall strategies—including Boolean, vectorized, and deep‑tree approaches—and the performance optimizations applied to meet real‑time constraints.
This article details how the MX recommendation team rebuilt their Faiss‑based similarity search service using gRPC, covering service selection, multi‑type vector handling, dynamic index updates, deployment strategies, and performance gains that doubled QPS and cut latency by two‑thirds.
This article introduces the background of vector representations in machine learning, explains the K‑Nearest Neighbors algorithm and its key parameters, reviews traditional tree‑based and modern high‑performance search solutions, and demonstrates how FAISS can achieve microsecond‑level KNN queries on large‑scale high‑dimensional data.
The article explains how to replace keyword matching with deep‑learning document embeddings in Elasticsearch by applying PCA dimensionality reduction, indexing vectors using Lucene’s KD‑tree structures via a custom plugin, and leveraging FAISS‑style nearest‑neighbour techniques to achieve fast, semantically aware similarity search.
Xianyu’s personalized recommendation system struggles with tiny, fast‑turnover item pools because traditional X2I matrices provide insufficient recall, so the team introduced pool‑specific pre‑filtering, high‑dimensional vector search, and a real‑time search‑engine recall, the latter boosting clicks by 14 % and transactions by 0.14 %.