This article explains why traditional databases like MySQL struggle with petabyte‑scale queries, introduces Elasticsearch’s architecture and inverted‑index advantages, and details a practical pipeline using Hive, Canal, Otter, and a custom DTS service to achieve near‑real‑time indexing for massive data volumes.
JD.com’s order center faced massive query loads, prompting a migration from MySQL to Elasticsearch and a multi-stage evolution of its ES cluster—from a chaotic initial setup to isolated nodes, replica tuning, master‑slave dual clusters, and real‑time backup—culminating in a robust, high‑throughput, fault‑tolerant search architecture.
This guide explains how to define and use Elasticsearch index templates, manage index aliases, configure the _source field, and understand the roles of master, data, and coordinating nodes in a 7.8 cluster, with practical API examples and visual illustrations.
This article explains why MySQL queries become slow, how indexes work and fail, the impact of MDL locks, large‑table challenges, sharding and read‑write splitting strategies, then introduces Elasticsearch’s search capabilities and HBase’s column‑family storage, offering practical guidance for each technology.
This article examines common performance pitfalls in Elasticsearch—including slow queries, cluster architecture bottlenecks, and business‑scenario challenges—and provides practical guidance such as caching key fields, data pre‑heating, hot‑cold separation, avoiding joins, and using tribe nodes to improve accuracy and response time.
The article examines the strengths and weaknesses of combining HBase and Elasticsearch for massive data storage and retrieval, outlines three integration patterns and their challenges, and presents Alibaba Cloud's Lindorm Searchindex as a SQL‑driven, low‑cost, strongly consistent solution that simplifies development and improves performance.
This article explains how Elasticsearch processes queries across shards, identifies two key performance bottlenecks—query computation time and segment file I/O—and offers practical optimization strategies such as simplifying query logic, maximizing OS file cache usage, increasing memory, reducing stored data, and applying hot‑cold data separation.
This comprehensive guide walks through Elasticsearch fundamentals, including its architecture, core concepts like indices, shards, and replicas, the write and update processes, search workflow, consistency mechanisms, master election, performance tuning, and strategies for deep pagination and scroll searches.
This article examines JD Daojia's order fulfillment system, detailing the challenges of high‑volume prompt‑sound queries, the division of responsibilities among Redis, MySQL, and Elasticsearch, the adoption of Canal for asynchronous data replication, deployment practices with Kafka and Zookeeper, and the key operational lessons learned.
This article analyzes common causes of slow MySQL queries such as index misuse, MDL locks, and large‑table bottlenecks, then presents practical solutions like proper indexing, sharding, read/write splitting, and evaluates when to complement MySQL with ElasticSearch or HBase for better performance.
This guide walks through installing Apache SkyWalking, setting up an ElasticSearch storage cluster, configuring agents, launching services, and integrating DingTalk notifications for alarm handling, providing complete commands, configuration snippets, and troubleshooting tips.
This article examines the complementary strengths of HBase and Elasticsearch, outlines three integration patterns and their associated challenges, and introduces Alibaba Cloud's Lindorm Searchindex as a SQL‑driven, low‑cost solution that simplifies storage and full‑text search for massive data workloads.
This guide walks you through downloading and installing ElasticSearch, explains core concepts like indices, types, documents, and fields, demonstrates CRUD operations via RESTful APIs, shows advanced query techniques, and provides complete Java integration examples using Maven and Docker.
Elasticsearch routes client write requests to the primary shard, buffers documents in memory, logs operations to a translog, periodically refreshes to create segment files, and uses flush and merge processes to ensure data durability, optimize I/O, and consolidate segments for efficient search.
This article presents a detailed case study of moving a 2‑petabyte smart‑product data workload from Elasticsearch to MongoDB, covering business background, source cluster architecture, MongoDB design and resource planning, step‑by‑step performance optimizations, latency and cost comparisons, and guidance on when each database is appropriate.
This guide walks through the complete architecture and step‑by‑step deployment of a billion‑scale ELK logging system, covering Filebeat agents, Kafka buffering, Logstash processing, Elasticsearch indexing, and Kibana visualization, including configuration files, version details, and best‑practice tips for scaling and security.
This guide walks through configuring Spring Cloud microservices with Nacos service discovery, adding Zipkin for distributed tracing, and persisting trace data to Elasticsearch, including all necessary Maven dependencies, YAML settings, controller code, and verification steps.
This talk explains JD.com’s order‑fulfillment workflow, the underlying storage stack of Redis, MySQL and Elasticsearch, the challenges of high‑traffic alert‑sound queries, the adoption of Canal for data heterogeneity, and a 4R‑based complexity‑oriented design methodology for building resilient backend systems.
Learn step‑by‑step how to design and deploy a billion‑scale log collection and analysis platform using the ELK stack—Filebeat, Kafka, Logstash, Elasticsearch, and Kibana—covering architecture, configuration, installation, and best practices for high‑availability and performance.
This article explains Elasticsearch’s three pagination methods—From + size, search_after, and Scroll—detailing their definitions, code examples, advantages, disadvantages, and suitable scenarios, while also discussing max_result_window limits, PIT views, and best practices for handling large result sets.
This article provides a detailed overview of the ELK stack—including Elasticsearch, Logstash, Kibana, and Beats—explaining its components, why to use it for centralized log management, various deployment architectures, system tuning, security setup, and step‑by‑step installation and configuration commands for a production‑grade environment.
This article details 58.com’s journey with Elasticsearch, covering the challenges of disparate deployments, common problems like disk exhaustion and write slowdown, the governance and automation platform they built, development standards, service architecture, real‑world application cases, and future plans for version upgrades and intelligent diagnostics.
This article provides a comprehensive introduction to the ELK stack—including component descriptions, architectural diagrams, reasons for adoption, and step‑by‑step installation and configuration of Filebeat, Logstash, Elasticsearch, and Kibana on Linux, with optional Kafka integration for advanced pipelines.
This article explains how to construct near real‑time Elasticsearch indexes for petabyte‑level datasets by comparing MySQL limitations, describing Elasticsearch fundamentals, and detailing a pipeline that uses Hive, wide tables, MySQL binlog, Canal, and Otter to achieve second‑level index updates.
This article describes a real-time log processing platform that integrates Kafka, Flink, Drools rule engine, Redis, and Elasticsearch to unify heterogeneous log formats, extract business metrics, and provide configurable, dynamic data processing for large‑scale logging scenarios.
This article explains how to collect, process, and visualize Flink job logs on Kubernetes using an EFK stack with Fluentd, covering logging architectures, deployment of Elasticsearch, Kibana, and Fluentd, and the backend logic for querying and displaying logs in a feature platform.
This case study details how JD’s "Browse" UGC project evolved from rapid agile delivery to a performance bottleneck as data grew, and how introducing Elasticsearch, redesigning the query flow, and refactoring storage components restored fast, flexible searches for both front‑end and operations users.
AWS introduced OpenSearch, an Apache‑licensed, fully open‑source fork of Elasticsearch and Kibana that removes commercial restrictions, offers advanced features like security and machine‑learning, and promises seamless migration for existing users while gaining broad industry support.
This tutorial demonstrates how to integrate Elasticsearch into a Spring Boot microservice, covering component selection, Maven configuration, client setup, index creation, data insertion, complex query execution, synchronization of question data, and front‑end search handling, all illustrated with complete Java code examples.
This tutorial explains how to set up a Spring Boot project with Elasticsearch 6.2.1, configure RestHighLevelClient and RestClient beans, define ES host settings, and perform index creation, deletion, pagination, various query types, sorting, and highlighting through the Java API.
This article explains why Elasticsearch handles complex conditional queries more efficiently than MySQL by using inverted indexes, term dictionaries, skip‑list and roaring bitmap structures, while also discussing the trade‑offs such as slower write performance.
This article explains why Elasticsearch's built‑in tokenizers struggle with Chinese text, introduces the IK analyzer plugin, provides step‑by‑step Docker and file‑based installation methods, shows how to configure custom dictionaries via Nginx, and demonstrates smart and max‑word tokenization queries.
The article describes how algorithm testing teams tackled data‑quality and interface‑semantic monitoring problems by building a unified business monitoring platform that checks table, storage and service consistency, validates response semantics, and, through dashboards, alerts and correction tools, quickly identified dozens of offline and online issues, guiding future reliability enhancements.
This article shares an interview experience that compares MongoDB and MySQL, explains the advantages and disadvantages of document‑oriented and relational databases, introduces other NoSQL solutions such as Elasticsearch, and offers guidance on choosing the right storage technology based on workload characteristics.
This article explains how to configure Elasticsearch nodes for hot and cold data, assign special node attributes, adjust index templates, and use API calls to migrate data, demonstrating significant query speed improvements through real‑world performance tests.
This article examines Elasticsearch’s distributed search architecture, explains the two‑phase query and fetch process, identifies performance and relevance scoring challenges, and presents optimization techniques such as Search After with point‑in‑time, Scroll API usage, and DFS query‑then‑fetch for accurate scoring.
This article explains how a growing e‑commerce platform replaced a heavyweight MySQL intermediate table with a binlog‑driven pipeline that streams changes to Elasticsearch via Kafka, detailing the architecture, modules, customizations, monitoring, and performance results.
This article explains Filebeat's role as a lightweight log shipper, its relationship to the Beats family, internal architecture, installation steps, configuration of inputs, outputs, keystore usage, module activation, and practical examples for sending logs to Logstash or Elasticsearch.
This article presents practical best‑practice guidelines for configuring Elasticsearch in production, covering heap memory sizing, CPU considerations, shard and replica planning, hot‑warm node architecture, node role settings, common pitfalls, monitoring APIs, and troubleshooting tips.
This article presents multiple real‑world Elasticsearch deployments across Chinese tech giants, detailing how they handle massive query loads, scale clusters, integrate with Kafka and Kibana, and apply the Elastic Stack for search, analytics, and security monitoring.
This article explains the architecture and core concepts of Elasticsearch and Lucene, outlines the requirements for cross‑month and high‑speed queries on massive datasets, and provides detailed index and search performance tuning techniques—including bulk writes, shard routing, doc‑values management, and pagination strategies—to achieve sub‑second response times on billions of records.
Zhongtong’s ESPaaS platform automates deployment, unified monitoring, real‑time alerting, and diagnostic analysis for over 40 Elasticsearch clusters, leveraging custom exporters, Prometheus, Grafana, and DingTalk integrations to track resource, cluster, and node metrics, reduce noise, and prevent production incidents.
The article describes Uber’s Gairos platform—a unified real‑time data ingestion, storage, and query system built on Kafka, Elasticsearch and RT‑Gairos—detailing its architecture, Uber use cases such as dynamic pricing, the scalability and reliability challenges faced, and the suite of optimization techniques (sharding, query routing, caching, index merging, template tuning, and data‑pruning) applied to achieve low latency and high throughput.
This guide explains what Filebeat is, how it fits into the Beats ecosystem, its architecture and processing flow, installation steps, key configuration options, keystore usage, and practical examples of sending logs to Logstash or Elasticsearch, helping you set up reliable log collection on Linux.
This article explains how Elasticsearch handles data ingestion, retrieval, and full‑text search by describing the roles of coordinating, primary, and replica nodes, the refresh‑commit‑flush cycle, segment files, translog, and the Lucene‑based inverted index that powers its near‑real‑time capabilities.
This article analyzes common causes of slow MySQL queries—especially index misuse—offers practical indexing techniques, explains MDL locks and large‑table bottlenecks, and then compares ElasticSearch and HBase as complementary solutions for high‑performance search and storage.
This article explains how to optimize Elasticsearch for cross‑month and multi‑year queries on billions of records, covering Lucene fundamentals, index and search performance tweaks, configuration settings, and practical testing results to achieve sub‑second response times.
This article provides a comprehensive introduction to the ELK stack—Elasticsearch, Logstash, Kibana, and Filebeat—including its components, why it’s used for centralized log management, detailed architecture diagrams, step‑by‑step installation commands, configuration examples, and a practical Kafka‑based data pipeline demonstration.
This article recounts how Shay Banon’s early cooking‑app project led to the creation of Compass, the evolution into Elasticsearch, and the pivotal role of Apache Lucene, distributed testing, and visionary design in building today’s powerful search platform.
This article explores why MySQL queries become slow, delves into index pitfalls and optimization techniques, then compares ElasticSearch and HBase architectures, offering practical guidance on when to use each technology and how to combine them for high‑performance data retrieval.
This article explains how to design and implement a flexible, accurate data reporting platform using Elasticsearch aggregation, covering architecture, data synchronization, metric definition, SDK integration, and future scaling considerations for enterprise analytics.
This guide provides a comprehensive walkthrough of Kibana, covering installation, connection to Elasticsearch, data discovery, query syntax, visualizations, dashboards, and monitoring, with step‑by‑step instructions and screenshots to help users effectively explore and visualize their Elasticsearch data.
This article recounts how Shay Banon's early cooking‑app project led to the creation of Compass, the evolution of Apache Lucene, and ultimately the development of Elasticsearch—a powerful, distributed search platform built with extensive testing infrastructure and inspired by futuristic data‑interaction concepts.
This article explains Elasticsearch's internal mechanisms for writing, reading, and searching data, covering the roles of coordinating nodes, primary and replica shards, buffers, translog, segment files, refresh cycles, commit and flush operations, as well as Lucene's inverted index and how deletions and updates are handled.
The article details Elastic's shift of Elasticsearch and Kibana from Apache 2.0 to SSPL/Elastic License, the ensuing dispute with AWS over alleged coercion and trademark issues, AWS's plan to fork the projects under Apache 2.0, and the broader community backlash against the licensing change.
The article compares three approaches—synchronous dual write, asynchronous dual write with a message queue, and CDC via Canal—to keep Elasticsearch and a relational database consistent for cross‑database aggregated full‑text search, outlining their steps, advantages, and drawbacks.
The article explains Elastic's switch to SSPL/Elastic License for Elasticsearch and Kibana, Logz.io's criticism, AWS's fork under Apache 2.0, and the broader open‑source licensing battle between cloud providers and the original vendor.
The article examines how cloud service providers exploit open‑source software, why Elastic changed Elasticsearch and Kibana’s license to SSPL/Elastic License to curb this practice, and how similar moves by MongoDB, Redis and others reflect a broader shift in open‑source business models.
This article explains Ctrip's design and implementation of a flexible Elasticsearch data‑synchronization framework that handles full, incremental, ID‑based, and time‑based syncing from multiple sources, addressing the shortcomings of existing tools and simplifying complex data assembly for business search.
This article describes a high‑throughput solution for calculating hourly feature coverage, positive‑sample ratio and negative‑sample ratio on billions of records by streaming data with Spark, indexing per experiment‑hour in Elasticsearch, and executing parallel aggregation tasks with Java code.
This article summarizes a year‑long performance tuning of an ELK logging system, analyzing bottlenecks such as write thread pool saturation, JVM heap and GC settings, refresh intervals, translog durability, merge threads, shard and replica counts, and provides concrete configuration changes that reduced latency, eliminated data loss, and stabilized node resource usage.
This guide consolidates everyday Elasticsearch tuning techniques—covering configuration file tweaks, system‑level settings, and usage‑level optimizations such as memory locking, discovery settings, fault detection, queue sizing, translog handling, bulk indexing, shard management, and disk I/O—to help you build a stable, high‑throughput search cluster.
This article explains how to design, tune, and optimize an Elasticsearch‑based data platform handling hundreds of billions of records, covering Lucene fundamentals, shard routing, index and query performance tricks, and practical benchmark results for large‑scale deployments.
This article walks through the concepts, architecture, download, installation, agent setup, service startup, and alert configuration of Apache SkyWalking, an open‑source APM platform for cloud‑native microservices, including Elasticsearch integration and DingTalk notifications.
This article introduces Elasticsearch’s core concepts—including clusters, node roles, documents, mappings, and shards—and walks through the complete indexing workflow from client request to replica synchronization, highlighting key settings, routing calculations, and the role of refresh and flush operations.
This guide explains what Cerebro is, how to install it (including binary and Docker options), how to run it on Linux, macOS, and Windows, and how to use its UI to connect to an Elasticsearch node, view cluster overviews, manage shards, and execute DSL queries.
This article details how Vivo transformed its monolithic e‑commerce order service into a scalable, service‑oriented system by applying data archiving, sharding‑JDBC based sharding, MySQL‑to‑Elasticsearch sync, zero‑downtime migration, and distributed‑transaction techniques, while sharing practical pitfalls and performance results.
ebatis is an Elasticsearch ORM framework that simplifies query building by providing a MyBatis-like interface, eliminating the need for manual DSL construction and enabling seamless version upgrades.
The guide explains how to configure Tencent Cloud Elasticsearch clusters for multi‑AZ high availability by using zone‑aware node attributes, deploying data nodes in multiples of AZs, assigning three dedicated masters across zones, setting replica shards and force‑awareness rules, and safely upgrading single‑AZ clusters without service interruption.
This article explains the different index mechanisms used by MySQL (MyISAM and InnoDB) and Lucene, compares them with Elasticsearch's inverted index, and discusses how these structures affect storage, memory usage, and query performance.
This step‑by‑step guide shows how to deploy a production‑grade log collection pipeline by installing Filebeat on log sources, forwarding logs to a hardened Kafka cluster, processing them with Logstash, storing them in an Elasticsearch 7.2.0 cluster secured with X‑Pack, and visualizing the data in Kibana, all on CentOS 7.6 with detailed security hardening, configuration scripts, and verification commands.
This article explains how Alibaba Cloud’s intelligent syntax service leverages LSIF, Elasticsearch, and distributed scheduling to provide fast, cloud‑backed code navigation—eliminating the need for local clones and dramatically improving code review speed, accuracy, and resource efficiency.
This article explains Elasticsearch's data ingestion, retrieval, and search workflows, details the underlying indexing mechanisms, and provides comprehensive system‑level, shard‑level, and query‑level tuning recommendations—including configuration snippets and best‑practice strategies for high‑throughput and low‑latency deployments.
The article explains how to improve query performance by flattening multi‑table MySQL data and synchronizing it to Elasticsearch—using DataX for one‑time bulk loading and Canal (with Canal‑Adapter) for real‑time binlog‑driven incremental updates—while detailing configuration steps, job examples, and common pitfalls.
This tutorial walks through installing and configuring Apache SkyWalking, an open‑source APM system for micro‑services and cloud‑native environments, covering its architecture, Elasticsearch and InfluxDB storage setup, agent deployment, service startup, alarm integration, and essential documentation links.
Elasticsearch 7.10 adds searchable snapshots, letting users query indices stored directly in remote repositories such as S3 or COS, which halves storage costs, decouples storage from compute, supports manual mounting and ILM cold‑phase policies, and promises future full storage‑compute separation without local caching.
This article explains how Elasticsearch processes text before indexing by describing the analysis pipeline, built‑in and custom analyzers, tokenizers, token filters, n‑gram techniques, the analysis API, and the IK Chinese tokenizer plugin, providing practical curl examples throughout.
This article provides a comprehensive guide to Elasticsearch search requests, detailing the routing process, request structure, core modules like query, size, from, _source, and sort, and illustrating various query and filter types such as match, term, range, bool, and wildcard with practical curl examples.
This article explains how to define Elasticsearch mappings, use automatic and manual mapping, work with basic data types, multi‑fields and predefined fields, and perform document updates, version‑based concurrency control, and deletions, providing practical curl examples for each operation.
This article explains Elasticsearch's logical and physical design, how documents are structured and indexed, the role of shards and replicas, and provides practical examples of indexing, searching, aggregations, and retrieving documents using RESTful APIs.
This article outlines five practical methods for synchronizing MySQL data to external systems such as Elasticsearch, Redis, and message queues, covering business‑layer hooks, middleware integration, scheduled tasks using updated_at, binlog parsing with ROW format, and handling mixed or statement binlog formats, plus open‑source tools.
Elasticsearch combines Lucene’s inverted index with a distributed cluster of master‑eligible, data, and coordinating nodes, using Zen discovery for node election and split‑brain prevention, while indexing writes to primary shards, replicating to replicas, storing immutable segments that are periodically merged for efficient search.
This guide explains how to prepare resources, create a local‑volume storage class, install the Elasticsearch and Cluster Logging operators, and configure a persistent EFK stack on OpenShift, including YAML definitions, command‑line steps, and best‑practice notes on node selectors and tolerations.
This article provides a detailed collection of basic Elasticsearch interview questions and answers, covering core concepts such as cluster and node architecture, indexing, mapping, search APIs, analyzers, tokenizers, aggregation, security, and operational tools, along with practical code snippets for common tasks.
The article compares five data synchronization approaches—periodic Logstash pulls, synchronous dual writes, asynchronous dual writes with MQ, Canal-based binlog streaming, and a Canal‑MQ hybrid—detailing their architectures, advantages, drawbacks, and suitable scenarios for integrating databases with Elasticsearch.
This article presents a comprehensive design and deployment plan for a bank's unstructured data service platform, covering data growth challenges, lifecycle management, three‑tier storage architecture, Elasticsearch indexing, fault‑tolerant disaster recovery, monitoring, and future development directions.
This article shares practical Elasticsearch and Lucene optimization techniques—including index structure tuning, shard routing, DocValues management, and query pagination—to achieve sub‑second search performance on datasets exceeding a billion records while supporting multi‑year historical queries.
This article explains fundamental Elasticsearch concepts such as indexes, index types, and the immutable nature of shard counts, demonstrates how to define custom shard and replica settings when creating or updating indices, and outlines essential monitoring considerations and commands for cluster health.
This guide demonstrates how to configure Nginx log formatting, write Grok patterns, and set up Logstash to ingest Nginx access logs into Elasticsearch for visualization in Kibana, covering step‑by‑step configuration and regex extraction techniques.
This guide explains how to install Kibana 7.6.2 on a Linux server, configure its kibana.yml settings, start the service, troubleshoot common permission issues, and access the web interface for visualizing Elasticsearch data in real-time.
Elasticsearch provides two search types—query_then_fetch (default) and dfs_query_then_fetch—each involving a multi-step process where the client node distributes queries to relevant shards, shards execute searches using local or global term frequencies, aggregate results, and retrieve full documents, with noted trade‑offs.
This article introduces Elasticsearch as a Lucene‑based distributed search engine and demonstrates how to use curl commands to create, read, update, and delete documents and indices, providing step‑by‑step examples with command‑line output and screenshots.
This article explains how to separate Elasticsearch DSL queries from application code by defining reusable search templates with Mustache placeholders, managing them via the scripts API, and rendering them with parameters to produce flexible, maintainable search requests.
This guide explains how to set up a simple two‑node Elasticsearch cluster in a cloud environment, covering prerequisite checks, installation, configuration of master and slave nodes, certificate handling, password setup, and verification of the cluster’s health.
This article provides a complete tutorial for installing Java, downloading Elasticsearch 7.6.2, configuring elasticsearch.yml, starting the service, setting up user passwords, and verifying the single‑node deployment on a CentOS 7 server, complete with command examples and screenshots.
This article explains the principles and practical steps for implementing distributed tracing in microservice architectures using Spring Cloud Sleuth and Zipkin, covering why tracing is needed, core concepts like Span and Trace, Maven integration, Docker deployment, Zipkin UI usage, and performance analysis.
The article describes a real‑time video recommendation indexing system that replaces 30‑minute batch builds with an Elasticsearch‑based service, integrates prior and posterior data pipelines, ensures consistency via locking and version checks, enables zero‑downtime upgrades, smooths write spikes, and boosts recall performance through multi‑level caching and ES tuning, delivering sub‑40 ms latency and significant business growth.
This article explains why a terms query with an array fails inside an Elasticsearch search template, demonstrates the correct Mustache‑based template syntax, and provides step‑by‑step code examples for defining the index, inserting data, creating the template, and executing the query.
This article describes Ctrip's experience building a flexible, high‑performance Elasticsearch synchronization framework that supports full‑load, MQ‑based, ID‑based, and time‑based incremental indexing for diverse data sources such as Hive, MySQL, SOA services, and message queues.
Elasticsearch calculates a relevance score for each document, but using the function_score query you can customize this scoring by combining the original query_score with a user-defined func_score through various functions such as weight, random_score, field_value_factor, decay_function, and script_score, allowing flexible ranking based on business needs.
