This article explains Elasticsearch’s cluster architecture, detailing node roles, index, shard, and replica concepts, compares mixed and tiered deployment models, examines data storage mechanisms, and discusses the advantages and trade‑offs of different distributed system designs.
This article describes the background, design goals, architecture, implementation details, and lessons learned from building a low‑overhead, low‑intrusion distributed tracing system using Kafka, Elasticsearch, and OpenTracing to monitor microservice interactions and support performance analysis and DevOps decision‑making.
This article expands on Elasticsearch performance monitoring by detailing host‑level system and network metrics, cluster health and node availability, resource saturation, thread‑pool behavior, cache usage, and common error indicators, offering practical guidance for maintaining a stable and efficient search cluster.
This article explains how to anticipate and monitor Elasticsearch issues such as node unavailability, OOM errors, and long garbage‑collection pauses by tracking key performance metrics across query, indexing, memory, and system levels, helping prevent service disruptions.
This article introduces Apache Solr and Elasticsearch, explains their shared Lucene foundation, compares their scalability, deployment ease, and key features, and highlights why Elasticsearch is often considered more efficient for large‑scale, multi‑tenant search applications.
To cope with Meituan Hotel’s exploding micro‑service complexity, the infrastructure team built the Satellite System—combining MTrace and a selective, zero‑intrusion Log4j2‑based logging pipeline that streams enriched logs through Kafka, Storm, Redis and Elasticsearch, delivering second‑level trace‑log queries and six‑month retention, dramatically speeding up debugging.
This tutorial walks you through installing Elasticsearch, understanding its core concepts such as nodes, clusters, indexes, documents and types, configuring Chinese analyzers, performing CRUD operations, and executing various search queries with practical command‑line examples.
This article explains Elasticsearch’s core mechanisms, covering the cluster startup sequence, node discovery and election, handling of failed nodes, cluster management APIs, and the detailed read/write processes including coordinating nodes, shard allocation, memory buffering, transaction logs, and query‑then‑fetch execution.
The guide explains how to optimize Elasticsearch performance by managing cluster state size through shard planning, assigning dedicated master, data, ingest, and coordinating node roles, and preventing memory swapping with bootstrap.memory_lock and proper heap and OS settings.
This article explains how 360’s Hulk private cloud platform deploys Elasticsearch with a dedicated master architecture, load‑balancing, per‑business isolated clusters, SearchGuard security, dynamic tokenization, self‑service user features, and advanced monitoring to achieve high‑performance, scalable search services.
This article explains the core components of the ELK stack, compares three typical deployment architectures, and provides practical solutions for multiline log merging, timestamp correction, and module‑based log filtering using Filebeat, Logstash, Kafka, and Kibana.
This third‑part guide for Elasticsearch 5+ shows how rounding date ranges, eagerly loading global ordinals for keyword fields, and pre‑loading index files into the OS cache—each illustrated with curl commands—can improve search latency while requiring careful trade‑offs between cache friendliness, query complexity, and resource usage.
This guide walks through the design, installation, configuration, performance tuning, and daily operation of a large‑scale network packet capture and analysis platform using Moloch, Elasticsearch, and pfring on a high‑performance Linux server.
The article introduces the ELKB stack—a combination of Elasticsearch, Logstash, Kibana, and Beats—explaining its background, user needs, architecture, component functions, typical scenarios, and the team’s practical implementations for real‑time log and time‑series data processing.
The second part of the Elasticsearch search‑tuning series explains how to pre‑index data, choose appropriate keyword or text mappings, minimize script use by preferring Painless or Lucene expressions, and efficiently force‑merge read‑only indices into single segments for better performance.
This guide explains how to tune Elasticsearch 5.x search performance by modeling documents with flattened, nested, or parent‑child structures, managing global ordinals, allocating JVM heap under 50 % of RAM, monitoring garbage collection, and choosing SSD storage and ample file‑system cache.
This article explains the design and implementation of a unified seller‑operation logging platform that uses Kafka for ingestion, Storm for real‑time processing, Elasticsearch for hot‑data search, and HBase for cold‑data storage, detailing the challenges faced and the optimizations applied.
This article describes how the OPS team designed and implemented a private‑cloud Elasticsearch service using Mesos for resource management, Docker containers orchestrated by Marathon, and a suite of monitoring, self‑service configuration, and continuous deployment tools to improve resource utilization and operational efficiency.
Part 3 of the Elasticsearch indexing optimization guide advises balancing shard and replica counts, using index aliases, leveraging the Bulk API with appropriately sized batches, dedicating data nodes, and upgrading storage (SSD/RAID 0) while monitoring resources to achieve higher throughput and scalable, reliable clusters.
This article shares practical insights on operating large‑scale Elasticsearch clusters for log analysis, covering use cases, essential tools, hardware choices, node role separation, shard management, hot‑cold data strategies, version upgrades, and key monitoring metrics to ensure stability and performance.
This article, based on Xu Peng’s Gdevops 2017 talk, details the rationale for choosing Elasticsearch, outlines the overall architecture, provides step‑by‑step OS, JVM, and index parameter settings, and explains comprehensive monitoring strategies to ensure high‑availability and performance of large‑scale ES clusters.
This tutorial walks you through installing Elasticsearch, configuring Java and network settings, understanding core concepts like nodes, clusters, indices and documents, setting up Chinese analyzers, performing CRUD operations, and executing powerful full‑text queries using the Elasticsearch REST API.
This article details JD's end‑to‑end seller log system architecture, explaining why Kafka, Storm, Elasticsearch and HBase were chosen, the challenges faced during scaling, and the practical solutions implemented to achieve a unified, high‑throughput logging platform for merchants and operations.
To maximize Elasticsearch bulk-indexing speed, temporarily disable refreshes and replicas, tune merge throttling and scheduler threads, enlarge translog and index buffer thresholds, and adjust indexing and bulk thread-pool sizes, then restore defaults after the load completes.
Jkes is a Java‑based search framework built on Kafka and ElasticSearch that provides annotation‑driven JPA‑style mapping, REST APIs for indexing and searching, and detailed integration with Spring Boot, offering developers a complete backend solution for scalable document search.
This article explains how the ELK stack—Elasticsearch, Logstash, and Kibana—provides a flexible, low‑learning‑curve solution for real‑time collection, analysis, and visualization of diverse operational metrics and log data, comparing it with Hadoop, Spark, and traditional databases while highlighting its features, configuration, and best practices.
This tutorial walks through creating a demo mobile app using Ionic that fetches performance data from a public Elasticsearch server via a Flask proxy, visualizes it with Chart.js, and outlines the project structure, code snippets, and known limitations.
This article details the architectural choices, component selection, performance tuning, and team organization behind Ctrip's big‑data platform, covering Kafka, Presto, Elasticsearch, Gobblin, Zeppelin, REST APIs, and job scheduling to achieve scalable, interactive data analysis and visualization.
This article summarizes Xu Peng's DAMS 2017 presentation on selecting big‑data platform components, designing ETL pipelines, choosing analysis engines, optimizing Elasticsearch, and building a data‑driven team at Ctrip.
The article explains how a schemaless JSON document model stored in a relational database can replace traditional relational tables for trading system orders, reducing schema change costs, improving flexibility, and ensuring consistency through version‑based optimistic locking and real‑time Elasticsearch indexing.
This article explains the background, installation, certificate generation, Elasticsearch and Kibana configuration, role mapping, performance testing, and final recommendations for securing an Elasticsearch 5.1.1 cluster with the open‑source Search Guard plugin, focusing on index‑level access control and RPC traffic encryption.
This article compiles a curated list of Spring Data tutorials covering core JPA configuration, method inheritance, query creation, named queries, @Query usage, sorting, as well as Elasticsearch architecture, overview, basic and practical examples, all sourced from the Spring4All community.
This article explains how Elasticsearch computes relevance scores using Lucene's practical scoring formula, term frequency, inverse document frequency, field-length norms, and query normalization, and demonstrates query-time boosting, constant_score, function_score, decay functions, and script_score with practical DSL examples.
This guide walks through setting up a distributed log collection and analysis platform using ELK (Elasticsearch, Logstash, Kibana) together with Redis on two CentOS machines, covering environment preparation, component installation, configuration, testing, and visualizing nginx access logs.
This article describes the architecture, deployment, configuration, maintenance, and performance results of a large‑scale near real‑time logging platform built with Kafka, Flume, and Elasticsearch, highlighting practical lessons and future plans for resource‑efficient operation.
This article reviews the latest Elasticsearch 5.x enhancements—including append‑only indexing, range fields, removal of the _all field, unified highlighter, keyword normalizer, multi‑word synonyms, field collapsing, cancellable searches, partitioned term aggregations, cluster allocation explain, Java REST client updates, cross‑cluster search, batched reduce phases—and previews the major features expected in Elasticsearch 6.0 such as sparse doc values, index sorting, sequence numbers, seamless rolling upgrades, type removal, index‑template inheritance, load‑aware shard routing, and X‑Pack extensions like SQL and machine learning.
This tutorial walks you through installing and configuring Haystack, Elasticsearch, and the IK Chinese analyzer in a Django project, covering both a simple Whoosh‑based setup and a scalable Elasticsearch backend with code snippets and version‑compatibility tips.
This guide explains how to set up the ELK (Elasticsearch, Logstash, Kibana) logging platform using Docker Compose, covering prerequisites, environment configuration, single‑host and multi‑host deployments, and verification steps to get a functional ELK stack in minutes.
This article provides a comprehensive, step‑by‑step tutorial on installing and configuring Search Guard for Elasticsearch, covering feature overview, version compatibility, downloading required packages, local installation commands, SSL/TLS certificate generation, and detailed security settings to protect both transport and REST layers.
The article shows how attackers automate mass exploitation of widely‑known flaws—scanning the Internet for open MongoDB, Redis, ElasticSearch or Struts2 services, using unauthenticated access or public PoCs to encrypt data, execute code, or build botnets, and stresses that timely patching and secure defaults are essential to stop such N‑day attacks.
This guide walks through setting up ElastAlert—a Python‑based alerting framework for Elasticsearch—by covering the required environment, dependencies, installation steps, index creation, configuration files, rule testing, and execution on an Ubuntu system.
This article introduces Elasticsearch as a reliable open‑source search and analytics engine, explains its key concepts such as clusters, nodes, indexes, types and documents, walks through installation steps, and demonstrates essential CRUD operations using RESTful curl commands.
Meituan Waimai’s Feature Archive platform processes billions of daily orders by managing ~200 user and 400 merchant tags through a three‑layer architecture—Hive, Elasticsearch, HBase, and MySQL—offering visual tag selection, instant self‑service queries, full data extraction, and a predicate‑logic query language, while supporting future extensibility.
This guide walks you through downloading, extracting, and starting Elasticsearch, linking it with Logstash, then installing and configuring Kibana, including setting host IP, connecting to Elasticsearch, and accessing the Kibana UI to visualize logs on a Linux server.
Learn how to install and configure the ELK stack (Elasticsearch, Logstash, Kibana, and Filebeat) on Ubuntu 16.04, covering architecture, component roles, download sources, command-line installation, pipeline testing, and integration of Filebeat for real-time log collection and visualization.
This article explains the concept of centralized logging, introduces the ELK stack (Elasticsearch, Logstash, Kibana), compares it with other log solutions, and provides a detailed, cross‑platform installation and configuration guide for Ubuntu 14.04 and CentOS 7.1, including SSL setup and Nginx reverse‑proxy.
This article reviews four ELK deployment patterns, compares their strengths and weaknesses, and demonstrates how ELK can be applied to big‑data operations such as log aggregation, system monitoring, fault diagnosis, and performance analysis with practical examples.
This article recounts how Ctrip’s operations team replaced costly commercial tools with an open‑source ELK stack, detailing requirements, architecture, optimizations, and the impressive scale achieved for real‑time log analysis across thousands of servers.
This guide walks through the complete installation, configuration, and integration of the ELK stack components—Logstash, Redis, Elasticsearch, and Kibana—providing command‑line examples, configuration snippets, testing procedures, and visual verification using Kibana and the Kopf plugin.
This article outlines practical Elasticsearch security measures—including network, OS, and server hardening, port obfuscation, and the open‑source Search Guard plugin—to protect clusters from recent hacker attacks that previously compromised MongoDB instances.
This guide introduces Logstash as an open‑source data collection engine, explains its core input‑filter‑output architecture, walks through installation, and demonstrates three practical examples: console I/O, output to Elasticsearch, and reading from Redis with real‑time output.
This article describes how Qunar's operations team designed and implemented a cloud‑native Elasticsearch‑as‑a‑Service platform using Mesos, Marathon, and Docker, covering requirements analysis, technology selection, resource quota management, cluster isolation, service discovery, data reliability, monitoring, automated deployment, and future improvements.
The paper presents a production‑grade, low‑cost mobile‑app exception monitoring platform built on Spark Streaming, Kafka, and Elasticsearch that achieves high availability through exactly‑once processing and checkpointing, minute‑level latency by decoupling raw and symbolized logs, high throughput via reservoir sampling, and dynamic scalability without code changes.
This tutorial shows how to quickly set up a log analysis platform using the ELK stack, OpenResty, Ansible, and Vagrant, covering architecture, required tools, server provisioning, deployment commands, testing procedures, and next‑step enhancements for real‑world use.
The article details the Qunar Order Center's architecture evolution—addressing fragmented order data, introducing unified query APIs, JSON‑based storage, merge/diff services, asynchronous I/O, Elasticsearch integration, data sharding, business isolation, rate limiting, adapter layers, micro‑service decomposition, and a centralized configuration system to improve scalability, reliability, and maintainability.
This article walks through the full ELK Stack installation on a Linux test environment, covering component overview, host planning, required packages, detailed command‑line steps for Elasticsearch, Logstash, Kibana and Filebeat, and the necessary firewall and service checks.
This guide explains what the ELK Stack is, outlines its key advantages, and provides detailed, command-line instructions for installing JDK, Elasticsearch 2.2.0, Kibana 4.4.1, and Logstash on Linux, along with testing, Log4j integration, and a common startup permission fix.
This article presents a comprehensive technical case study of Qunar's Order Center, describing its initial architecture, the challenges faced with scaling and reliability, and the successive redesigns in versions 2.0 and 3.0 that introduced Elasticsearch, asynchronous I/O, micro‑service decomposition, rate‑limiting, and a configurable adapter layer to support massive, multi‑line order processing.
This guide walks you through centralized log collection, explains the roles of Elasticsearch, Logstash, and Kibana, and provides step‑by‑step commands to install, configure, and verify a complete ELK logging pipeline for Linux servers.
This article explains how JD.com designs a multi‑layer data storage architecture—using MySQL for core metadata, HBase and MongoDB for comment text, Solr/Elasticsearch for indexing, and Redis for caching—to handle billions of daily review requests with high performance and availability.
This article provides a comprehensive overview of Elasticsearch 2.x, covering its distributed architecture, core concepts such as clusters, nodes, indices, shards and replicas, the ZenDiscovery master‑election process, scaling mechanisms, recovery, query features, and the underlying system components like Guice, Netty, and thread‑pool designs.
This article shares the author's experience and lessons learned while building a log‑monitoring framework with the ELK stack, discussing performance issues, configuration of Logstash filters using Grok, and practical tips for deploying ElasticSearch, Logstash, and Kibana in production environments.
This article explains the fundamental differences between distributed systems and clusters, illustrates them with real‑world examples such as Dubbo, Elasticsearch and Git, and details various cluster types, high‑availability and load‑balancing strategies, as well as fault‑tolerance mechanisms.
The article describes the design, components, and implementation details of Youzan's unified log platform, covering log ingestion via rsyslog, Logstash, and Flume, centralized processing with Kafka, real‑time analysis using Storm/Spark, and storage in HDFS, Elasticsearch, and Hawk, while also discussing challenges and future improvements.
This article details Youzan's practical architecture, indexing pipeline, advanced search module, and performance‑tuning techniques for building a commercial e‑commerce search engine using Elasticsearch, Hadoop, Kafka, and custom backend integrations.
This article describes the practical architecture of Youzan's commercial e‑commerce search engine, covering data source integration, distributed real‑time indexing with Elasticsearch, Hadoop and Kafka, advanced search modules, and several performance‑tuning techniques for large‑scale deployments.
This guide walks you through the fundamentals of big‑data characteristics, the challenges of log management for small‑scale operations, and provides a detailed, step‑by‑step tutorial for installing and configuring the ELK stack (Elasticsearch, Logstash, Kibana) on a CentOS 6.6 server, including all required dependencies.
This article walks through installing Elasticsearch on Ubuntu, adding the IK Chinese analyzer and synonym filter, configuring custom analyzers, and using the Node.js client to index and query documents, providing a complete, reproducible setup for site‑wide full‑text search.
This article walks through installing Elasticsearch on Ubuntu, adding the IK Chinese analyzer plugin, configuring synonym filters, and using the official Node.js client to create indices, mappings, and perform full‑text searches with custom analyzers and highlighting.
This article shares the author's practical experience building a real‑time log analysis platform at Sina, covering service scope, ELK architecture, performance optimizations, usability improvements, new features, common pitfalls, and a concise Q&A for engineers handling massive log streams.
This guide walks through ElasticSearch fundamentals, core concepts, step‑by‑step installation, Python indexing examples, index‑level tuning, and memory‑usage optimizations, providing practical tips for deploying and maintaining a high‑performance search cluster.
Time‑series data, defined by a timestamp field, appears everywhere, and the article explains how to choose an appropriate time‑series database by comparing two schema models, their query patterns, performance trade‑offs, and why modern solutions like Elasticsearch, columnar stores, and Druid excel at real‑time massive aggregation.
This article explains the distinctive features, architectural patterns, core modules, technology choices, performance optimizations, and operational challenges of e‑commerce search engines, illustrating how they differ from general web search and how to build a robust, real‑time, high‑availability search system for online retail platforms.
This article outlines the design and implementation of Weidian’s end‑to‑end big data processing platform, covering dataset definition, data collection via Flume‑based DataAgent, transmission through Databus, storage options such as HDFS, Kafka and Elasticsearch, and the monitoring and resource‑integration strategies that support massive mobile commerce logs.
This article shares practical Elasticsearch experience covering index and query performance optimization, shard routing strategies, JVM tuning, daily maintenance, and answers to common production questions, providing actionable guidance for building high‑availability search clusters.
This article compiles the Q&A from a high‑level operations talk where experts discuss ELK cluster sizing, performance tuning, schema handling, integration with Hadoop and Splunk, and practical tips for managing massive log streams in production environments.
This article explores how the ELK stack can transform large‑scale log processing into fast, flexible, and interactive analysis for troubleshooting, security auditing, and monitoring, sharing practical examples, common pitfalls, and best‑practice recommendations from real‑world deployments at Sina.
This guide explains why log analysis is critical, outlines common challenges, compares heavyweight trace systems with the lightweight LEK stack, and provides step‑by‑step installation, configuration, and best‑practice tips for Logstash, Elasticsearch, and Kibana.
