Tagged articles
Data Ingestion
64 articles · Page 1 of 1
This article explains how AgentLoop uses the OpenTelemetry protocol and probes to ingest high‑quality runtime data, offering four integration methods—one‑click generic agents, SDK framework integration, annotation‑based high‑code, and eBPF—demonstrated with a Claude Code customer‑service agent and end‑to‑end verification on the observation page.
Agent IntegrationAgentLoopData Ingestion
0 likes · 9 min read

The article describes a fully automated pipeline that lets pre‑sales engineers upload a raw CSV/JSON sample, automatically infer mappings, mask sensitive fields, ingest data into Easysearch, generate a searchable, chart‑driven dashboard, and clean up the session with a single click, eliminating the tedious manual preparation that normally dominates POC demos.
DashboardData IngestionEasysearch
0 likes · 14 min read

The article breaks down the essential big data components—data ingestion, distributed storage, resource management, batch and stream processing, analytics, search, and cluster operations—using a restaurant metaphor and lists typical tools such as Flume, Kafka, HDFS, Spark, Flink, Elasticsearch, and Kubernetes.
Data Ingestionbatch processingbig data
0 likes · 7 min read

Faced with language barriers in cross‑border e‑commerce, a seller with no English, Japanese, or Korean skills built a multilingual storefront by combining the no‑code web builder Kouzi with Viking AI SearchCLI, automating data ingestion, multilingual search, recommendation, QA, evaluation, and one‑click deployment to sell across Japan, Korea and English‑speaking regions.
Data IngestionE‑commerceMultilingual AI
0 likes · 11 min read

The article analyzes common pitfalls in RAG data ingestion—connection failures and incomplete records—advocates defining required metadata fields before integration, and provides source‑specific guidelines for databases, APIs, object storage, web crawlers, and manual uploads to ensure reliable downstream governance.
AIData IngestionETL
0 likes · 17 min read

LDA‑1B unifies world modeling and VLA in a latent dynamics action model, ingesting over 30 000 hours of heterogeneous embodied data via a five‑layer AstraData pipeline, employing a unified end‑effector space and quality‑based data allocation, and achieving state‑of‑the‑art success rates on RoboCasa‑GR1 while being fully open‑sourced.
Data IngestionEmbodied AILatent Dynamics
0 likes · 13 min read

The article breaks down a big‑data project into four essential phases—data ingestion, preprocessing, storage, and analysis—explaining common pitfalls, practical tool choices, and why aligning technology with business needs is crucial for success.
Data IngestionData StorageHadoop
0 likes · 10 min read

This article explains how to centralize and analyze massive multi‑cloud log data stored in object storage by moving AWS S3 logs into Alibaba Cloud Log Service (SLS) using dual‑mode file discovery, SQS event‑driven import, elastic scaling, and pre‑ingestion processing to achieve low latency, high reliability, and cost efficiency.
AWS S3Data IngestionElastic Scaling
0 likes · 12 min read


21CTONov 10, 2025 · Databases
This article presents a detailed performance benchmark comparing MySQL 9.0 and PostgreSQL 17.0, measuring data‑ingestion latency, throughput, saturation, CPU and memory usage, as well as query efficiency, and concludes which open‑source database delivers superior write and read performance.
Data IngestionDatabase PerformanceMySQL
0 likes · 10 min read

An LLM-driven intelligent data-ingestion framework replaces manual, months-long integration with an automated code-generation and execution loop that auto-recognizes schemas, maps structures, extracts quality rules, builds deployment packages, cutting onboarding time from three months to three days while eliminating human effort.
Data IngestionLLMautomated ETL
0 likes · 19 min read

This article presents an LLM‑driven intelligent data platform ingestion solution that automates schema recognition, mapping, quality rule extraction, and package building, reducing integration cycles from three months to three days while eliminating manual effort and enhancing scalability and control.
AIData IngestionLLM
0 likes · 21 min read

This article presents a complete Java‑based architecture for high‑throughput Doris stream loading, covering project structure, Maven dependencies, configuration properties, field‑mapping annotations, automatic mapper utilities, a robust parallel loader with retry and compression, plus performance tuning recommendations.
Annotation MappingData IngestionDoris
0 likes · 23 min read

This guide demonstrates using StarRocks Routine Load to ingest JSON and CSV data from Kafka, covering environment setup, table creation, load job configuration with critical parameters like jsonpaths, troubleshooting null-field errors, task management commands, and performance optimization tips for real-time analytics.
CSVData IngestionJSON
0 likes · 14 min read

This article details HuoLaLa's logistics platform challenges with petabyte‑scale data, the selection of Apache Flink CDC for stable, compatible, and low‑latency data ingestion, the construction of a multi‑layer CDC capability, migration strategies, measurable performance gains, and future open‑source contributions.
Apache FlinkData IngestionFlink CDC
0 likes · 15 min read

Facing the challenge of importing billions of image assets, we redesigned the pipeline using Dataworks open‑API, clustered tables, data sharding, cube tables, and custom key generation, achieving faster parallel processing, fault tolerance, and flexible attribute storage, and share practical insights on scheduling, view parametrization, and output services.
Data Ingestioncube tablehash algorithm
0 likes · 18 min read

This article introduces Apache Hudi Streamer, a versatile Spark‑based data ingestion tool likened to a Swiss Army knife, detailing its core options—including table configuration, continuous mode, source classes, transformers, table services, catalog synchronization, and advanced features—while guiding users on practical pipeline setup.
Apache HudiData IngestionSpark
0 likes · 10 min read

This article summarizes a live session where Shawn explains how understanding business scenarios drives effective OpenGemini database design, and provides comprehensive best‑practice guidance on library and table design, data ingestion, query optimization, and performance tuning for time‑series workloads.
Data IngestionPerformance TuningSharding
0 likes · 7 min read

This article explains how Apache Pulsar is used within Apache InLong to collect, sort, and reliably deliver massive MySQL binlog incremental data, covering component architecture, job isolation, client and producer management, consumption strategies, common pitfalls, performance tuning, and practical code examples.
Apache PulsarData IngestionInLong
0 likes · 21 min read

This article introduces time series data concepts, outlines the challenges of storing and analyzing high‑frequency data, and presents best‑practice guidelines for building MongoDB‑based time‑series applications, covering ingestion, read/write workloads, retention, security, and real‑world use cases.
AnalyticsData IngestionMongoDB
0 likes · 12 min read

This tutorial explains how to install and configure Elasticsearch and Kibana, demonstrates CRUD operations, bulk data import, and shows how to use Logstash to ingest, transform, and index flight JSON data, covering both batch and near‑real‑time processing techniques.
Data IngestionElasticsearchJSON
0 likes · 31 min read

This article presents an overview of a smart city unified perception platform, detailing its modular architecture, solutions for multi-source heterogeneity, incremental synchronization strategies, and real-time API data collection, while discussing extensibility and practical implementation considerations.
API integrationData IngestionIncremental Sync
0 likes · 20 min read

Apache Druid is a distributed, column‑store OLAP engine designed for massive real‑time data ingestion and sub‑second queries; this article explains its LSM‑tree‑inspired architecture, DataSource and Segment structures, memory‑based querying, practical deployment steps, common pitfalls, and optimization techniques for high‑throughput analytics.
Apache DruidData IngestionOLAP
0 likes · 20 min read

This guide explains ClickHouse fundamentals, OLAP versus OLTP concepts, columnar storage benefits, core performance techniques, the MergeTree family and its indexing, specialized table engines, installation on Linux, Docker deployment, and integration with HDFS, MySQL, and Kafka for modern analytical workloads.
ClickHouseData IngestionDocker
0 likes · 30 min read

This article demonstrates two approaches for efficiently importing massive CSV data into MySQL using Python: a direct pymysql method with chunked inserts and a concise pandas‑SQLAlchemy method, comparing performance, code complexity, and offering tips for further speed improvements.
Data IngestionMySQLPandas
0 likes · 5 min read

Apache Druid manages metadata through a layered, distributed system where the Overlord coordinates ingestion tasks, MiddleManagers launch Peons to create segments, Coordinators and Historical nodes store and serve segment data, Brokers route queries, while MySQL, Zookeeper, memory, and local files synchronize metadata for fault‑tolerant, high‑performance OLAP analytics.
Data IngestionDruidQuery Processing
0 likes · 19 min read

This guide explains how to use standard PostgreSQL INSERT, multi‑row INSERT, INSERT … SELECT, COPY for bulk loading, upserts with ON CONFLICT, and UPDATE/DELETE on Citus‑distributed tables, including routing by distribution column, performance tips, and two‑phase commit considerations.
CitusData IngestionDistributed Database
0 likes · 13 min read

This article details the design and implementation of an advertising data platform at eBay, explaining the business scenario, why ClickHouse was chosen over alternatives, the technical challenges faced, and the solutions involving lambda architecture, table engine choices, compression techniques, data ingestion pipelines, consistency guarantees, and deployment practices.
AdvertisingClickHouseData Ingestion
0 likes · 26 min read

JD.com’s customer service team leverages the open‑source MPP database Doris to power real‑time and offline OLAP dashboards, detailing data ingestion pipelines, full‑link monitoring, dual‑stream high‑availability design, dynamic partition management, multi‑layer caching strategies, and performance optimizations applied during the 2020 11.11 shopping festival.
Data IngestionDorisOLAP
0 likes · 15 min read

This article explains the typical architecture of a big‑data platform, detailing its four core layers—data collection, storage & analysis, data sharing, and application—and describing the key technologies such as Flume, DataX, HDFS, Hive, Spark, Spark Streaming, and task scheduling components.
Data IngestionDataXFlume
0 likes · 8 min read

Snuba is a ClickHouse‑based service that offers rich data models, fast Kafka ingestion, and a rule‑based query optimizer for Sentry, replacing the original Postgres‑Redis stack and supporting multiple time‑series datasets, streaming and point‑in‑time queries with configurable consistency guarantees.
ClickHouseData IngestionKafka
0 likes · 11 min read

This article explains Elasticsearch's core mechanisms for indexing, querying, and searching data, covering the roles of coordinating nodes, primary and replica shards, refresh cycles, translog, commit/flush processes, and the underlying Lucene inverted index.
Data IngestionElasticsearchLucene
0 likes · 13 min read


ITPUBSep 9, 2021 · Big Data
This article explains the origins and purpose of data lakes, outlines four key construction goals, details common ingestion methods and storage technologies, and describes essential governance practices such as cataloging, data quality, and regulatory compliance.
Data IngestionETLdata governance
0 likes · 18 min read

This article outlines a typical big data platform architecture, detailing its core layers—data acquisition, storage and analysis, sharing, application, real‑time computation, and task scheduling—while introducing key technologies such as Flume, HDFS, Hive, Spark, DataX, and monitoring considerations.
Data IngestionHadoopSpark
0 likes · 9 min read

This guide walks through installing the Python Elasticsearch client, building a reusable class with CRUD methods, importing data from MongoDB, writing a simple Baidu Baike crawler, and scaling the workflow with Celery and Flask for a complete search‑engine pipeline.
Data IngestionElasticsearchFlask
0 likes · 9 min read

This article details 58 Group's comprehensive evaluation of DorisDB, TiFlash, and ClickHouse for large‑scale analytical workloads, covering functional and performance benchmarks, real‑world use cases such as security analysis and DBA operations, data ingestion methods, cluster architecture, automation practices, and lessons learned.
Data IngestionDorisDBOperations
0 likes · 10 min read

Meituan evaluated open‑source distributed graph databases against strict latency, scale, and import criteria, selected NebulaGraph for its superior multi‑hop query and bulk‑load performance, and built a four‑layer, highly available platform that ingests petabyte‑scale data in real time, supports diverse business use cases, and provides interactive visualization.
Data IngestionHigh AvailabilityNebulaGraph
0 likes · 21 min read

This article provides a comprehensive overview of Apache Flume, detailing its purpose as a distributed log aggregation system, explaining its core components such as sources, channels, and sinks, and illustrating its architecture, multi‑agent setups, and key features like reliability, scalability, compression, and monitoring.
Data IngestionFlumelog aggregation
0 likes · 9 min read

This article details how a bank transformed its retail system data pipeline from a monolithic DB2 setup to a distributed Oracle‑Hadoop architecture, evaluated five extraction tools, selected Sqoop direct mode, and implemented customizations to achieve over 50% performance gains and reliable incremental data capture.
Data IngestionDirect ModeHadoop
0 likes · 11 min read

This article details ByteDance's evolution from offline ClickHouse ingestion to a robust real‑time analytics pipeline, covering external transaction handling, risks of direct INSERTs, recommendation and ad‑delivery use cases, Kafka Engine design, multi‑threaded consumption, fault‑tolerance improvements, platform tooling, and future roadmap.
ClickHouseData IngestionKafka
0 likes · 22 min read

The article presents Entropy Simplify's AirWorks data intelligence platform, detailing its three‑layer architecture, cloud‑native multi‑source data ingestion system, low‑code ETL capabilities, technical features such as multi‑engine cooperation and data‑skew handling, and a financial asset‑management case study.
Data IngestionETLFinancial Services
0 likes · 16 min read

This article explains how JD’s Energy Management Platform uses ClickHouse as an MPP‑based OLAP engine to ingest, store, and provide multi‑dimensional real‑time analytics on energy consumption data, covering architecture decisions, data pipelines, replication, sharding, and a generic query interface.
ClickHouseData IngestionOLAP
0 likes · 12 min read

The article provides a comprehensive overview of the Qirin big‑data platform, detailing its architecture, core modules such as resource management, metadata, data ingestion, task development, interactive query, and self‑service analysis, and outlines future development plans for the system.
Data Ingestiondata platforminteractive query
0 likes · 12 min read

The article introduces the Qirin big data platform—a one‑stop solution covering resource management, metadata, data ingestion, task development, interactive querying, and self‑service analysis—detailing its modular architecture, typical processing workflow, and future development plans for enterprise‑wide data services.
Data IngestionTask Schedulingbig data
0 likes · 11 min read

The article describes Alibaba Cloud's Data Lake Formation service, presenting a unified, real‑time, and low‑latency solution for ingesting heterogeneous data sources—including RDS, DTS, TableStore, and SLS—into an OSS‑backed data lake using templates, a Spark‑based ingestion engine, and modern file formats such as Delta Lake.
Alibaba CloudData IngestionDelta Lake
0 likes · 10 min read

This guide walks through downloading Apache Flume, setting up a master‑slave cluster, and configuring NetCat, Exec, and Avro sources with corresponding sinks and memory channels, including verification commands to ensure the agents run correctly.
Apache FlumeCluster SetupData Ingestion
0 likes · 5 min read

This article explains how to identify and resolve performance bottlenecks in Apache Flume by configuring Taildir sources, optimizing channel capacities, tuning Kafka sinks, adjusting JVM options, and using simple monitoring scripts, enabling a single Flume‑NG agent to sustain over 50,000 RPS in production.
ConfigurationData IngestionFlume
0 likes · 10 min read

This article presents a systematic guide to real‑time MySQL binlog ingestion, outlining three core principles—decoupling from business data, handling schema changes, and ensuring traceability—followed by concrete Debezium‑Kafka‑Hive solutions, scenario‑specific tactics, and practical tips for reliable data pipelines.
Data IngestionDebeziumHive
0 likes · 15 min read

This article details ByteDance's engineering practices for using ClickHouse to ingest, store, and query massive real‑time recommendation and advertising data, covering early external‑transaction mechanisms, the risks of direct INSERTs, the design and evaluation of Kafka Engine versus Flink pipelines, and a series of performance and reliability improvements implemented to support high‑frequency workloads.
ClickHouseData IngestionKafka
0 likes · 20 min read

This article explains Elasticsearch's internal mechanisms for indexing, querying, and retrieving data, covering the roles of coordinating nodes, primary and replica shards, the refresh and commit cycles, near‑real‑time search, and the underlying Lucene inverted index.
Data IngestionElasticsearchLucene
0 likes · 12 min read

This article describes how JD.com’s Algorithmic Intelligence team built a ClickHouse‑based real‑time analytics engine, covering ClickHouse fundamentals, MergeTree table design, Kafka‑Flink data pipelines, JDBC batch loading, query‑optimization techniques, and monitoring for handling billions of rows with sub‑second response times.
ClickHouseData IngestionMergeTree
0 likes · 14 min read

This article explains how Kafka can be used for data ingestion and event distribution by illustrating production‑consumer and publish‑subscribe models, describing core concepts such as topics, partitions and consumer groups, and offering practical design options for handling different event scenarios.
Data IngestionEvent DistributionKafka
0 likes · 9 min read

This article explains how Apache Hudi provides an incremental processing framework that enables efficient, low‑latency data ingestion, storage, and query capabilities on Hadoop, detailing its architecture, storage layout, compaction, write and read paths, and support for real‑time and batch analytics.
Data IngestionHadoopHudi
0 likes · 15 min read

OPPO's ESA DataFlow is a self‑developed high‑performance data‑flow framework that processes over a trillion events per day, offering flexible routing, scalable sources and sinks, persistent mmap‑based channels, built‑in monitoring, and easy extensibility for diverse data‑collection scenarios.
Data IngestionESA DataFlowOPPO
0 likes · 11 min read

This article explains how Apache Hudi enables efficient, low‑latency incremental data ingestion and processing on Hadoop by providing a unified service layer, describing its motivation, architecture, storage components, write and read paths, compaction, fault recovery, and incremental query capabilities.
Apache HudiData IngestionHadoop
0 likes · 17 min read

This article traces the evolution of Youzu's data platform ingestion, comparing early HTTP/script methods with modern DTS and real‑time ETL solutions, evaluating middleware choices, detailing core system architectures, and outlining future improvements for reliable, scalable data access.
DTSData IngestionETL
0 likes · 6 min read

The article explains how DBus ingests data from relational databases and log sources by detailing its extractor, incremental conversion, and full‑pull modules, the use of Canal and Kafka, rule‑based log structuring, the unified UMS message format, and heartbeat monitoring for reliability.
CanalDBusData Ingestion
0 likes · 13 min read

This article reviews six popular data collection platforms—Apache Flume, Fluentd, Logstash, Chukwa, Scribe, and Splunk Forwarder—explaining their architectures, strengths, and typical use cases within modern big‑data pipelines.
Apache FlumeData IngestionFluentd
0 likes · 10 min read


21CTOSep 28, 2017 · Operations
This article explains how LogHub enables real-time log collection and unified management for an e‑commerce takeout platform, covering operational challenges, logstore configuration, user promotion tracking, server and client logging methods, and network access options.
Data IngestionLog CollectionLogHub
0 likes · 9 min read

This guide introduces Logstash as an open‑source data collection engine, explains its core input‑filter‑output architecture, walks through installation, and demonstrates three practical examples: console I/O, output to Elasticsearch, and reading from Redis with real‑time output.
Data IngestionElasticsearchLogstash
0 likes · 6 min read

This article explains how a billion‑scale data analytics system can achieve second‑level data ingestion and query without predefined metrics, detailing the product requirements, technical choices, and the end‑to‑end architecture from collection to storage and real‑time querying.
Data IngestionImpalareal-time analytics
0 likes · 16 min read


21CTOJun 15, 2016 · Big Data
This article reviews major data collection platforms—including Apache Flume, Fluentd, Logstash, Chukwa, Scribe, and Splunk Forwarder—explaining their architectures, strengths, and limitations to help engineers select the most reliable and scalable solution for big‑data pipelines.
Apache FlumeData IngestionFluentd
0 likes · 10 min read

This article provides a comprehensive overview of Flume NG, covering its architecture, core components (source, channel, sink), reliability mechanisms, common deployment scenarios, installation steps, configuration examples, compilation instructions, and practical best‑practice recommendations for building robust log‑collection pipelines.
ApacheConfigurationData Ingestion
0 likes · 16 min read

This article introduces Apache Flume NG, a distributed and reliable log collection system, explains its core architecture components such as Event, Flow, Agent, Source, Channel, and Sink, and provides detailed configuration examples for various pipelines, including load‑balancing, failover, and integration with HDFS.
Apache FlumeConfigurationData Ingestion
0 likes · 12 min read

The article outlines the purposes, architecture, data ingestion, storage, analysis, sharing, application, real‑time processing, scheduling, monitoring, and best‑practice recommendations for building a fast, flexible, and reliable big‑data platform in the fast‑changing internet industry.
Data IngestionHadoopSpark
0 likes · 12 min read
