Tagged articles
bigdata
73 articles · Page 1 of 1

JD TechDec 18, 2025 · Backend Development
This article showcases a series of real‑world development cases—from implementing a guided inventory task in a Java backend and generating Vue rule code, to writing unit tests, analyzing report data, converting SQL to Hive, debugging startup errors, publishing Maven APIs, optimizing slow SQL queries, and resolving MySQL deadlocks—demonstrating how AI‑driven prompts can accelerate coding, testing, and troubleshooting across multiple domains.
BackendDebuggingMaven
0 likes · 31 min read

This article explains Spark Streaming's rate control mechanisms, covering static limits, the dynamic back‑pressure feature introduced in Spark 1.5, the PID‑based estimator, RPC communication, and how Guava's token‑bucket RateLimiter enforces the calculated thresholds to ensure stability and optimal throughput.
RateControlSparkbackpressure
0 likes · 13 min read

This article examines the current challenges of enterprise data analysis platforms, outlines three traditional analysis modes, and presents a next‑generation solution that combines Headless BI’s semantic modeling with Chat BI’s large‑language‑model interaction to deliver a more efficient, secure, and user‑friendly analytics experience.
ChatBIDataGovernanceHeadlessBI
0 likes · 15 min read

This article explains how a CDP tackles billions‑scale user ID tags and groups by replacing naïve text‑file storage with bitmap techniques, detailing Bitmap basics, encoding strategies, Java BitSet limitations, and the adoption of RoaringBitmap for efficient compression and fast set operations.
RoaringBitmapStoragebigdata
0 likes · 10 min read

The article explains how to store and deduplicate billions of identifiers efficiently by using a bitmap backed by Redis and extending it with a Bloom filter implementation in Java, highlighting memory calculations, practical commands, and code examples.
BitmapBloomFilterDataDeduplication
0 likes · 5 min read

This article details the background, planning, step‑by‑step procedures, encountered issues, and rollback strategies for upgrading a Hadoop HDFS cluster from version 2.6.0‑cdh to 3.1.2, including mixed‑deployment of DataNodes across different federations and necessary configuration changes.
DataNodeHDFSHadoop
0 likes · 16 min read


ITPUBDec 14, 2023 · Big Data
This step‑by‑step guide shows how to install and configure a single‑node Hadoop 3.2.0 environment on CentOS 7, set up Python 3.7, write MapReduce mapper and reducer scripts in Python, and run a word‑count job using Hadoop streaming, illustrating core Hadoop concepts and their relevance today.
HadoopMapReducePython
0 likes · 21 min read

This article introduces Alibaba Cloud DataWorks' intelligent data modeling tool, outlines the data demand flow, shares best practices and practical demonstrations of data warehouse modeling, discusses model application and data asset management, and answers common questions while highlighting its commercial availability.
AlibabaCloudDataGovernanceDataWarehouse
0 likes · 12 min read

This article details Zhihu's evaluation of PrestoDB and Trino, the integration of Alluxio for query acceleration, the architectural choices and deployment modes, extensive TPC‑DS and production performance tests, encountered challenges, and future optimization directions for their OLAP platform.
AlluxioCachingOLAP
0 likes · 28 min read

This article details Lenovo’s implementation of a unified scheduling center using Apache DolphinScheduler, covering background requirements, reasons for choosing the platform, architectural evolution, feature enhancements, and practical deployments such as HTTP task parameter passing, Java task plugins, global parameters, and future roadmap.
DolphinSchedulerLenovoWorkflow
0 likes · 19 min read

We Analysis, the official data‑analysis platform for WeChat mini‑program providers, delivers a zero‑learning‑curve user‑profiling system that combines basic tag analysis and flexible, rule‑based segmentation, using an ETL pipeline to store pre‑computed data in TDSQL and online bitmap‑optimized queries in ClickHouse with RoaringBitmap, ensuring low‑latency, stable, and comprehensive analytics.
ClickHouseDataPipelineSpark
0 likes · 20 min read

This comprehensive guide introduces ClickHouse as a high‑performance columnar DBMS, outlines its main application scenarios, details the various table engines and their creation syntax, and provides practical development, deployment, and operational recommendations for building reliable ClickHouse clusters.
ClickHouseClusterArchitectureSQLGuidelines
0 likes · 22 min read

This article examines a business requirement to filter and rank up to 100,000 records from a pool of tens of millions, presenting and evaluating four technical solutions—multithreaded ClickHouse pagination, Elasticsearch scroll‑scan deep paging, an ES‑HBase combined query, and a RediSearch + RedisJSON approach—along with performance data and code examples.
ClickHouseElasticsearchHBase
0 likes · 12 min read


ITPUBJan 20, 2023 · Big Data
This article details Bilibili's practical deployment of an Iceberg lake‑warehouse architecture within its OLAP platform, covering the motivations for lake‑warehouse integration, core Iceberg optimizations such as data‑organization sorting, Z‑order and secondary indexes, the Magnus intelligent management platform, and future roadmap plans.
IndexingPrecomputationbigdata
0 likes · 16 min read

An incident report details how a scheduled machine reboot on Alibaba Cloud triggered a Flink TaskManager failover, leading to excessive data replay, increased ES pressure, and significant business latency, and explains the root cause involving disabled checkpoints and timestamp‑based offset consumption.
FailoverFlinkKafka
0 likes · 10 min read

This comprehensive tutorial walks you through Alibaba DataWorks' data warehouse architecture, covering technical stack selection, three‑layer warehouse design (ODS, CDM, ADS), detailed data modeling with DDL examples, storage strategies, dimension and fact table conventions, and best‑practice hierarchical call standards.
DataModelingDataWarehouseDataWorks
0 likes · 27 min read

This tutorial walks through building a real‑time data processing pipeline using ElasticSearch, Kibana and Logstash, covering core concepts such as data volume, velocity, variety and accuracy, detailed installation steps, CRUD operations, bulk data import, Java‑based data conversion, and Logstash pipeline configuration with filters and date parsing.
BulkImportDataPipelineJava
0 likes · 31 min read

This article provides a comprehensive guide to Elasticsearch 7.x DSL query syntax, explaining core keywords, field mappings, various query types such as match, term, range, fuzzy, and bool, and includes practical code examples for building effective search queries.
DSLElasticsearchElasticsearch7
0 likes · 8 min read

This article investigates why Spark tasks that appear killed in the Web UI continue running on the driver, analyzes the role of daemon versus non‑daemon threads and SparkContext shutdown mechanisms, reproduces the issue with sample code, and provides practical solutions such as using daemon threads or checking SparkContext status.
DaemonThreadSparkbigdata
0 likes · 8 min read

This article provides a comprehensive overview of Kafka consumer group rebalance, covering version compatibility, rebalance triggers, assignment strategies, generation handling, protocol details, the full rebalance workflow, listener usage, and complete Java code examples for offset management with database integration.
ConsumerGroupJavaKafka
0 likes · 19 min read

This article provides a comprehensive guide on using Flink CDC with Debezium to capture MySQL changes, covering serialization, adding dependencies, configuring SQL client and Java/Scala APIs, creating source and sink tables, enabling checkpoints, and streaming data into a Hudi data lake.
CDCDataLakeFlink
0 likes · 10 min read

This article details how Hangzhou Zhiyi Technology leverages AI, big‑data pipelines, and Alibaba Cloud services to create a scalable fashion‑focused image‑recognition and visual‑search platform, covering company background, system architecture, model training, vector search, and future technical upgrades.
AICloudComputingFashionTech
0 likes · 13 min read

When executing a Spark SQL query with dozens of UNION subqueries that each contain JOIN operations on Spark 3.1.2, the job fails because the total serialized result size of the tasks exceeds the driver’s maxResultSize limit, and the issue can be resolved by reducing the initial partition number used by Adaptive Query Execution.
DriverMaxResultSizePerformanceTuningSQL
0 likes · 10 min read


政采云技术Jul 12, 2022 · Big Data
This article explains Spark SQL's physical execution plan, detailing each operator, how to interpret the plan, and practical optimization tips for data warehouse developers to improve SQL performance and resource utilization.
DataWarehouseExecutionPlanPerformanceOptimization
0 likes · 10 min read

This article explains Doris's compaction mechanism, covering its producer‑consumer architecture, tablet scoring, permission control, cumulative and base compaction processes, parameter tuning, monitoring metrics, and manual compaction commands to help optimize performance and resource usage.
Dorisbigdatacompaction
0 likes · 38 min read

The article details Ant Group’s AntMonitor observability platform, covering its development timeline, holographic monitoring capabilities, integrated performance analysis, efficient data integration, built‑in AI‑driven analytics, Monitoring‑as‑a‑Service, and the underlying high‑performance time‑series database and cloud‑native architecture that support massive real‑time data processing.
AIOpsCloudNativeTimeSeriesDatabase
0 likes · 17 min read

MaFengWo’s data platform replaced Kylin, Presto, and Druid with StarRocks, redesigning its four‑layer architecture, unifying metadata, and optimizing single‑table, multi‑table, and precise‑deduplication queries, which cut query latency by four times, reduced storage by 87%, and lowered operational complexity.
Kylinbigdatadata-warehouse
0 likes · 15 min read

This article describes how Sohu Intelligent Media built a microservice tracing system using Zipkin for data collection and StarRocks for storage and analysis, covering architecture, data model, ingestion pipeline, SQL analytics, performance monitoring, and future improvements.
StarRocksTracingZipkin
0 likes · 27 min read

This article explains the core concepts of ClickHouse's MergeTree engine, its main features, how to create tables with various clauses, and detailed settings that control data storage, partitioning, replication, sampling, TTL, and granule management for efficient analytical workloads.
ClickHouseMergeTreeSQL
0 likes · 10 min read

This article explains how to use Async I/O in Flink, describes the ordered and unordered output modes, details the internal AsyncWaitOperator implementation with its producer‑consumer model, and discusses fault‑tolerance mechanisms including state snapshot and recovery.
FaultToleranceFlinkJava
0 likes · 17 min read

The author shares a personal journey through Java backend development, big‑data frameworks, database evolution, and team decision‑making, analyzing efficiency, environmental influences, and the impact of community and leadership on technology selection, while emphasizing practical trade‑offs over theoretical performance gains.
BackendJavaTeamManagement
0 likes · 31 min read

This article presents an in‑depth overview of ClickHouse's Projection feature, covering its background, definition, storage and query mechanisms, practical use‑case demonstrations, performance comparisons with competing OLAP systems, and real‑world production results that highlight its advantages and limitations.
ClickHouseDataWarehouseMaterializedView
0 likes · 20 min read

This guide walks through preparing the environment, creating a MySQL source table, configuring Flink CDC to ingest data into an Apache Hudi table, and then querying the Hudi data using both Hive and Spark‑SQL, including handling of partitions, realtime input formats, and required configuration settings.
CDCDataPipelineFlink
0 likes · 10 min read

This article explains Elasticsearch’s three pagination methods—From + size, search_after, and Scroll—detailing their definitions, code examples, advantages, disadvantages, and suitable scenarios, while also discussing max_result_window limits, PIT views, and best practices for handling large result sets.
BackendElasticsearchPagination
0 likes · 13 min read

This article explains how Spark implements shuffle write and shuffle read, compares its high‑level and low‑level processes with Hadoop MapReduce, and details the internal data structures, memory‑disk trade‑offs, and configuration options that affect performance.
MapReduceMemoryManagementRDD
0 likes · 21 min read

This article explains how Spark builds the logical execution graph for RDDs, describes the four-step job processing pipeline, details the various dependency types such as NarrowDependency and ShuffleDependency, and reviews common transformations and their data‑flow characteristics.
RDDShuffleSpark
0 likes · 19 min read

At the 2021 Greater Bay Area Smart Retail Conference, Suning’s Director Wang Junjie revealed how the company’s unified, cross‑scenario membership platform leverages big data and AI to boost private‑domain traffic, streamline member lifecycle management, and deliver seamless digital marketing across all retail formats.
AIPrivateDomainbigdata
0 likes · 4 min read

DiDi migrated over 10,000 Hive SQL tasks to Spark SQL using a lightweight dual‑run pipeline that extracts, rewrites, compares, and switches tasks, fixing syntax and UDF differences while adding features such as small‑file merging and enhanced partition pruning, resulting in Spark handling 85 % of workloads with 40 % faster execution, 21 % lower CPU and 49 % lower memory usage.
DataMigrationHiveSQLOptimization
0 likes · 18 min read

DiDi upgraded its massive real‑time computing engine from Flink 1.4.2 to Flink 1.10, implementing a transparent migration across 1500 machines, adding native DDL, binary rows, MiniBatch, improved scheduling and window functions, and establishing a rigorous testing pipeline that achieved 99.9 % compatibility while preventing OOM issues.
FlinkPerformanceOptimizationRealTimeComputing
0 likes · 11 min read

This article describes how 58.com introduced ClickHouse to handle massive daily user‑behavior logs, detailing its features, multi‑layer architecture, configuration management, monitoring, performance optimizations, and operational automation to build a high‑availability, low‑cost analytics platform.
ClickHouseDataWarehouseOLAP
0 likes · 20 min read

This article details how to migrate Spark Streaming jobs to Flink Streaming by creating custom SourceFunction and SinkFunction implementations, including a RocketMQ source connector and an HBase sink, with code examples, configuration tips, and discussion of checkpointing and watermark handling.
FlinkHBaseRocketMQ
0 likes · 20 min read

This article systematically details Didi's migration of over 30 Elasticsearch clusters, 3,500 nodes and 8 PB of data from version 2.3.3 to 6.6.1, covering background, problem analysis, multi‑version architecture redesign, capacity planning, tiered storage, FastIndex, query replay, upgrade pitfalls, and the resulting cost reduction and performance improvements.
CapacityPlanningElasticsearchUpgrade
0 likes · 15 min read

This article explains how JD.com’s JUST platform leverages the open‑source columnar database ClickHouse to store, query and analyze massive time‑series data, covering data modeling, lifecycle management, system goals, technology selection, cluster architecture, deployment, scaling and future enhancements.
ClickHouseDistributedSystemsbigdata
0 likes · 20 min read

This article explains how Flink's Timer mechanism works, covering its usage in KeyedProcessFunction, the underlying TimerService and InternalTimerService implementations, the role of triggers, and the detailed code paths for processing‑time and event‑time timers, while highlighting performance considerations.
FlinkInternalTimerServiceKeyedProcessFunction
0 likes · 16 min read

This article explains what HBase is, describes its Master, RegionServer, and Zookeeper components, and provides detailed environment preparation and configuration steps—including host setup, SSH key distribution, JDK installation, HBase deployment, configuration file edits, and cluster startup—so you can run HBase on a Hadoop cluster.
HBaseHadoopbigdata
0 likes · 8 min read

The article explains HBase's RegionServer management of regions and stores, detailing HStore composition, MemStore flushing, split conditions, column family sharing within regions, and the performance implications of multiple column families, recommending a single column family design for optimal I/O efficiency.
ColumnFamilyHBaseRegionServer
0 likes · 3 min read

This article explains how Spark SQL processes a SELECT query—detailing parsing, binding, optimization, planning, and execution steps—including the roles of SQLContext, HiveContext, Catalyst optimizer, logical and physical plans, and provides code excerpts from the Spark source.
CatalystHiveContextQueryExecution
0 likes · 13 min read

This article presents a comprehensive guide for transferring massive MySQL datasets to HBase, covering environment setup on Ubuntu, three synchronization methods—MySQL LOAD DATA, a Kafka‑Thrift pipeline using Maxwell, and real‑time Flink processing—along with performance comparisons and practical tips for Hadoop, HBase, Kafka, Zookeeper, Phoenix, and related tools.
DataSyncFlinkHBase
0 likes · 24 min read

This article reviews B+ trees, introduces log‑structured merge (LSM) trees, compares their strengths and weaknesses, and explains how HBase leverages LSM trees, HFiles, compaction, and Bloom filters to achieve high‑performance storage for write‑intensive workloads.
B+TreeHBaseLSMTree
0 likes · 8 min read

This article explains how Hadoop's Namenode persists metadata using FsImage and EditLog, describes the checkpoint process during startup, and details the role of SecondaryNamenode in merging these files for efficient recovery, while also encouraging readers to like and share the content.
EditLogFsImageHadoop
0 likes · 4 min read

This tutorial walks through installing ElasticSearch on CentOS, setting up the ElasticSearch‑head visual plugin, and integrating ElasticSearch with a Spring Boot application, including environment preparation, configuration, CRUD API implementation, and testing via Postman, providing a comprehensive guide for developers.
bigdatasearch
0 likes · 14 min read

This article explains how to compute the top‑N hot products or brands within a time window using Apache Flink, covering data modeling, event‑time handling, sliding windows, custom aggregation functions, and result sorting with complete Java code examples.
EventTimeFlinkJava
0 likes · 14 min read

This article explains HBase compaction—a key operation in the Log‑Structured Merge‑Tree model—covering minor and major compaction differences, trigger conditions, configuration parameters, selection algorithms, thread‑pool handling, and the effects on read and write performance in a big‑data database environment.
HBaseLSMbigdata
0 likes · 10 min read

This article explains the three time notions supported by Apache Flink—ProcessTime, EventTime, and IngestionTime—detailing their semantics, how Watermarks enable event‑time processing, and provides Scala code samples for configuring time characteristics, assigning timestamps, and generating Watermarks in a streaming job.
EventTimeFlinkScala
0 likes · 16 min read

This article provides a comprehensive overview of SparkSQL's logical plan architecture, detailing the stages of logical plan creation, analysis, rule‑based optimization, and the underlying catalog and rule systems that transform SQL queries into efficient execution plans.
LogicalPlanOptimizerScala
0 likes · 12 min read

This article explains how Tubi Data Runtime (TDR) streamlines data access on JupyterHub by introducing an optimized Redshift connector, custom SQL cell magic, and JupyterLab extensions for data exploration, reducing latency and resource usage while enhancing collaboration and usability for data scientists and engineers.
DataConnectorJupyterHubKubernetes
0 likes · 12 min read

The article explains how HBase tracks each region's lifecycle states in hbase:meta and ZooKeeper, lists all possible states with their color codes, and describes the master‑region server interactions for opening, closing, splitting, and merging regions.
HBaseHadoopRegionState
0 likes · 7 min read

This article outlines the core concepts of a data middle platform, covering data aggregation, ingestion tools, offline and real‑time development, data governance, service layers, and practical implementation details such as ODS, DWD, and monitoring, illustrating how enterprises build scalable, secure data ecosystems.
DataGovernanceDataMiddlePlatformDataWarehouse
0 likes · 32 min read

This article explains how to use HBase's built‑in PerformanceEvaluation tool to run baseline read/write latency tests (P99 and P999), describes key command‑line parameters, presents benchmark results for random and sequential operations, and discusses the implications for HBase performance tuning.
BenchmarkDatabasePerformanceHBase
0 likes · 11 min read

This article demonstrates how to build a one‑second‑refresh real‑time dashboard for e‑commerce order data using Apache Flink, Kafka, and Redis, covering JSON message parsing, processing‑time windows, stateful aggregation for site‑level KPIs, and efficient top‑N product ranking via Redis sorted sets.
DashboardFlinkKafka
0 likes · 11 min read

This article analyzes the performance and stability bottlenecks of a Hadoop 2.7.3 NameNode caused by memory limits, RPC QPS, and long restart times, and presents a comprehensive solution stack—including HDFS federation, ViewFS, FastCopy, and tuned Balance/Mover tools—to improve scalability and reduce downtime.
BalanceFastCopyFederation
0 likes · 11 min read

This article explains the three time concepts in Flink—Process Time, Event Time, and Ingestion Time—illustrates their impact on windowed computations with examples, introduces watermarks and allowed lateness for handling out‑of‑order data, and provides complete Scala code for both processing‑time and event‑time streaming applications.
EventTimeFlinkScala
0 likes · 13 min read

This article introduces Spark SQL fundamentals, including its architecture, DataFrame and Dataset abstractions, query methods, interoperability with RDD, user-defined functions, integration with Hive, data source handling, and provides step‑by‑step Scala code examples for loading data, performing aggregations, and solving common analytical tasks.
DataFramesHiveSQL
0 likes · 15 min read

This article explains HBase's compaction mechanism, covering why it is needed, the differences between minor and major compaction, the conditions that trigger compaction, key configuration parameters, thread‑pool handling, compaction policies, and how compaction influences read and write performance in a large‑scale NoSQL database.
HBasebigdatacompaction
0 likes · 12 min read

This article summarizes practical Kafka best‑practice guidelines covering hardware sizing, OS and JVM tuning, disk layout choices, replica and controller settings, broker and topic evaluation, as well as producer and consumer configuration, monitoring metrics, and strategies to prevent data loss.
Kafkabigdatastreaming
0 likes · 14 min read

The article explains why HBase read operations are complex due to its LSM‑Tree storage and multi‑version design, details the step‑by‑step Scan workflow, discusses the reasons for its multi‑request architecture, compares scan performance with Parquet and Kudu, and offers recommendations for large‑scale data scanning.
HBaseLSM-treeSCAN
0 likes · 7 min read

The article provides a comprehensive overview of Hadoop's Capacity Scheduler, describing its resource‑allocation features, configurable XML parameters, queue access controls, dynamic configuration updates, and the internal workflow of application initialization and resource scheduling within YARN.
CapacitySchedulerHadoopResourceManagement
0 likes · 13 min read

This article explains Hadoop's DistributedCache mechanism, its APIs for adding cache files and archives, common use cases, important considerations, the basic workflow, and provides a complete Java Map-side join example demonstrating how to distribute and access cached data in MapReduce jobs.
DistributedCacheHadoopJava
0 likes · 10 min read

This article examines Spark's memory management and the shuffle process, identifies the components that consume the most memory during shuffle write and read, analyzes common OOM scenarios such as task concurrency and data skew, and offers configuration tips to prevent out‑of‑memory failures.
MemoryManagementOutOfMemoryShuffle
0 likes · 14 min read

This article presents a comprehensive overview of Ant Financial's data analysis platform, detailing its departmental role, the data analysis lifecycle, the platform's evolution from version 1.0 to 3.0, core technical components such as intelligent sync and pre‑computation, and a practical case study of performance optimization.
AnalyticsDataAnalysisDataEngineering
0 likes · 24 min read

This article explains how HDFS DataNode stores data blocks on local disks, detailing the configuration of storage directories, the two volume‑choosing policies (round‑robin and available‑space), their implementation via the VolumeChoosingPolicy interface, and the logic used to balance disk usage.
AvailableSpaceDiskBalancingHDFS
0 likes · 10 min read


21CTOJul 6, 2017 · Big Data
Facing the challenge of storing over 120 billion daily monitoring points from hundreds of thousands of servers, Tencent’s monitoring platform migrated from a custom solution and OpenTSDB to a finely tuned HBase architecture, achieving 3‑5× higher throughput, improved reliability, and significant storage savings.
DistributedStorageHBasePerformanceTuning
0 likes · 11 min read

This article examines YouTube's large‑scale architecture, detailing its platform components, web and video services, database evolution, data‑center strategy, and key lessons for building high‑concurrency, fault‑tolerant backend systems.
BackendOperationsYouTube
0 likes · 9 min read
