BestHub
Sign in
Tagged articles

big data

3804 articles · Page 33 of 39
Big Data Technology & Architecture
Big Data Technology & Architecture
Jan 2, 2019 · Big Data

Optimizing Spark Direct Kafka Consumption: Subpartition Concurrency and Repartition Strategies

To address the long processing time caused by uneven Spark partitions when reading Kafka via the Direct approach, this article explains the SPARK‑22056 solution that modifies KafkaRDD.getPartitions to support a configurable 'topic.partition.subconcurrency' parameter, discusses its trade‑offs, and presents alternative repartition and multithreading techniques.

PartitioningScalaSpark
0 likes · 6 min read
Optimizing Spark Direct Kafka Consumption: Subpartition Concurrency and Repartition Strategies
Big Data Technology & Architecture
Big Data Technology & Architecture
Jan 2, 2019 · Big Data

Understanding Spark Streaming Backpressure Mechanism

The article explains how Spark Streaming backpressure, introduced in version 1.5, automatically adjusts data ingestion rates based on processing delays, replaces manual rate limits, and details its architecture, configuration parameters, and usage for preventing data backlog and executor OOM.

Rate ControlSparkStreaming
0 likes · 6 min read
Understanding Spark Streaming Backpressure Mechanism
Big Data Technology & Architecture
Big Data Technology & Architecture
Jan 1, 2019 · Big Data

Insights from the Real-Time Big Data Meetup: Spark Structured Streaming Overview

The meetup on September 8, co‑hosted by InfoQ and Huawei Cloud, featured Databricks engineer Tathagata Das explaining Spark Structured Streaming’s concepts, fault‑tolerance, performance, event‑time handling, and real‑world use cases such as Apple’s security platform, highlighting its scalability and integration with various data sources.

SparkStructured Streamingbig data
0 likes · 8 min read
Insights from the Real-Time Big Data Meetup: Spark Structured Streaming Overview
Architects Research Society
Architects Research Society
Dec 30, 2018 · Big Data

Overview of Major Apache Big Data Processing Frameworks

This article provides a concise overview of numerous Apache open‑source projects—including Ignite, MapReduce, Pig, JAQL, Spark, Storm, Flink, Apex, REEF, Twill, and Beam—that enable distributed in‑memory storage, real‑time and batch processing, and advanced analytics for large‑scale data workloads.

ApacheDistributed ComputingFlink
0 likes · 22 min read
Overview of Major Apache Big Data Processing Frameworks
Tencent Cloud Developer
Tencent Cloud Developer
Dec 28, 2018 · Big Data

Intelligent Operations for Tencent Cloud Big Data Platform: Challenges, Practices, and Future Directions

Tencent Cloud’s big‑data platform tackles massive, multi‑component clusters by deploying an AIOps framework that aggregates logs and metrics, applies statistical and machine‑learning anomaly detection, uses regression and reinforcement‑learning for job‑parameter optimization, and integrates offline‑online pipelines, achieving over 88 % precision while planning automated root‑cause analysis, productized tools, platformized algorithm integration, and cross‑domain model reuse.

AIOpsCloud Computingbig data
0 likes · 20 min read
Intelligent Operations for Tencent Cloud Big Data Platform: Challenges, Practices, and Future Directions
Meituan Technology Team
Meituan Technology Team
Dec 27, 2018 · Artificial Intelligence

Meituan’s AI Initiatives: Large‑Scale Scheduling, Unmanned Delivery, and the Meituan Brain Knowledge Graph

Meituan’s AI division, now over 1,000 engineers with a 2 billion‑CNY quarterly budget, powers massive real‑time scheduling for 20 million daily orders, unmanned delivery pilots, and the “Meituan Brain” knowledge graph of billions of entities, delivering AI‑driven services across its entire platform.

AILarge-Scale SchedulingMeituan
0 likes · 16 min read
Meituan’s AI Initiatives: Large‑Scale Scheduling, Unmanned Delivery, and the Meituan Brain Knowledge Graph
Xianyu Technology
Xianyu Technology
Dec 27, 2018 · Big Data

Device Fingerprinting and User Growth Architecture in Alibaba's Xianyu Platform

Alibaba’s Xianyu platform uses a multi‑signal device fingerprinting system, UMID, to uniquely identify users across Android and iOS devices, storing the data in sharded MySQL, HiStore OLAP, and Tair caches, enabling precise ad bidding, conversion tracking, and scalable user‑growth strategies.

Advertising OptimizationSystem Architecturebig data
0 likes · 9 min read
Device Fingerprinting and User Growth Architecture in Alibaba's Xianyu Platform
Didi Tech
Didi Tech
Dec 26, 2018 · Industry Insights

How Didi Implements Full‑Chain Data Tiered Protection for Reliable Operations

Facing growing data‑driven pressures, Didi designed a full‑link data tiered protection framework that defines classification standards, integrates data levels across the entire pipeline, and applies concrete safeguards and tooling to improve resource allocation, backup reliability, and overall data reliability.

DidiTiered Protectionbig data
0 likes · 7 min read
How Didi Implements Full‑Chain Data Tiered Protection for Reliable Operations
Alibaba Cloud Developer
Alibaba Cloud Developer
Dec 20, 2018 · Big Data

Unlocking Alibaba’s Massive Cluster Data V2018: A Treasure Trove for Big‑Data Research

Alibaba has released the comprehensive Cluster Data V2018 dataset, detailing eight days of operation for 4,000 servers and their mixed online and offline workloads, including DAG information, enabling researchers to study large‑scale data‑center performance, resource utilization, scheduling algorithms, and derive new insights.

DAGResource Utilizationbig data
0 likes · 7 min read
Unlocking Alibaba’s Massive Cluster Data V2018: A Treasure Trove for Big‑Data Research
Didi Tech
Didi Tech
Dec 18, 2018 · Big Data

Evolution and Architecture of Didi's Real-Time Computing Platform

From early self‑built Storm and Spark Streaming clusters to a unified YARN‑based Spark platform and finally a low‑latency Flink system with extended CEP and StreamSQL capabilities, Didi’s real‑time computing platform evolved through three stages, delivering multi‑tenant isolation, rich SQL processing, and dramatically reduced development costs.

CEPFlinkReal-Time Computing
0 likes · 9 min read
Evolution and Architecture of Didi's Real-Time Computing Platform
Qunar Tech Salon
Qunar Tech Salon
Dec 18, 2018 · Big Data

Practical Insights on Deploying and Operating Elasticsearch at Scale

This article shares extensive practical experience from Qunar's large‑scale Elasticsearch deployment, covering suitable use cases, index‑type design, document ID strategies, scaling considerations for index and data volume, hardware sizing, and storage architecture recommendations to help newcomers avoid common pitfalls.

Elasticsearchbig dataindexing
0 likes · 10 min read
Practical Insights on Deploying and Operating Elasticsearch at Scale
JD Tech
JD Tech
Dec 17, 2018 · Operations

Improving JD Intelligent Supply Chain Efficiency and System Stability for Major Sales Events

The article details JD's intelligent supply chain enhancements—including machine‑learning demand forecasting, a new "explosive product warehouse" model, non‑stock fulfillment visualization, blockchain‑based product traceability, and comprehensive system‑stability measures such as data‑consistency checkpoints, throughput buffering, and 24/7 incident response—to boost efficiency and reliability during large‑scale promotions.

Operationsbig datablockchain
0 likes · 7 min read
Improving JD Intelligent Supply Chain Efficiency and System Stability for Major Sales Events
Youzan Coder
Youzan Coder
Dec 14, 2018 · Operations

Youzan Full‑Link Load Testing Architecture and Implementation

Youzan’s full‑link load‑testing architecture combines a traffic generator, a data‑factory pipeline, and the Maxim platform to replay realistic e‑commerce user actions, tag and isolate test traffic via unified headers, route reads/writes to shadow storage, and integrate Gatling for capacity planning, degradation, alarm, disaster‑recovery and throttling drills.

Data IsolationGatlingLoad Testing
0 likes · 13 min read
Youzan Full‑Link Load Testing Architecture and Implementation
JD Retail Technology
JD Retail Technology
Dec 12, 2018 · Big Data

Construction and Architecture of JD Overseas Data Analysis Platform (Columbus Platform)

JD.com’s overseas data analysis platform, dubbed the Columbus platform, combines a lightweight data warehouse deployment with standardized, customizable BI tools to provide real‑time and offline analytics, visualization, KPI management, and future self‑service reporting and predictive capabilities for its global e‑commerce operations.

AnalyticsBIJD.com
0 likes · 9 min read
Construction and Architecture of JD Overseas Data Analysis Platform (Columbus Platform)
Manbang Technology Team
Manbang Technology Team
Dec 12, 2018 · Big Data

Kafka Overview: Core Concepts, Architecture, Configuration, and Usage in Real-Time Computing

This article provides a comprehensive technical overview of Kafka, covering its core concepts, producer and consumer models, architecture, configuration parameters, replication mechanisms, performance optimizations, operational monitoring, tooling scripts, and related product implementations for real-time data processing.

ArchitectureKafkaReal-Time Computing
0 likes · 18 min read
Kafka Overview: Core Concepts, Architecture, Configuration, and Usage in Real-Time Computing
JD Tech
JD Tech
Dec 11, 2018 · Big Data

Introduction to Graph Computing and the JoyGraph System

This article introduces graph computing, compares it with graph databases, surveys notable graph processing systems, and details the architecture, NUMA‑aware design, execution model, push/pull dual mode, and load‑balancing strategies of the JoyGraph framework while outlining its future development directions.

JoyGraphNUMAbig data
0 likes · 9 min read
Introduction to Graph Computing and the JoyGraph System
NetEase Game Operations Platform
NetEase Game Operations Platform
Dec 5, 2018 · Big Data

Presto + Alluxio Architecture for Interactive Ad‑hoc Queries in NetEase Game Data Warehouse

This article describes how NetEase Games built a Presto‑based interactive ad‑hoc query platform backed by Alluxio caching to achieve sub‑10‑second query latency, outlines the architectural design, performance comparisons with other Hadoop‑based solutions, encountered issues, and future improvement plans.

AlluxioPerformancead-hoc query
0 likes · 10 min read
Presto + Alluxio Architecture for Interactive Ad‑hoc Queries in NetEase Game Data Warehouse
AntTech
AntTech
Dec 4, 2018 · Artificial Intelligence

Highlights from the 7th China Small‑and‑Medium Bank Development Summit on FinTech and Risk Management (Nov 29‑30 2018, Guangzhou)

The 7th China Small‑and‑Medium Bank Development Summit held in Guangzhou on November 29‑30 2018 gathered over 200 banking and fintech leaders to discuss the latest trends, challenges, and strategies in financial technology, digital transformation, risk control, and emerging technologies such as AI, big data, cloud and blockchain.

Artificial IntelligenceCloud Computingbanking innovation
0 likes · 14 min read
Highlights from the 7th China Small‑and‑Medium Bank Development Summit on FinTech and Risk Management (Nov 29‑30 2018, Guangzhou)
DataFunTalk
DataFunTalk
Dec 4, 2018 · Artificial Intelligence

Application and Exploration of Financial Knowledge Graphs

This article presents a comprehensive overview of financial knowledge graphs, covering their historical evolution, theoretical foundations, technical stack, implementation steps, and real‑world case studies in banking, regulatory technology, and securities, while highlighting community resources for AI and big‑data practitioners.

AIKnowledge Extractionbig data
0 likes · 14 min read
Application and Exploration of Financial Knowledge Graphs
JD Tech
JD Tech
Nov 28, 2018 · Operations

Technical Systems Behind JD Logistics for the 11.11 Global Shopping Festival

The article details how JD Logistics’ extensive warehouse, routing, distribution, and fulfillment systems—leveraging big data, AI, GIS, IoT, and distributed architectures—were engineered and optimized to handle the massive order surge during the 11.11 Global Shopping Festival with high throughput, low latency, and zero incidents.

AIGISOperations
0 likes · 8 min read
Technical Systems Behind JD Logistics for the 11.11 Global Shopping Festival
DataFunTalk
DataFunTalk
Nov 24, 2018 · Big Data

The Evolution of iQIYI's Big Data Analytics Platform

This article chronicles iQIYI’s journey from a simple Hive‑based data pipeline to the sophisticated, multi‑engine “Tongtian Tower” platform, detailing the development of the Magic Mirror system, the Gear workflow manager, BabelBD, the Monet visual analytics tool, and the integrated BI ecosystem that now supports billions of daily users.

BIbig datadata engineering
0 likes · 18 min read
The Evolution of iQIYI's Big Data Analytics Platform
Tencent Cloud Developer
Tencent Cloud Developer
Nov 23, 2018 · Big Data

20 Free and Open-Source Data Visualization Tools

These 20 free and open‑source data visualization tools—from JavaScript libraries like D3.js and Chartist.js to user‑friendly platforms such as Datawrapper, Google Data Studio, and Tableau Public—enable businesses and analysts to transform raw data into interactive charts, maps, timelines, and dashboards, improving insight, decision‑making, and profitability.

Data AnalysisData VisualizationJavaScript libraries
0 likes · 12 min read
20 Free and Open-Source Data Visualization Tools
21CTO
21CTO
Nov 20, 2018 · Big Data

What Languages and Tools Do Big Data Experts Use? Insights from 31 IT Leaders

Based on interviews with 31 IT leaders from 28 organizations, this article reveals the most popular programming languages, frameworks, and platforms—such as Python, Scala, Spark, Kafka, TensorFlow, and Tableau—currently driving big‑data extraction, analysis, and reporting, and highlights emerging trends and tool preferences.

KafkaPythonSpark
0 likes · 12 min read
What Languages and Tools Do Big Data Experts Use? Insights from 31 IT Leaders
Architects' Tech Alliance
Architects' Tech Alliance
Nov 19, 2018 · Cloud Computing

Suning’s Cloud‑Era Digital Transformation: Architecture Evolution, Technology Roadmap, and Organizational Change

The article details Suning’s Internet‑plus transformation, describing its strategic “one body, two wings, three clouds, four ends” model, the evolution of its enterprise architecture across three generations, the adoption of cloud, SOA, micro‑services, big‑data and AI platforms, and the accompanying R&D and organizational reforms.

Cloud ComputingDigital TransformationMicroservices
0 likes · 13 min read
Suning’s Cloud‑Era Digital Transformation: Architecture Evolution, Technology Roadmap, and Organizational Change
21CTO
21CTO
Nov 7, 2018 · Big Data

Why Data Streams Are the Backbone of Real-Time Big Data Analytics

Data streams, akin to endless rivers, enable continuous, real-time processing of diverse sources such as IoT telemetry, web logs, and e-commerce events, offering advantages over batch processing, while presenting challenges like scalability and fault tolerance, and are supported by tools like Kinesis, Kafka, Flink, and Storm.

Amazon KinesisApache KafkaData Streams
0 likes · 6 min read
Why Data Streams Are the Backbone of Real-Time Big Data Analytics
JD Retail Technology
JD Retail Technology
Nov 7, 2018 · Cloud Computing

Technical Preparations for Double 11 Sales Event at JD.com

JD.com's commercial team conducted extensive technical preparations for the Double 11 sales event, including system optimizations, stress testing, and data platform enhancements to handle massive traffic and ensure system stability.

Double 11 PreparationTechnical Readinessbig data
0 likes · 6 min read
Technical Preparations for Double 11 Sales Event at JD.com
Programmer DD
Programmer DD
Nov 7, 2018 · Big Data

Choosing the Right SQL Engine for Big Data: A Practical Guide

This article explores various SQL engines and storage options for big‑data workloads, compares their performance and capabilities, shows practical code examples, and offers guidance on writing efficient SQL in complex data environments.

HiveSQLSQL Engines
0 likes · 6 min read
Choosing the Right SQL Engine for Big Data: A Practical Guide
Xianyu Technology
Xianyu Technology
Nov 6, 2018 · Big Data

Technical Evolution of Xianyu Real-Time Selection System for Double Eleven

To meet Double‑Eleven’s sub‑second, billion‑item feed demands, Alibaba’s Xianyu selection system evolved from a Solr‑based search pipeline through offline batch and PostgreSQL attempts to a Blink‑powered real‑time stream platform using Niagara’s low‑latency LSM storage, delivering high‑throughput, personalized product feeds.

AlibabaFlinkXianyu
0 likes · 23 min read
Technical Evolution of Xianyu Real-Time Selection System for Double Eleven
Architects' Tech Alliance
Architects' Tech Alliance
Nov 5, 2018 · Big Data

Alluxio as a Virtual Distributed File System for Data Lake Solutions

The article explains how Alluxio provides a virtual distributed file system that acts as a "virtual data lake," enabling unified, high‑performance access to structured and unstructured data across heterogeneous storage back‑ends while reducing storage costs through intelligent caching and eliminating the need for permanent data copies.

AlluxioCachingData Lake
0 likes · 16 min read
Alluxio as a Virtual Distributed File System for Data Lake Solutions
dbaplus Community
dbaplus Community
Nov 1, 2018 · Big Data

How Vipshop Scales Real‑Time Data with Flink on Kubernetes

This article details Vipshop's real‑time platform architecture, the migration from Storm and Spark to Flink, Flink's deployment on Kubernetes, and the latest Unified Data Management system that unifies data access across Kafka, Redis, Tair and HDFS.

FlinkKubernetesUDM
0 likes · 12 min read
How Vipshop Scales Real‑Time Data with Flink on Kubernetes
Tencent Cloud Developer
Tencent Cloud Developer
Oct 30, 2018 · Big Data

Big Data Technology Trends and Cloud Data Warehouse Architecture Practices

The article reviews recent big-data trends—from Hadoop’s evolution and Spark’s in-memory advances to emerging storage like Ozone—while detailing data-warehouse models, query-optimizer techniques, and cloud-native architectures that integrate diverse data sources, enabling scalable, AI-ready analytics and modern data-lake capabilities.

Cloud Data WarehouseData LakeDistributed Computing
0 likes · 30 min read
Big Data Technology Trends and Cloud Data Warehouse Architecture Practices
Alibaba Cloud Developer
Alibaba Cloud Developer
Oct 26, 2018 · Artificial Intelligence

How Alibaba’s Digital Vision Is Shaping AI‑Powered Smart Cities and New Retail

At the 2018 China Computer Conference, Alibaba CTO Zhang Jianfeng highlighted how digitalization underpins AI innovations across smart city infrastructure, new manufacturing, and retail, emphasizing massive data utilization, cognitive graph research, and immersive 3D store experiences.

AlibabaArtificial IntelligenceDigital Transformation
0 likes · 4 min read
How Alibaba’s Digital Vision Is Shaping AI‑Powered Smart Cities and New Retail
Qunar Tech Salon
Qunar Tech Salon
Oct 25, 2018 · Big Data

Why Alibaba Chose Apache Flink: Architecture, Scale, and Future Directions

This article explains how Alibaba adopted Apache Flink as a unified, low‑latency, high‑throughput big‑data engine, detailing its stream‑first design, state management, checkpointing, massive production deployment, community contributions, and upcoming plans for a unified API, SQL layer, broader language support, and AI integration.

AlibabaApache FlinkCheckpoint
0 likes · 13 min read
Why Alibaba Chose Apache Flink: Architecture, Scale, and Future Directions
DataFunTalk
DataFunTalk
Oct 24, 2018 · Artificial Intelligence

The Technical Growth Path of an Algorithm Engineer in the Big Data Era

This article summarizes Zeng Xianglin’s presentation on the stages of an algorithm engineer’s career—from academic Beta research and feature engineering through online deployment, model training, and deep‑learning applications—highlighting practical challenges and best practices in large‑scale advertising systems.

Algorithm EngineeringOnline Advertisingbig data
0 likes · 13 min read
The Technical Growth Path of an Algorithm Engineer in the Big Data Era
Programmer DD
Programmer DD
Oct 21, 2018 · Big Data

How to Choose the Right Number of Kafka Partitions for Optimal Throughput

This article explains how to determine the optimal Kafka partition count by balancing throughput gains, key‑based ordering requirements, file descriptor limits, and availability impacts, offering practical guidelines such as testing hardware limits and using broker‑count multiples for scalable deployments.

Throughputbig datapartitions
0 likes · 8 min read
How to Choose the Right Number of Kafka Partitions for Optimal Throughput
21CTO
21CTO
Oct 19, 2018 · Big Data

How Meituan Scales Real‑Time Computing with Flink: Architecture, Challenges & Solutions

This article summarizes Meituan’s real‑time computing platform, detailing its layered architecture built on Kafka, Flink on YARN, state management, resource isolation, fault tolerance, monitoring, and the Petra metric aggregation system, while highlighting the challenges faced and the solutions implemented to achieve high‑throughput, low‑latency stream processing at massive scale.

FlinkResource Isolationbig data
0 likes · 18 min read
How Meituan Scales Real‑Time Computing with Flink: Architecture, Challenges & Solutions
Tencent Cloud Developer
Tencent Cloud Developer
Oct 17, 2018 · Industry Insights

Why Graph Databases Are Redefining Enterprise Data Strategy

The article provides a detailed market and application analysis of graph databases, highlighting rapid growth, key use cases in finance and social networks, Tencent's StarGraph solution, advantages over relational databases, current limitations, and future industry adoption trends.

Cloud ComputingTencentbig data
0 likes · 6 min read
Why Graph Databases Are Redefining Enterprise Data Strategy
Xianyu Technology
Xianyu Technology
Oct 16, 2018 · Big Data

Millisecond-Level Counting for Billion-Scale Data via Offline Batch and Online Incremental Statistics

To achieve millisecond‑level counting on billion‑scale data, the Xianyu team replaced slow MySQL count queries with an offline batch that snapshots relational tables and computes totals, then uses KV‑store incremental statistics for online updates, delivering sub‑10 ms responses with near‑100 % success.

big datadatabaseincremental counting
0 likes · 7 min read
Millisecond-Level Counting for Billion-Scale Data via Offline Batch and Online Incremental Statistics
Java Backend Technology
Java Backend Technology
Oct 13, 2018 · Big Data

Check a New Integer Among 4 Billion Records in Seconds Using Bitmap & Distributed Methods

An interviewee faces the challenge of determining whether a newly given integer exists within a set of 4 billion numbers, and the article explores efficient solutions—from naive disk‑I/O approaches to distributed processing and the memory‑saving bitmap technique—highlighting their performance trade‑offs and implementation details.

BitmapDistributed ComputingLarge Dataset
0 likes · 6 min read
Check a New Integer Among 4 Billion Records in Seconds Using Bitmap & Distributed Methods
Alibaba Cloud Developer
Alibaba Cloud Developer
Oct 10, 2018 · Artificial Intelligence

How Alibaba’s Uni‑Marketing Boosted Brand Conversions with AI‑Driven Audience Selection

This article details Alibaba's Uni‑Marketing case study where a brand‑targeted audience selection algorithm, built on big‑data and AI techniques, improved the O→IPL deepening rate by 47% during the New‑Year Festival, outlining the technical pipeline, models, evaluation metrics, challenges, and future directions.

big databrand optimizationdigital marketing
0 likes · 20 min read
How Alibaba’s Uni‑Marketing Boosted Brand Conversions with AI‑Driven Audience Selection
Programmer DD
Programmer DD
Oct 6, 2018 · Big Data

Elastic Search IPO: What It Means for Search and Big Data

Elastic announced its IPO on the NYSE under ticker ESTC, highlighting its origins, rapid growth to over 5000 customers worldwide, a $160 million FY2018 revenue, and its Elastic Stack suite that powers search and analytics across industries, while investors celebrated the stock surge.

ElasticsearchIPObig data
0 likes · 6 min read
Elastic Search IPO: What It Means for Search and Big Data
JD Tech
JD Tech
Sep 29, 2018 · Artificial Intelligence

JD.com Prediction Technology: Architecture, Applications, and Future Directions

The article outlines JD.com's evolution of prediction technology from early book‑category sales forecasting to a comprehensive AI‑driven platform that supports sales, order, and GMV forecasts, describes its modular architecture and core algorithm choices, and discusses future enhancements for smarter supply‑chain collaboration.

big dataforecastingmachine learning
0 likes · 6 min read
JD.com Prediction Technology: Architecture, Applications, and Future Directions
Architects' Tech Alliance
Architects' Tech Alliance
Sep 26, 2018 · Operations

How Goldeneye Enables Adaptive, Intelligent Business Monitoring at Scale

Goldeneye, Alibaba Mom's monitoring platform, uses big‑data pipelines, dynamic threshold prediction, mean‑shift change‑point detection, and automated metric discovery to replace manual alarm settings, reduce false alerts, and provide intelligent, scalable business monitoring across hundreds of services.

Operationsbig databusiness monitoring
0 likes · 19 min read
How Goldeneye Enables Adaptive, Intelligent Business Monitoring at Scale
HomeTech
HomeTech
Sep 25, 2018 · Operations

Design and Implementation of an Integrated Log Collection, Analysis, and Monitoring System

This article describes how a rapidly growing technical team built a unified log system that consolidates program, web access, and slow logs, introduces host‑agent and process‑agent collection, leverages Kafka, Elasticsearch, and Storm for high‑throughput processing, and provides monitoring, alerting, and reporting features to improve reliability and operational efficiency.

ElasticsearchLog ManagementSystem Architecture
0 likes · 20 min read
Design and Implementation of an Integrated Log Collection, Analysis, and Monitoring System
Tencent Cloud Developer
Tencent Cloud Developer
Sep 20, 2018 · Industry Insights

How Big Data Drives Intelligent Outbound Calls and AI Customer Service

This article explains how a data‑driven platform combines big‑data preprocessing, behavior‑prediction models, and AI‑powered voice and text services to improve pre‑sale lead scoring, targeted SMS campaigns, and post‑sale customer support, using Tencent Cloud's TI One platform as a case study.

AI Customer ServiceMarketing Automationbehavior prediction
0 likes · 17 min read
How Big Data Drives Intelligent Outbound Calls and AI Customer Service
Tencent Cloud Developer
Tencent Cloud Developer
Sep 20, 2018 · Artificial Intelligence

What Everyone Should Know About Machine Learning

Machine learning lets computers learn patterns from examples instead of explicit code, enabling tasks like image and fraud detection, predictive maintenance, and personalized services, now feasible thanks to big data, cloud compute, and open-source tools, and increasingly discussed by executives for strategic automation.

Predictive Maintenancebig dataneural networks
0 likes · 11 min read
What Everyone Should Know About Machine Learning
Big Data and Microservices
Big Data and Microservices
Sep 17, 2018 · Big Data

5 Essential Data Mining Techniques Every Analyst Should Know

This article outlines five widely used data‑mining methods—association rules, classification/tagging, clustering, decision trees, and sequential pattern mining—explaining their principles, real‑world examples, and how they help organizations extract actionable insights from massive datasets.

ClusteringDecision TreesSequential Pattern Mining
0 likes · 6 min read
5 Essential Data Mining Techniques Every Analyst Should Know
Qunar Tech Salon
Qunar Tech Salon
Sep 14, 2018 · Big Data

AIGOV Five‑Star Model for Data Asset Management: Framework, Capabilities, and Enterprise Practices

The article presents the AIGOV five‑star data asset management model, analyzes its five management domains and thirteen capability items, compares it with domestic and international frameworks, and illustrates its practical value through detailed enterprise case studies and references to maturity models.

Maturity Modelbig datadata asset management
0 likes · 19 min read
AIGOV Five‑Star Model for Data Asset Management: Framework, Capabilities, and Enterprise Practices
Programmer DD
Programmer DD
Sep 13, 2018 · Big Data

How Deleting a Kafka Topic Removes Consumer Offsets and Why It Matters

This article examines a real‑world Kafka scenario where a topic is created, messages are produced and consumed, the topic is deleted, and then recreated, revealing that deleting the topic also removes its consumer offset metadata from the __consumer_offsets internal topic, causing new consumers to rely on their auto.offset.reset configuration.

Consumer OffsetsGroupCoordinatorTopic Deletion
0 likes · 6 min read
How Deleting a Kafka Topic Removes Consumer Offsets and Why It Matters
JD Tech
JD Tech
Sep 7, 2018 · Information Security

Big Data and AI Security Insights from ISC 2018 Conference

The ISC 2018 conference highlighted the growing importance of big data and artificial intelligence security, presenting JD's research on anti‑scraping techniques, AI‑driven defenses against black‑market attacks, and a service‑oriented approach to protecting user data across enterprises.

AI securityanti-scrapingbig data
0 likes · 5 min read
Big Data and AI Security Insights from ISC 2018 Conference
Tencent Cloud Developer
Tencent Cloud Developer
Sep 6, 2018 · Big Data

Real-Time Stream Computing: Concepts, Challenges, and Tencent Cloud Solutions

As mobile and IoT data surge, real-time stream computing—especially Flink’s low-latency, high-throughput, exactly-once engine—addresses challenges of latency, accuracy, and usability, and Tencent Cloud’s managed Flink service provides elastic, secure, integrated pipelines for applications ranging from online status monitoring to fraud detection and smart transportation.

Apache StormFlinkSpark Streaming
0 likes · 30 min read
Real-Time Stream Computing: Concepts, Challenges, and Tencent Cloud Solutions
Big Data and Microservices
Big Data and Microservices
Sep 4, 2018 · Big Data

Exploring Five Big Data Architectures—from Traditional to Unified AI Designs

The article examines the evolution of big‑data processing by comparing five prevalent architectures—traditional Hadoop‑based stacks, streaming‑only designs, Kappa, Lambda, and the unified Unifield model—highlighting their strengths, weaknesses, and suitable scenarios while discussing the limitations of classic BI systems and the role of distributed storage, computation, and machine‑learning integration.

HadoopKappaStreaming
0 likes · 14 min read
Exploring Five Big Data Architectures—from Traditional to Unified AI Designs
Big Data and Microservices
Big Data and Microservices
Aug 28, 2018 · Big Data

Turning Idle Hadoop Clusters into Valuable Data-Driven Products and Processes

The article examines how enterprises can transform big data from idle Hadoop clusters into valuable assets by adopting data-driven processes and products, outlining the distinction between technology-driven and business-driven approaches, describing data and service product models, and highlighting process optimization across various business functions.

Product Strategybig datadata-driven processes
0 likes · 7 min read
Turning Idle Hadoop Clusters into Valuable Data-Driven Products and Processes
Big Data and Microservices
Big Data and Microservices
Aug 26, 2018 · Big Data

Why Data, Not Process, Is the New Core of Business: 10 Big‑Data Principles Explained

The article outlines ten core big‑data principles—shifting from process‑centric to data‑centric thinking, emphasizing data value, efficiency, relevance, full‑sample analysis, prediction, information‑finding, machine understanding, e‑commerce intelligence, and mass customization—illustrated with real‑world examples and their impact on modern industry.

big datacorrelationdata‑centric
0 likes · 26 min read
Why Data, Not Process, Is the New Core of Business: 10 Big‑Data Principles Explained
DataFunTalk
DataFunTalk
Aug 21, 2018 · Artificial Intelligence

iQIYI Traffic Anti-Cheat: Techniques, System Architecture, and Future Directions

This article provides a comprehensive overview of iQIYI's traffic anti‑cheat mechanisms, covering definitions of fraudulent traffic, industry challenges, data cleaning relationships, system design, rule‑based and machine‑learning solutions, feature engineering, model evaluation, monitoring, service applications, and future prospects.

System ArchitectureTraffic analysisanti-fraud
0 likes · 11 min read
iQIYI Traffic Anti-Cheat: Techniques, System Architecture, and Future Directions
Meitu Technology
Meitu Technology
Aug 17, 2018 · Big Data

Meitu Distributed Bitmap System (Naix): Architecture, Implementation, and Performance Evaluation

Meitu’s Naix distributed bitmap system accelerates massive user‑data analytics by using a three‑layer architecture, sharded RoaringBitmap storage, and PalDB, delivering over 600× faster queries than Hive, supporting fast generation plugins, fault‑tolerant replication, and millisecond‑level RPC query responses while reducing storage by 67%.

BitmapData StorageNaix
0 likes · 16 min read
Meitu Distributed Bitmap System (Naix): Architecture, Implementation, and Performance Evaluation
Big Data and Microservices
Big Data and Microservices
Aug 16, 2018 · Big Data

Mastering Big Data Analysis: 5 Core Aspects and 4 Key Methods

This article outlines the five fundamental aspects of big data analysis—visualization, data‑mining algorithms, predictive analytics, semantic engines, and data quality management—and explains four primary analytical approaches: descriptive, diagnostic, predictive, and prescriptive analysis.

Data Analysisbig datadata mining
0 likes · 6 min read
Mastering Big Data Analysis: 5 Core Aspects and 4 Key Methods
Meitu Technology
Meitu Technology
Aug 14, 2018 · Big Data

Meitu Data Platform Architecture and Practices

Meitu’s data platform, serving dozens of apps with 500 million monthly active users and billions of daily events, combines the Arachnia log‑collection system, Kafka ingestion, multi‑layer storage (HDFS, MongoDB, HBase, Elasticsearch), offline Hive/MapReduce processing and real‑time Storm/Flink/Naix pipelines, supported by data‑workshop tools, staged evolution for scalability, and robust security and query‑validation mechanisms.

ETLHiveStreaming
0 likes · 16 min read
Meitu Data Platform Architecture and Practices
Alibaba Cloud Developer
Alibaba Cloud Developer
Aug 13, 2018 · Big Data

How Ele.me Evolved Its Real‑Time Engine: From Storm to Flink

This article examines Ele.me’s big‑data platform evolution, comparing Storm, Spark Streaming, Structured Streaming, and Flink, detailing their architectures, consistency semantics, performance trade‑offs, and why Flink became the preferred real‑time computation engine for the company.

FlinkSparkStorm
0 likes · 15 min read
How Ele.me Evolved Its Real‑Time Engine: From Storm to Flink
Meitu Technology
Meitu Technology
Aug 11, 2018 · Big Data

Meitu Technology Salon: Evolution of the Big Data Platform, Distributed Bitmap (Naix), and Apache Kylin

At Meitu’s Technology Salon, senior big‑data experts detailed the end‑to‑end architecture and stability measures of Meitu’s large‑scale data platform, introduced the high‑performance distributed bitmap solution Naix, showcased the evolution of Meizu’s user‑insight system, and highlighted Apache Kylin’s OLAP capabilities and Superset integration for scalable, real‑time analytics.

Apache KylinData Analyticsbig data
0 likes · 9 min read
Meitu Technology Salon: Evolution of the Big Data Platform, Distributed Bitmap (Naix), and Apache Kylin
Big Data and Microservices
Big Data and Microservices
Aug 10, 2018 · Big Data

5 Ways Big Data Empowers Modern Enterprises

Big data has become a critical asset for companies, enabling them to understand users, precisely locate resources, enhance marketing and operations, deliver refined services, and anticipate crises, thereby turning raw information into strategic advantage across multiple business functions.

Resource Optimizationbig datacrisis management
0 likes · 7 min read
5 Ways Big Data Empowers Modern Enterprises
iQIYI Technical Product Team
iQIYI Technical Product Team
Aug 10, 2018 · Big Data

Data-Driven Entertainment: iQIYI’s Big Data Platform and AI Applications

iQIYI’s unified “Tongtian Tower” big‑data platform integrates analytics, AI and open APIs to turn viewer behavior and public sentiment into market insights, personalized recommendations, smart casting and churn‑prediction tools, embedding a data‑driven culture that fuels its rapid subscriber growth and revenue surge.

AIbig datacontent recommendation
0 likes · 12 min read
Data-Driven Entertainment: iQIYI’s Big Data Platform and AI Applications
Architects' Tech Alliance
Architects' Tech Alliance
Aug 8, 2018 · Big Data

High‑Performance Data Analytics (HPDA): Architecture, Market Trends, and Fujitsu Reference Model

The article provides a comprehensive overview of High‑Performance Data Analytics (HPDA), detailing its market drivers, technical classifications, integration of HPC with big‑data workloads, Fujitsu's reference architecture, hardware configurations, benchmark results, and the economic benefits of deploying HPDA on existing HPC infrastructures.

FujitsuHPCHPDA
0 likes · 14 min read
High‑Performance Data Analytics (HPDA): Architecture, Market Trends, and Fujitsu Reference Model
Architecture Digest
Architecture Digest
Aug 7, 2018 · Big Data

Apache Kafka Overview, Architecture, and Sample Producer/Consumer Code

This article provides a comprehensive overview of Apache Kafka, comparing it with ActiveMQ, explaining its distributed architecture, topics, partitions, consumption models, high‑availability mechanisms, exactly‑once semantics, and includes detailed Java producer and consumer code examples for practical implementation.

ConsumerDistributed MessagingExactly-Once
0 likes · 22 min read
Apache Kafka Overview, Architecture, and Sample Producer/Consumer Code
Youzan Coder
Youzan Coder
Aug 3, 2018 · Big Data

Youzan Data Warehouse Metadata System: From Manual Tables to Metadata‑Driven Architecture

Youzan’s data‑warehouse metadata system evolved from manually maintained tables to an automated data dictionary and finally to a metadata‑driven architecture that automatically captures technical, business, and process metadata, visualizes lineage, tracks resource usage, manages synchronization rules and permissions, and now aims to improve novice usability with visual models and impact‑analysis tools.

HiveLineageResource Monitoring
0 likes · 11 min read
Youzan Data Warehouse Metadata System: From Manual Tables to Metadata‑Driven Architecture
Meituan Technology Team
Meituan Technology Team
Aug 2, 2018 · Big Data

R for Fine‑Grained Data Operations: Engineering Practices and Performance at Meituan

Meituan’s in‑store dining team demonstrates how R’s open‑source packages, powerful data manipulation, rich visualization libraries, and reproducible reporting can be engineered into scalable, parallelized workflows that turn secondary data processing into fast, interactive dashboards and analytics, proving R’s enterprise‑grade performance and adoption.

Data ProcessingData VisualizationR
0 likes · 18 min read
R for Fine‑Grained Data Operations: Engineering Practices and Performance at Meituan
MaGe Linux Operations
MaGe Linux Operations
Aug 2, 2018 · Big Data

Unlocking PUBG Victory: Data‑Driven Insights on Drop Zones, Final Circles, Weapons, and Kill Strategies

This article analyzes 18 million PUBG match records using Python to reveal optimal drop locations, high‑probability final‑circle spots, preferred weapons, and the relationship between kill distance, kill count, and winning chances, providing data‑driven strategies for players seeking more chicken dinners.

Data AnalysisGame AnalyticsPUBG
0 likes · 13 min read
Unlocking PUBG Victory: Data‑Driven Insights on Drop Zones, Final Circles, Weapons, and Kill Strategies
Efficient Ops
Efficient Ops
Aug 1, 2018 · Operations

How Tencent Revolutionized Monitoring: From IDC Crises to AI‑Driven AIOps

This talk by Tencent’s monitoring R&D lead outlines a decade of evolution in large‑scale monitoring, covering real‑world incident cases, the three drivers behind architectural upgrades, the implementation of a three‑dimensional monitoring framework, and the application of AI‑powered AIOps for precise, rapid anomaly detection.

AIOpsCloud ComputingOperations
0 likes · 18 min read
How Tencent Revolutionized Monitoring: From IDC Crises to AI‑Driven AIOps
Big Data and Microservices
Big Data and Microservices
Jul 29, 2018 · Industry Insights

Top 5 Big Data & AI Trends Shaping 2018 and Beyond

According to recent Forrester and Forbes reports, 2018 will see AI overtaking big-data hype, driving five key trends—from heightened cybersecurity in healthcare to expanded IoT, plug-and-play AI solutions, the rise of chief digital officers, and smarter community policing—each reshaping how organizations leverage data.

AI TrendsIoTbig data
0 likes · 8 min read
Top 5 Big Data & AI Trends Shaping 2018 and Beyond
Xianyu Technology
Xianyu Technology
Jul 28, 2018 · Big Data

Real-Time Computation Architecture for Non-Timeline Feed Ranking

The paper presents a real‑time computation architecture on Alibaba Cloud Blink that scores and ranks non‑timeline feed items within a sliding 72‑hour window, updating rankings every few minutes, using Redis ZSET for fast retrieval, and discusses scaling optimizations such as interval tuning and external join‑and‑rank services.

Real-Time ComputingRedisbig data
0 likes · 6 min read
Real-Time Computation Architecture for Non-Timeline Feed Ranking
Architects Research Society
Architects Research Society
Jul 27, 2018 · Big Data

Overview of Apache Hive Features, Usage, and Management

Apache Hive is an open‑source data‑warehouse system built on Hadoop that enables users to read, write, and manage large distributed datasets using SQL‑like queries, offering features such as ETL support, various file‑format connectors, extensible UDFs, and integration with tools like Tez, Spark, and MapReduce.

Apache HiveETLHadoop
0 likes · 5 min read
Overview of Apache Hive Features, Usage, and Management
58 Tech
58 Tech
Jul 27, 2018 · Big Data

Sun Dial: 58.com’s General‑Purpose AB Testing Platform – Architecture, Features, and Real‑Time Data Processing

The Sun Dial platform is a universal A/B testing system built for 58.com that supports single‑layer and multi‑layer experiments, provides uniform traffic splitting, real‑time OLAP analytics with Druid, and offers a web interface for easy configuration, enabling data‑driven product optimization across multiple business lines.

A/B testingDruidbig data
0 likes · 14 min read
Sun Dial: 58.com’s General‑Purpose AB Testing Platform – Architecture, Features, and Real‑Time Data Processing
Big Data and Microservices
Big Data and Microservices
Jul 26, 2018 · Industry Insights

How Big Data is Transforming the Financial Industry: Applications and Challenges

This article examines how big data technologies are reshaping banking, insurance, and securities by enabling customer profiling, precision marketing, risk management, and operational optimization, while also outlining the key challenges such as data quality, integration complexity, standards, and governance that the sector must overcome.

Data AnalyticsFinancial IndustryInsurance
0 likes · 19 min read
How Big Data is Transforming the Financial Industry: Applications and Challenges
Meituan Technology Team
Meituan Technology Team
Jul 26, 2018 · Backend Development

Evolution of Meituan Delivery System Architecture and Practices

Meituan Delivery’s architecture has progressed from a rapid MVP with coarse services to a scalable, fine‑grained platform comprising fulfillment, operation, and master‑data subsystems, employing reliability engineering, capacity planning, AI‑driven simulation, and location services to ensure high availability, efficiency, and future‑ready scalability.

AIMicroservicesSystem Architecture
0 likes · 16 min read
Evolution of Meituan Delivery System Architecture and Practices
JD Tech
JD Tech
Jul 24, 2018 · Databases

Understanding Graph Databases: Concepts, History, Use Cases, and Comparative Overview

This article explains what graph databases are, traces their evolution from early navigational models to modern distributed systems, highlights their core concepts and advantages over relational databases, showcases typical application scenarios, and provides a comparative overview of popular open‑source graph database engines to guide technology selection.

NoSQLbig datagraph database
0 likes · 8 min read
Understanding Graph Databases: Concepts, History, Use Cases, and Comparative Overview
ITPUB
ITPUB
Jul 23, 2018 · Big Data

What China's Vaccine Procurement Data Reveals: A Province‑Level Analysis

This article documents the collection, cleaning, and statistical analysis of publicly released second‑category vaccine procurement data from 28 Chinese provinces, highlighting data sources, processing steps with pandas, top manufacturers, regional market shares, and the challenges encountered during the effort.

ChinaData Analysisbig data
0 likes · 9 min read
What China's Vaccine Procurement Data Reveals: A Province‑Level Analysis
Tencent Cloud Developer
Tencent Cloud Developer
Jul 23, 2018 · Big Data

Analysis of Chinese Second-Class Vaccine Procurement Data

The study aggregates and cleans 2017‑2020 Chinese second‑class vaccine procurement data from 28 provinces into a 1,529‑record CSV, revealing a right‑skewed distribution where a handful of manufacturers—led by Beijing Kexing and Changchun Changsheng—account for the majority of entries, while noting gaps in several regions and encouraging further collaborative refinement.

Chinese healthcareData Analysisbig data
0 likes · 10 min read
Analysis of Chinese Second-Class Vaccine Procurement Data
Alibaba Cloud Developer
Alibaba Cloud Developer
Jul 23, 2018 · Big Data

How Alibaba’s MaxCompute Became the Backbone of 99% Data Processing

This article reviews Alibaba's MaxCompute evolution from ODPS to a unified, multi‑cluster big‑data platform, detailing its architecture, development tools, large‑scale deployments, performance optimizations, typical workload scenarios, and why it is the preferred choice for enterprise data processing.

Alibaba CloudDistributed ComputingMaxCompute
0 likes · 22 min read
How Alibaba’s MaxCompute Became the Backbone of 99% Data Processing
Youzan Coder
Youzan Coder
Jul 20, 2018 · Big Data

How Youzan Built a Scalable Big Data Development Platform (DP)

This article details the design, architecture, and operational experience of Youzan's Data Platform (DP), covering its scheduling, data‑sync, service, and monitoring modules, the custom Airflow‑based task scheduler, current production metrics, supported task types, and future improvement plans.

Airflowbig datadata platform
0 likes · 12 min read
How Youzan Built a Scalable Big Data Development Platform (DP)