Meitu’s data platform, serving dozens of apps with 500 million monthly active users and billions of daily events, combines the Arachnia log‑collection system, Kafka ingestion, multi‑layer storage (HDFS, MongoDB, HBase, Elasticsearch), offline Hive/MapReduce processing and real‑time Storm/Flink/Naix pipelines, supported by data‑workshop tools, staged evolution for scalability, and robust security and query‑validation mechanisms.
At Meitu’s Technology Salon, senior big‑data experts detailed the end‑to‑end architecture and stability measures of Meitu’s large‑scale data platform, introduced the high‑performance distributed bitmap solution Naix, showcased the evolution of Meizu’s user‑insight system, and highlighted Apache Kylin’s OLAP capabilities and Superset integration for scalable, real‑time analytics.
This article outlines the fundamental backend components, frameworks, and services—including API gateways, authentication centers, configuration management, service governance, scheduling, logging, data pipelines, and monitoring—required to build robust, scalable Java business applications for both online and internal use.
The article examines Michael I. Jordan’s critique of current AI research, highlights the need for intelligent infrastructure that integrates computing, data, and physical systems across domains, and uses real‑world examples to argue for a new engineering discipline beyond narrow deep‑learning hype.
This article traces the evolution of Weibo's recommendation system architecture across three major phases—independent 1.0, layered 2.0, and platform‑centric 3.0—detailing the environmental drivers, technical components, advantages, drawbacks, and key outcomes of each stage.
Huawei’s HCDA‑BigData V1.0 certification, launched in April 2018, offers engineer‑level credentials, comprehensive training materials, a four‑day hands‑on program, and a Prometric exam to equip developers, partners, students, and ICT professionals with advanced big‑data development skills on FusionInsight.
Meituan designed a Storm‑based real‑time data processing platform that guarantees at‑least‑once delivery and high availability, employs a custom spout, regression‑driven traffic smoothing, and a low‑latency KV store with atomic operations, persisting results in Kafka, MySQL and Cellar to power merchant dashboards and heat‑tag analytics, while planning broader real‑time analytics expansion.
Analyzing 2017’s big data boom, the article explores how the 4V characteristics—volume, variety, velocity, and value—spurred innovations like distributed storage, NoSQL, real‑time stream processing, and AI integration, and predicts future hotspots such as SQL resurgence, cloud‑based platforms, and AI‑driven analytics.
Alibaba’s Double 11 showcase reveals how its upgraded stream computing platform handled a 100% year‑over‑year data surge, achieving 256 K successful payments per second and processing 472 million events per second in real time through a highly optimized Flink‑based architecture.
In this detailed account, Alibaba senior technologist Zhang Lei explains the concept of a full‑domain data platform, its “four‑horizontal and three‑vertical” architecture, the OneData ecosystem, cost‑saving strategies, data quality tools, and practical challenges of building and operating massive big‑data infrastructure across the Alibaba ecosystem.
In this talk, a Baidu Waimai engineer explains the motivations, requirements, and architectural choices behind their open‑source ETL platform, covering data flow patterns, logical mappings, storage options, scheduling, metadata management, and quality monitoring to achieve scalable, transparent, and explainable data delivery.
This article introduces Flink’s background, architecture, and ecosystem, then demonstrates a step‑by‑step tutorial on using Flink SQL to clean and transform streaming data from Kafka, highlighting its advantages over Storm for real‑time ETL.
This article provides an extensive English overview of fifty additional big-data terminology and Apache open-source projects, explaining concepts such as Kafka, Hive, Spark, data cleaning, AI, graph databases, and many other tools and techniques essential for modern data engineering and analytics.
58.com’s intelligent recommendation system, evolving from a C++ monolith in 2014 to a Java-based micro‑service platform, integrates multi‑layer data processing, diverse recall and ranking algorithms, and a robust microservice architecture to deliver personalized listings across housing, jobs, cars, and more.
This guide walks beginners through the vast big data landscape, helping them choose the right role, understand essential terminology, plan a learning path, and access curated resources for becoming a data engineer or analyst, all illustrated with clear diagrams.
This guide walks beginners through the essential big‑data ecosystem—from understanding Hadoop’s core components and mastering MapReduce, to using Hive, SparkSQL, Kafka, and real‑time frameworks like Storm, while also covering data ingestion, export, scheduling, and introductory machine‑learning techniques.
This article outlines the critical competencies needed across the data field—from building and maintaining big data platforms and data warehouses to mastering visualization, analysis, mining, and deep learning—offering practical guidance for aspiring data professionals seeking long‑term career growth.
The article outlines the key competencies, responsibilities, and career development advice for data professionals across the entire data stack—from building big‑data platforms and data warehouses to visualization, analysis, algorithm engineering, and deep‑learning applications—emphasizing the importance of creating business value with data.
This article outlines the key competencies and career roadmap for data professionals, covering big‑data infrastructure, data‑warehouse engineering, visualization, analysis, algorithmic mining, and deep‑learning, while emphasizing the importance of business sense, cloud adoption, and continuous learning.
The article recounts the author’s early encounters with big‑data challenges, the shift from relational to NoSQL systems, the development of an immutable‑data batch architecture, and the eventual formulation of the Lambda Architecture, illustrating how simplicity and fault‑tolerance can replace complex incremental designs.
Alibaba senior technologist Luo Jinpeng explains how the company’s Double 11 media big‑screen platform evolved from chaotic early implementations to a robust, real‑time data system, detailing architectural redesigns, model refactoring, resource scheduling, and link reliability strategies that underpin its massive e‑commerce event.
This article outlines five practical commandments that help data scientists and data engineers collaborate more effectively, covering data awareness, tool familiarity, technical limits, mutual respect, and shared responsibility to ensure smooth project delivery.
This article explains what data is, classifies it into structured, semi‑structured and unstructured forms, describes data mining, databases, data warehouses, the full data lifecycle, and surveys the big‑data ecosystem including storage, batch and real‑time processing, resource scheduling, and visualization technologies.
This article details the architectural design and implementation of a scalable big data platform built on AWS services, highlighting the transition from HDFS to S3 for storage, the use of EMR for elastic compute, and a custom Execution Service integrated with Consul and Airflow for automated cluster management and task scheduling.
Airbnb’s engineering VP Mike Curtis explains how the company leverages Amazon Web Services, a Hadoop‑based big‑data platform, and custom tools like Aerosolve, Airflow, and Airpal to power its global marketplace, enabling rapid scaling, dynamic pricing, and personalized search through extensive cloud infrastructure and machine‑learning pipelines.
Explore a comprehensive big data skill map covering processing frameworks like Spark and Hadoop, databases, programming languages, analytics tools, visualization libraries, AI techniques, algorithms, data structures, and cloud computing services, providing a practical reference for building expertise in modern data engineering.
An in‑depth guide outlines technology‑agnostic best‑practice techniques for building high‑performance big data analytics systems, covering data acquisition, storage, processing, visualization, and security, and explains how to address the five V’s of big data to meet demanding operational and performance requirements.
In this expert Q&A, Baidu’s senior big-data specialists reveal common project pitfalls, argue for Java in Hadoop-style systems, discuss MongoDB deployment, outline criteria for choosing open-source versus self-built solutions, and evaluate the viability of NewSQL/Spanner-type startups.
This article explains Meituan's end‑to‑end recommendation system architecture, data processing pipeline, candidate generation strategies, model training and online ranking techniques, illustrating how data, algorithms, and real‑time signals are combined to improve relevance and conversion.
This article chronicles the step‑by‑step evolution of Dazhong Dianping’s data platform from 2012 to 2014, detailing changes in data models, storage and compute architecture, scheduling, monitoring, and data‑driven applications, offering practical insights for teams building early‑stage big‑data infrastructures.
This article explains what Hadoop is, outlines its key features, and presents a curated list of 15 high‑quality tutorials, video courses, and books to help beginners and professionals efficiently learn Hadoop and its MapReduce ecosystem.