Skip to content
Tagged articles

Data Engineering

331 articles · Page 4 of 4
Meitu Technology
Meitu Technology
Aug 14, 2018 · Big Data

Meitu Data Platform Architecture and Practices

Meitu’s data platform, serving dozens of apps with 500 million monthly active users and billions of daily events, combines the Arachnia log‑collection system, Kafka ingestion, multi‑layer storage (HDFS, MongoDB, HBase, Elasticsearch), offline Hive/MapReduce processing and real‑time Storm/Flink/Naix pipelines, supported by data‑workshop tools, staged evolution for scalability, and robust security and query‑validation mechanisms.

Data EngineeringETLHive
0 likes · 16 min read
Meitu Data Platform Architecture and Practices
Meitu Technology
Meitu Technology
Aug 11, 2018 · Big Data

Meitu Technology Salon: Evolution of the Big Data Platform, Distributed Bitmap (Naix), and Apache Kylin

At Meitu’s Technology Salon, senior big‑data experts detailed the end‑to‑end architecture and stability measures of Meitu’s large‑scale data platform, introduced the high‑performance distributed bitmap solution Naix, showcased the evolution of Meizu’s user‑insight system, and highlighted Apache Kylin’s OLAP capabilities and Superset integration for scalable, real‑time analytics.

Apache KylinData Engineeringbig data
0 likes · 9 min read
Meitu Technology Salon: Evolution of the Big Data Platform, Distributed Bitmap (Naix), and Apache Kylin
Architecture Digest
Architecture Digest
Jul 6, 2018 · Backend Development

Essential Backend Infrastructure and Services for Java Applications

This article outlines the fundamental backend components, frameworks, and services—including API gateways, authentication centers, configuration management, service governance, scheduling, logging, data pipelines, and monitoring—required to build robust, scalable Java business applications for both online and internal use.

API GatewayBackendData Engineering
0 likes · 20 min read
Essential Backend Infrastructure and Services for Java Applications
AntTech
AntTech
Apr 28, 2018 · Artificial Intelligence

The Future of AI: Intelligent Infrastructure, Engineering Challenges, and the Limits of Current Approaches

The article examines Michael I. Jordan’s critique of current AI research, highlights the need for intelligent infrastructure that integrates computing, data, and physical systems across domains, and uses real‑world examples to argue for a new engineering discipline beyond narrow deep‑learning hype.

AIData Engineeringfuture of AI
0 likes · 23 min read
The Future of AI: Intelligent Infrastructure, Engineering Challenges, and the Limits of Current Approaches
Huawei Cloud Developer Alliance
Huawei Cloud Developer Alliance
Apr 10, 2018 · Big Data

Unlock Big Data Expertise with Huawei’s HCDA‑BigData V1.0 Certification

Huawei’s HCDA‑BigData V1.0 certification, launched in April 2018, offers engineer‑level credentials, comprehensive training materials, a four‑day hands‑on program, and a Prometric exam to equip developers, partners, students, and ICT professionals with advanced big‑data development skills on FusionInsight.

Data EngineeringFusionInsightHCDA
0 likes · 5 min read
Unlock Big Data Expertise with Huawei’s HCDA‑BigData V1.0 Certification
Meituan Technology Team
Meituan Technology Team
Jan 26, 2018 · Big Data

Design and Implementation of a Real-Time Data Processing System at Meituan

Meituan designed a Storm‑based real‑time data processing platform that guarantees at‑least‑once delivery and high availability, employs a custom spout, regression‑driven traffic smoothing, and a low‑latency KV store with atomic operations, persisting results in Kafka, MySQL and Cellar to power merchant dashboards and heat‑tag analytics, while planning broader real‑time analytics expansion.

Data EngineeringDistributed SystemsStorm
0 likes · 10 min read
Design and Implementation of a Real-Time Data Processing System at Meituan
StarRing Big Data Open Lab
StarRing Big Data Open Lab
Jan 5, 2018 · Big Data

What Drove Big Data’s 2017 Surge and What’s Next? Insights & Predictions

Analyzing 2017’s big data boom, the article explores how the 4V characteristics—volume, variety, velocity, and value—spurred innovations like distributed storage, NoSQL, real‑time stream processing, and AI integration, and predicts future hotspots such as SQL resurgence, cloud‑based platforms, and AI‑driven analytics.

Data Engineeringartificial intelligencebig data
0 likes · 11 min read
What Drove Big Data’s 2017 Surge and What’s Next? Insights & Predictions
Alibaba Cloud Developer
Alibaba Cloud Developer
Oct 23, 2017 · Big Data

How Alibaba Built Its Full‑Domain Data Platform: Architecture & Lessons

In this detailed account, Alibaba senior technologist Zhang Lei explains the concept of a full‑domain data platform, its “four‑horizontal and three‑vertical” architecture, the OneData ecosystem, cost‑saving strategies, data quality tools, and practical challenges of building and operating massive big‑data infrastructure across the Alibaba ecosystem.

AlibabaData Engineeringdata middle platform
0 likes · 15 min read
How Alibaba Built Its Full‑Domain Data Platform: Architecture & Lessons
ITPUB
ITPUB
Sep 29, 2017 · Big Data

Designing an Open ETL System: Baidu Waimai’s Scalable Data Pipeline Practices

In this talk, a Baidu Waimai engineer explains the motivations, requirements, and architectural choices behind their open‑source ETL platform, covering data flow patterns, logical mappings, storage options, scheduling, metadata management, and quality monitoring to achieve scalable, transparent, and explainable data delivery.

Data EngineeringData PipelineETL
0 likes · 26 min read
Designing an Open ETL System: Baidu Waimai’s Scalable Data Pipeline Practices
Architects Research Society
Architects Research Society
Jul 23, 2017 · Big Data

Overview of 50 Additional Big Data Terms and Apache Projects

This article provides an extensive English overview of fifty additional big-data terminology and Apache open-source projects, explaining concepts such as Kafka, Hive, Spark, data cleaning, AI, graph databases, and many other tools and techniques essential for modern data engineering and analytics.

Data Engineeringdata analytics
0 likes · 20 min read
Overview of 50 Additional Big Data Terms and Apache Projects
21CTO
21CTO
Jul 17, 2017 · Artificial Intelligence

Inside 58.com’s Smart Recommendation Engine: Architecture, Algorithms, Data

58.com’s intelligent recommendation system, evolving from a C++ monolith in 2014 to a Java-based micro‑service platform, integrates multi‑layer data processing, diverse recall and ranking algorithms, and a robust microservice architecture to deliver personalized listings across housing, jobs, cars, and more.

Data EngineeringRankingmicroservices
0 likes · 27 min read
Inside 58.com’s Smart Recommendation Engine: Architecture, Algorithms, Data
21CTO
21CTO
Jul 7, 2017 · Big Data

How to Kickstart Your Big Data Career: A Complete Learning Roadmap

This guide walks beginners through the vast big data landscape, helping them choose the right role, understand essential terminology, plan a learning path, and access curated resources for becoming a data engineer or analyst, all illustrated with clear diagrams.

Career GuideData Engineeringbig data
0 likes · 16 min read
How to Kickstart Your Big Data Career: A Complete Learning Roadmap
21CTO
21CTO
Jun 9, 2017 · Big Data

From Hadoop to Spark: A Complete Roadmap to Becoming a Big Data Architect

This guide walks beginners through the essential big‑data ecosystem—from understanding Hadoop’s core components and mastering MapReduce, to using Hive, SparkSQL, Kafka, and real‑time frameworks like Storm, while also covering data ingestion, export, scheduling, and introductory machine‑learning techniques.

Data EngineeringHiveSpark
0 likes · 20 min read
From Hadoop to Spark: A Complete Roadmap to Becoming a Big Data Architect
MaGe Linux Operations
MaGe Linux Operations
May 31, 2017 · Big Data

Essential Skills for a Successful Data Career: From Big Data Platforms to AI

This article outlines the critical competencies needed across the data field—from building and maintaining big data platforms and data warehouses to mastering visualization, analysis, mining, and deep learning—offering practical guidance for aspiring data professionals seeking long‑term career growth.

Career GuideData Engineeringdata science
0 likes · 15 min read
Essential Skills for a Successful Data Career: From Big Data Platforms to AI
Qunar Tech Salon
Qunar Tech Salon
Mar 12, 2017 · Big Data

Essential Skills and Career Paths for Data Professionals: From Big Data Platforms to AI

The article outlines the key competencies, responsibilities, and career development advice for data professionals across the entire data stack—from building big‑data platforms and data warehouses to visualization, analysis, algorithm engineering, and deep‑learning applications—emphasizing the importance of creating business value with data.

Data AnalystData EngineeringDeep Learning
0 likes · 15 min read
Essential Skills and Career Paths for Data Professionals: From Big Data Platforms to AI
Ctrip Technology
Ctrip Technology
Mar 8, 2017 · Big Data

Essential Skills and Career Path for Data Professionals: From Big Data Platforms to AI Applications

This article outlines the key competencies and career roadmap for data professionals, covering big‑data infrastructure, data‑warehouse engineering, visualization, analysis, algorithmic mining, and deep‑learning, while emphasizing the importance of business sense, cloud adoption, and continuous learning.

Data Engineeringcareer advicedata analysis
0 likes · 15 min read
Essential Skills and Career Path for Data Professionals: From Big Data Platforms to AI Applications
Architecture Digest
Architecture Digest
Dec 26, 2016 · Big Data

My Journey into Big Data: From Early Mistakes to the Lambda Architecture

The article recounts the author’s early encounters with big‑data challenges, the shift from relational to NoSQL systems, the development of an immutable‑data batch architecture, and the eventual formulation of the Lambda Architecture, illustrating how simplicity and fault‑tolerance can replace complex incremental designs.

Data EngineeringImmutable DataLambda architecture
0 likes · 9 min read
My Journey into Big Data: From Early Mistakes to the Lambda Architecture
Alibaba Cloud Developer
Alibaba Cloud Developer
Dec 8, 2016 · Big Data

How Alibaba’s Double 11 Media Dashboard Leverages Cutting‑Edge Data Technology

Alibaba senior technologist Luo Jinpeng explains how the company’s Double 11 media big‑screen platform evolved from chaotic early implementations to a robust, real‑time data system, detailing architectural redesigns, model refactoring, resource scheduling, and link reliability strategies that underpin its massive e‑commerce event.

AlibabaData Engineering
0 likes · 2 min read
How Alibaba’s Double 11 Media Dashboard Leverages Cutting‑Edge Data Technology
ITPUB
ITPUB
Aug 15, 2016 · Big Data

5 Commandments to Bridge the Gap Between Data Scientists and Engineers

This article outlines five practical commandments that help data scientists and data engineers collaborate more effectively, covering data awareness, tool familiarity, technical limits, mutual respect, and shared responsibility to ensure smooth project delivery.

Best PracticesCollaborationData Engineering
0 likes · 9 min read
5 Commandments to Bridge the Gap Between Data Scientists and Engineers
Architecture Digest
Architecture Digest
Aug 15, 2016 · Big Data

Understanding Data: Types, Systems, and Big Data Technologies

This article explains what data is, classifies it into structured, semi‑structured and unstructured forms, describes data mining, databases, data warehouses, the full data lifecycle, and surveys the big‑data ecosystem including storage, batch and real‑time processing, resource scheduling, and visualization technologies.

Data EngineeringLambda architecturedata mining
0 likes · 22 min read
Understanding Data: Types, Systems, and Big Data Technologies
Liulishuo Tech Team
Liulishuo Tech Team
Jun 17, 2016 · Big Data

Building a Scalable Big Data Platform on AWS: Architecture and Execution Service Design

This article details the architectural design and implementation of a scalable big data platform built on AWS services, highlighting the transition from HDFS to S3 for storage, the use of EMR for elastic compute, and a custom Execution Service integrated with Consul and Airflow for automated cluster management and task scheduling.

AWS EMRAirflowData Engineering
0 likes · 11 min read
Building a Scalable Big Data Platform on AWS: Architecture and Execution Service Design
21CTO
21CTO
Apr 8, 2016 · Cloud Computing

Inside Airbnb’s AWS Cloud Architecture and Data Stack

Airbnb’s engineering VP Mike Curtis explains how the company leverages Amazon Web Services, a Hadoop‑based big‑data platform, and custom tools like Aerosolve, Airflow, and Airpal to power its global marketplace, enabling rapid scaling, dynamic pricing, and personalized search through extensive cloud infrastructure and machine‑learning pipelines.

AWSCloud ArchitectureData Engineering
0 likes · 15 min read
Inside Airbnb’s AWS Cloud Architecture and Data Stack
Java High-Performance Architecture
Java High-Performance Architecture
Mar 13, 2016 · Big Data

Essential Big Data Skill Map: Tools, Languages, and Techniques You Need

Explore a comprehensive big data skill map covering processing frameworks like Spark and Hadoop, databases, programming languages, analytics tools, visualization libraries, AI techniques, algorithms, data structures, and cloud computing services, providing a practical reference for building expertise in modern data engineering.

Data Engineeringcloud computing
0 likes · 3 min read
Essential Big Data Skill Map: Tools, Languages, and Techniques You Need
Architecture Digest
Architecture Digest
Feb 22, 2016 · Big Data

Building High‑Performance Big Data Analytics Systems: Techniques and Best Practices

An in‑depth guide outlines technology‑agnostic best‑practice techniques for building high‑performance big data analytics systems, covering data acquisition, storage, processing, visualization, and security, and explains how to address the five V’s of big data to meet demanding operational and performance requirements.

AnalyticsData Engineeringbig data
0 likes · 20 min read
Building High‑Performance Big Data Analytics Systems: Techniques and Best Practices
Efficient Ops
Efficient Ops
Dec 9, 2015 · Big Data

Big Data Lessons from Baidu: Pitfalls, Language Choices, and NewSQL Insights

In this expert Q&A, Baidu’s senior big-data specialists reveal common project pitfalls, argue for Java in Hadoop-style systems, discuss MongoDB deployment, outline criteria for choosing open-source versus self-built solutions, and evaluate the viability of NewSQL/Spanner-type startups.

Data EngineeringHadoopMongoDB
0 likes · 8 min read
Big Data Lessons from Baidu: Pitfalls, Language Choices, and NewSQL Insights
21CTO
21CTO
Nov 20, 2015 · Artificial Intelligence

How Meituan Builds and Optimizes Its Recommendation System

This article explains Meituan's end‑to‑end recommendation system architecture, data processing pipeline, candidate generation strategies, model training and online ranking techniques, illustrating how data, algorithms, and real‑time signals are combined to improve relevance and conversion.

AIData EngineeringMeituan
0 likes · 19 min read
How Meituan Builds and Optimizes Its Recommendation System
21CTO
21CTO
Nov 4, 2015 · Big Data

Evolution of Dazhong Dianping’s Data Platform (2012‑2014): Key Lessons for Growing Big Data Teams

This article chronicles the step‑by‑step evolution of Dazhong Dianping’s data platform from 2012 to 2014, detailing changes in data models, storage and compute architecture, scheduling, monitoring, and data‑driven applications, offering practical insights for teams building early‑stage big‑data infrastructures.

Data EngineeringETLbig data architecture
0 likes · 7 min read
Evolution of Dazhong Dianping’s Data Platform (2012‑2014): Key Lessons for Growing Big Data Teams
MaGe Linux Operations
MaGe Linux Operations
Aug 20, 2015 · Big Data

15 Must‑Try Resources to Master Hadoop Quickly

This article explains what Hadoop is, outlines its key features, and presents a curated list of 15 high‑quality tutorials, video courses, and books to help beginners and professionals efficiently learn Hadoop and its MapReduce ecosystem.

Data EngineeringHadoopLearning Resources
0 likes · 12 min read
15 Must‑Try Resources to Master Hadoop Quickly