Skip to content
Tagged articles

Data Engineering

331 articles · Page 2 of 4
Xiaohongshu Tech REDtech
Xiaohongshu Tech REDtech
Dec 5, 2024 · Big Data

Interview with Jianchen: Journey from Open Source Contributor to Data Engineer at Xiaohongshu

In this interview, Xiaohongshu data engineer Jianchen recounts his evolution from a computer‑science student discovering open‑source through MIT6.824 to contributing to SOFAJRaft and Apache RocketMQ, detailing his OSPP projects, the decision to join Xiaohongshu, and his work on a cloud‑native Kafka engine that cut storage and compute usage by half.

Apache RocketMQCloud NativeData Engineering
0 likes · 11 min read
Interview with Jianchen: Journey from Open Source Contributor to Data Engineer at Xiaohongshu
DataFunSummit
DataFunSummit
Dec 5, 2024 · Big Data

Ping An Financial Services' Big Data Platform Construction and Data Governance Practices

This article details Ping An Financial Services' journey in building a comprehensive big‑data platform, addressing fragmentation, low data timeliness, processing limits, and governance challenges through a four‑stage technical evolution, modular tool development, and a systematic data‑governance framework to support its digital transformation.

Data EngineeringFinancial Servicesdata governance
0 likes · 16 min read
Ping An Financial Services' Big Data Platform Construction and Data Governance Practices
ByteDance Data Platform
ByteDance Data Platform
Nov 6, 2024 · Big Data

How Douyin’s Data Platform Overcomes EB‑Scale Metric Challenges

This article explains how Douyin Group tackles massive data volume, quality, and efficiency issues by building a four‑layer intelligent platform, standardizing metric management, automating metric decomposition, and creating reusable metric services that boost agility, stability, and cross‑team collaboration.

Data Engineeringbig datadata platform
0 likes · 20 min read
How Douyin’s Data Platform Overcomes EB‑Scale Metric Challenges
Bilibili Tech
Bilibili Tech
Oct 25, 2024 · Big Data

DataFunSummit2024: Next-Generation Data Architecture Technology Summit

DataFunSummit2024, co-hosted by Bilibili, convenes industry experts, scholars, and enterprise leaders across six forums to discuss next‑generation data architecture, showcasing Bilibili’s Iceberg‑based stream‑batch innovations, AI‑BI analytics, NoETL practices, and emerging alternatives to Lambda architecture.

AI+BIData EngineeringIceberg
0 likes · 3 min read
DataFunSummit2024: Next-Generation Data Architecture Technology Summit
Baobao Algorithm Notes
Baobao Algorithm Notes
Oct 7, 2024 · Artificial Intelligence

Mastering LLM Supervised Fine‑Tuning: Practical Tips, Data Strategies, and Debugging

This article provides a comprehensive, experience‑driven guide to supervised fine‑tuning (SFT) of large language models, covering special tokens, latency considerations, data diversity and production, training frameworks and hyper‑parameters, over‑/under‑fitting diagnostics, and evaluation metrics such as helpfulness, honesty, and harmlessness.

AIData EngineeringLLM
0 likes · 40 min read
Mastering LLM Supervised Fine‑Tuning: Practical Tips, Data Strategies, and Debugging
AntData
AntData
Sep 26, 2024 · Artificial Intelligence

DB-GPT: Open-Source AI-Native Data Application Development Framework

DB‑GPT is an open‑source AI‑native data‑application framework that provides multi‑model management, Text‑to‑SQL optimization, RAG, multi‑agent collaboration, and intelligent workflow orchestration, enabling developers to build scalable large‑model database applications, with proven enterprise adoption, community growth, and academic publications.

AIData EngineeringLarge Language Models
0 likes · 6 min read
DB-GPT: Open-Source AI-Native Data Application Development Framework
JD Retail Technology
JD Retail Technology
Sep 25, 2024 · Big Data

From a Personal Journey to Data Platform Architecture: Insights on Big Data, Cloud Computing, and System Design

The article narrates the author’s 30‑year programming career and shares technical reflections on building business‑agnostic, configurable data platforms, covering batch, streaming, interactive computing, big‑data sharding, Spark, Flink, cloud migration, and the philosophy of software architecture.

Data Engineeringbatch processingcloud computing
0 likes · 23 min read
From a Personal Journey to Data Platform Architecture: Insights on Big Data, Cloud Computing, and System Design
AntTech
AntTech
Sep 10, 2024 · Big Data

From DATA for AI to AI for DATA: Evolution of Ant Group’s Intelligent Data System

The talk reviews the rapid evolution of data technologies—from early database foundations and big‑data breakthroughs to the rise of generative AI—highlighting how Ant Group’s data platform is shifting from a cost‑efficiency focus to a value‑centric, multimodal, AI‑driven ecosystem.

Data EngineeringData PlatformsMultimodal Data
0 likes · 17 min read
From DATA for AI to AI for DATA: Evolution of Ant Group’s Intelligent Data System
AntData
AntData
Sep 9, 2024 · Big Data

From Cost‑Efficiency to Value‑Centric: The Evolution of Data Systems in the Data+AI Era

The article reviews the rapid advances in generative AI and big‑data technologies, traces the historical development of data infrastructure, and argues that modern data systems are shifting from a cost‑efficiency focus to a value‑centric paradigm driven by multimodal, non‑structured data, vector search and machine‑oriented services.

Data Engineeringartificial intelligencebig data
0 likes · 18 min read
From Cost‑Efficiency to Value‑Centric: The Evolution of Data Systems in the Data+AI Era
StarRocks
StarRocks
Sep 5, 2024 · Big Data

Accelerate Lakehouse Queries: A Hands‑On Guide to StarRocks + Apache Iceberg

This tutorial walks you through the fundamentals of Apache Iceberg, its architecture and key features, explains why it’s advantageous for lakehouse workloads, and provides a step‑by‑step Docker‑Compose setup to integrate Iceberg with StarRocks for fast, ACID‑compliant analytics on real‑world taxi data.

Apache IcebergData EngineeringPySpark
0 likes · 15 min read
Accelerate Lakehouse Queries: A Hands‑On Guide to StarRocks + Apache Iceberg
Mike Chen's Internet Architecture
Mike Chen's Internet Architecture
Aug 16, 2024 · Big Data

Understanding the Lambda Architecture for Big Data Processing

This article explains the Lambda architecture—a three‑layer model combining batch and real‑time processing for large‑scale data, outlines its components, advantages, disadvantages, common tools, and compares it with the Kappa alternative while providing practical insights for data engineers.

Data EngineeringLambda architecturebatch processing
0 likes · 5 min read
Understanding the Lambda Architecture for Big Data Processing
StarRocks
StarRocks
Aug 14, 2024 · Big Data

Mastering StarRocks & Apache Paimon: A Fast‑Track Lakehouse Guide

This guide provides a comprehensive overview of Apache Paimon’s architecture, key features, and advantages, explains how to integrate it with StarRocks for real‑time lakehouse analytics, and walks through a complete quick‑start setup including component installation, Flink and Kafka deployment, data ingestion, table creation, and query execution with time‑travel support.

Apache PaimonData EngineeringFlink
0 likes · 18 min read
Mastering StarRocks & Apache Paimon: A Fast‑Track Lakehouse Guide
DataFunTalk
DataFunTalk
Aug 6, 2024 · Fundamentals

Solving Massive Data Retrieval Demands: From Problem Causes to OLAP Multidimensional Reporting Solutions

This article analyzes why data engineers face endless data‑extraction requests, identifies common missteps in data‑construction practices, and proposes a comprehensive solution based on dimensional modeling, OLAP multidimensional reporting, self‑service tools, and knowledge empowerment to dramatically improve efficiency and scalability.

Data EngineeringOLAPdimensional modeling
0 likes · 12 min read
Solving Massive Data Retrieval Demands: From Problem Causes to OLAP Multidimensional Reporting Solutions
Alibaba Cloud Observability
Alibaba Cloud Observability
Jul 31, 2024 · Cloud Native

How the New SLS Data Processing Boosts Performance, Cuts Cost, and Simplifies Debugging with SPL

This article explains how Alibaba Cloud's SLS data processing resolves the tension between simple log collection and the need for structured, analyzable data by introducing a unified SPL syntax, delivering over tenfold performance gains, reducing costs to one‑third, and providing powerful debugging tools for cloud‑native log analytics.

Data EngineeringDebuggingSPL
0 likes · 8 min read
How the New SLS Data Processing Boosts Performance, Cuts Cost, and Simplifies Debugging with SPL
DataFunSummit
DataFunSummit
Jul 5, 2024 · Artificial Intelligence

Building and Applying a User Profile Tagging System: Practices and Insights

This article presents a comprehensive overview of constructing and deploying a user and item profiling tag system at Qunar, covering tag taxonomy, integration challenges, technical architectures, algorithmic methods such as classification, recommendation, knowledge‑graph and causal inference, as well as real‑time streaming, ID‑mapping, and practical applications in marketing, attribution and A/B testing.

AB TestingData EngineeringTagging System
0 likes · 21 min read
Building and Applying a User Profile Tagging System: Practices and Insights
DevOps
DevOps
Jun 27, 2024 · Big Data

Agile Data Engineering: Code‑as‑Infrastructure, Reuse Strategies, and ETL‑Level Continuous Integration

This article explores agile data engineering, advocating code‑as‑infrastructure practices such as code‑everything, data and code reuse, and ETL‑level continuous integration, while discussing the trade‑offs between data‑centric and code‑centric reuse, cost considerations, and practical implementation tips for modern data projects.

Code as InfrastructureData EngineeringETL
0 likes · 22 min read
Agile Data Engineering: Code‑as‑Infrastructure, Reuse Strategies, and ETL‑Level Continuous Integration
Baobao Algorithm Notes
Baobao Algorithm Notes
Jun 27, 2024 · Artificial Intelligence

Engineering Data for R&D Large Language Models: From Pre‑training to Prompt Design

This article presents a comprehensive guide to data engineering for research‑focused large language models, covering domain‑adaptive pre‑training, supervised fine‑tuning, retrieval‑augmented generation, dataset construction, data cleaning pipelines, token‑izer adaptation, and prompt engineering best practices to boost model performance in specialized tasks.

Data EngineeringFine‑TuningLLM
0 likes · 20 min read
Engineering Data for R&D Large Language Models: From Pre‑training to Prompt Design
DataFunSummit
DataFunSummit
Jun 15, 2024 · Artificial Intelligence

Large‑Model‑Driven Data Governance: Technical Outlook and Research Highlights

This article reviews the rising importance of data quality for large models, explores data‑centric AI, large‑model pre‑training data engineering, and presents recent Fudan University research on using large models to improve data governance across multiple domains such as attribute normalization, geographic cleaning, compliance checking, and multimodal retrieval.

AIData EngineeringMultimodal Data
0 likes · 19 min read
Large‑Model‑Driven Data Governance: Technical Outlook and Research Highlights
Data Thinking Notes
Data Thinking Notes
May 30, 2024 · Databases

Why Your Data Team Is Drowning in Requests—and How OLAP Can Save You

This article examines why data departments get overwhelmed by massive data‑retrieval requests, identifies root causes such as mindset, requirement handling, and lack of tools, and presents a technical solution centered on dimensional modeling and OLAP multi‑dimensional reporting to streamline data access and empower teams.

Data EngineeringData ModelingOLAP
0 likes · 12 min read
Why Your Data Team Is Drowning in Requests—and How OLAP Can Save You
StarRocks
StarRocks
May 14, 2024 · Artificial Intelligence

How Tencent Games Boosted AI‑Generated SQL Accuracy to 89% with a Lakehouse Architecture

Tencent Games tackled the low accuracy of AI‑generated SQL in production by combining large language models with a StarRocks lake‑warehouse, introducing a semantic layer, async materialized views, and an agent‑based multi‑intelligence framework, ultimately raising one‑shot SQL correctness to 89% and cutting delivery time from 2 hours to 0.33 hours.

AIData EngineeringLLM
0 likes · 13 min read
How Tencent Games Boosted AI‑Generated SQL Accuracy to 89% with a Lakehouse Architecture
DataFunTalk
DataFunTalk
Apr 20, 2024 · Big Data

Tencent Video Metrics Middle Platform and Lakehouse Integration: Architecture, Governance, and Practices

This article details Tencent Video’s data business, describing the design and implementation of its metrics middle platform and lake‑warehouse integration, covering architecture, governance, consistency, timeliness, usability, cost optimization, and future plans, with insights into technology choices such as Iceberg, StarRocks, and MQL.

Data EngineeringMetrics PlatformStarRocks
0 likes · 18 min read
Tencent Video Metrics Middle Platform and Lakehouse Integration: Architecture, Governance, and Practices
DataFunTalk
DataFunTalk
Apr 14, 2024 · Big Data

Third‑Generation Metric Platform: Enabling a Light Data Warehouse with NoETL

This article explains how a third‑generation metric platform replaces traditional ETL‑heavy data‑warehouse pipelines with a semantic‑driven NoETL approach, reducing cost, improving quality and efficiency, and delivering automated, self‑service analytics for both IT and business users.

Data EngineeringNoETLbig data
0 likes · 16 min read
Third‑Generation Metric Platform: Enabling a Light Data Warehouse with NoETL
Data Thinking Notes
Data Thinking Notes
Mar 27, 2024 · Big Data

How to Build and Optimize a Scalable User Profiling Platform from Scratch

This article explains the value of user profiling platforms, outlines their core functions, presents a layered architecture with open‑source options, and details engineering optimizations—from wide‑table design to BitMap caching and task‑mode execution—while also discussing current industry trends.

Data EngineeringPerformance Optimizationbig data
0 likes · 18 min read
How to Build and Optimize a Scalable User Profiling Platform from Scratch
DataFunTalk
DataFunTalk
Mar 26, 2024 · Big Data

Building an Enterprise Real-Time Data Warehouse with Hologres and Flink at Cao Cao Mobility

This article presents a comprehensive case study of Cao Cao Mobility's transition from a traditional Lambda architecture to an enterprise‑grade real‑time data warehouse built on Hologres and Flink, detailing business background, pain points, architectural design, performance optimizations, metadata management, and future development directions.

Data EngineeringFlinkHologres
0 likes · 20 min read
Building an Enterprise Real-Time Data Warehouse with Hologres and Flink at Cao Cao Mobility
DataFunSummit
DataFunSummit
Mar 21, 2024 · Big Data

Kuaishou Analytics Service 3.0: Architecture, Evolution, and Practice

This article presents Kuaishou's end‑to‑end analytics platform, detailing the evolution from the early tool‑based stage through Service 1.0 and 2.0 to the unified Service 3.0 architecture, its unified analysis and query engines, data acceleration techniques, performance gains, and future intelligent analytics roadmap.

Data EngineeringKuaishouUnified Engine
0 likes · 16 min read
Kuaishou Analytics Service 3.0: Architecture, Evolution, and Practice
Alipay Experience Technology
Alipay Experience Technology
Mar 19, 2024 · Big Data

How Alipay Cut Merchant Bill Complexity by 60% Using a Five‑Step Method

This article details how Alipay's data engineering team applied Elon Musk's five‑step work method to completely refactor a decade‑old merchant billing system, reducing overall complexity by over 60%, improving timeliness by an hour, cutting storage and compute costs by a third, and dramatically lowering operational and maintenance burdens.

Data Engineeringautomationbig data
0 likes · 23 min read
How Alipay Cut Merchant Bill Complexity by 60% Using a Five‑Step Method
DataFunSummit
DataFunSummit
Mar 15, 2024 · Product Management

How to Build a Good Data Platform: Insights from Tencent’s Senior Product Manager

This presentation shares the speaker’s experience and practical methods for creating an effective data platform, covering the transition from technical roles to product management, deep understanding of data workers' needs, Tencent Oura asset‑factory practices, a product‑management methodology, and a Q&A session that addresses governance, performance, and engineering challenges.

Data Engineeringdata governanceproduct management
0 likes · 15 min read
How to Build a Good Data Platform: Insights from Tencent’s Senior Product Manager
DataFunSummit
DataFunSummit
Mar 12, 2024 · Big Data

Solving Massive Data Retrieval Demands: From Root Causes to OLAP Multidimensional Reporting Solutions

This article analyzes why data engineers face endless data‑retrieval requests, identifies common missteps in data‑construction such as demand‑driven development, lack of modeling and OLAP concepts, and proposes a dimension‑model‑based data warehouse with OLAP reporting, tooling, and knowledge‑empowerment to break the cycle.

Data EngineeringOLAPdimension modeling
0 likes · 13 min read
Solving Massive Data Retrieval Demands: From Root Causes to OLAP Multidimensional Reporting Solutions
DataFunSummit
DataFunSummit
Mar 11, 2024 · Big Data

Evolution of iQIYI's Event Tracking System and Its Data Processing Pipeline

This article outlines the importance of event tracking for data, describes iQIYI's five‑stage tracking system evolution, analyzes the challenges of the self‑service phase, presents the middle‑platform improvements, explains the migration strategy, and details the downstream data lake, real‑time stream, and data‑warehouse processing workflows.

Data EngineeringData PipelineiQIYI
0 likes · 13 min read
Evolution of iQIYI's Event Tracking System and Its Data Processing Pipeline
Huolala Tech
Huolala Tech
Mar 7, 2024 · Big Data

Integrating Apache Tez with Remote Shuffle Service via Uniffle: HuoLala’s Experience

Facing exploding data volumes and rising cluster costs, HuoLala adopted Apache Tez’s Remote Shuffle Service built on Apache Uniffle, redesigning the Tez client to operate without source modifications, detailing architecture, implementation challenges, testing, stability measures, and future plans to enhance big‑data shuffle performance and cost efficiency.

Apache TezData EngineeringRemote Shuffle Service
0 likes · 14 min read
Integrating Apache Tez with Remote Shuffle Service via Uniffle: HuoLala’s Experience
Airbnb Technology Team
Airbnb Technology Team
Mar 1, 2024 · Big Data

Riverbed: A Scalable Data Framework for Real‑time and Batch Processing at Airbnb

Airbnb’s Riverbed framework unifies streaming CDC events and batch Spark jobs behind a GraphQL‑based declarative API to automatically build and maintain distributed materialized views, using Kafka‑partitioned ordering and version control to deliver billions of daily updates with low‑latency reads for features such as payments and search.

AirbnbApache SparkData Engineering
0 likes · 8 min read
Riverbed: A Scalable Data Framework for Real‑time and Batch Processing at Airbnb
DataFunTalk
DataFunTalk
Feb 25, 2024 · Big Data

Implementation Practice of Bilibili's Tag System: Evolution, Architecture, and Future Plans

This article details Bilibili's tag system from its 2021 inception through successive redesigns, describing the three‑layer architecture, data flow pipelines using Hive, Iceberg, Spark and ClickHouse, crowd selection DSL, online services with Redis, performance optimizations, and upcoming governance and quality initiatives.

ClickHouseData EngineeringSpark
0 likes · 12 min read
Implementation Practice of Bilibili's Tag System: Evolution, Architecture, and Future Plans
Ctrip Technology
Ctrip Technology
Feb 22, 2024 · Backend Development

Design and Implementation of a Serverless Data Filling Engine for UnifiedPB in Ctrip Hotel Recommendation System

This article describes how Ctrip's hotel recommendation team built a serverless, configuration‑driven data‑filling engine based on UnifiedPB protobuf schemas to improve development efficiency, reduce cost, ensure data quality, and achieve unified three‑region data delivery across more than twenty recommendation scenarios.

BackendData EngineeringEfficiency
0 likes · 12 min read
Design and Implementation of a Serverless Data Filling Engine for UnifiedPB in Ctrip Hotel Recommendation System
Amap Tech
Amap Tech
Feb 5, 2024 · Artificial Intelligence

Gaode Tech 2023 Highlights: 15 Popular Articles on AI, Data, Mapping, and Navigation Technologies

Gaode Technology’s 2023 roundup showcases fifteen of its most-read articles, spanning AI infrastructure evolution, cloud‑native data optimization, BEV‑based perception, real‑time crowdsourced mapping, ETA prediction, lane‑level navigation, AR HUD, architecture design, low‑code platforms, and high‑performance Android testing.

AIData EngineeringMapping
0 likes · 9 min read
Gaode Tech 2023 Highlights: 15 Popular Articles on AI, Data, Mapping, and Navigation Technologies
Big Data Technology & Architecture
Big Data Technology & Architecture
Jan 31, 2024 · Big Data

2023 Data Development Trends and Outlook for 2024

The article reviews how data development accelerated in 2023—with mature offline computing, rapid adoption of real‑time and lake‑warehouse solutions, and a clearer technical layering—while offering practical insights and future directions for professionals entering 2024.

Data EngineeringReal-time Computingbig data
0 likes · 8 min read
2023 Data Development Trends and Outlook for 2024
Bilibili Tech
Bilibili Tech
Jan 23, 2024 · Databases

Unique Engine Design and Implementation in ClickHouse for Bilibili Live Guild Data

Bilibili migrated its live‑guild analytics from MySQL to ClickHouse, creating a custom ReplicatedUniqueMergeTree engine that uses delete‑on‑insert, min‑max and hash‑bucketed indexes with delete bitmaps to achieve 10‑20× faster queries and scalable near‑real‑time reporting despite higher write latency.

ClickHouseData EngineeringUnique Engine
0 likes · 18 min read
Unique Engine Design and Implementation in ClickHouse for Bilibili Live Guild Data
DataFunTalk
DataFunTalk
Jan 20, 2024 · Big Data

How ByteDance Leverages the Data Flywheel in Large‑Scale Projects

This article explains how ByteDance (Douyin) transforms its data infrastructure from isolated workshops to a unified middle platform and finally to a data flywheel, detailing the three development stages, the Data BP organizational model, real‑time analytics, A/B testing, and the resulting business benefits for large‑scale event projects.

Data Engineeringbig datadata flywheel
0 likes · 13 min read
How ByteDance Leverages the Data Flywheel in Large‑Scale Projects
DataFunSummit
DataFunSummit
Jan 15, 2024 · Artificial Intelligence

Financial Large Language Model: Characteristics, Construction, Architecture, and Practical Applications

This article presents a comprehensive overview of financial large language models, covering their unique characteristics, construction methods, layered technical architecture, evaluation strategies, and real‑world use cases such as quality inspection, AIGC‑driven material generation, sales‑lead mining, and knowledge‑graph‑enhanced intelligent Q&A.

Data Engineeringfinancial AImodel architecture
0 likes · 14 min read
Financial Large Language Model: Characteristics, Construction, Architecture, and Practical Applications
DataFunTalk
DataFunTalk
Dec 28, 2023 · Product Management

Building an Effective Data Platform: Insights and Practices from Tencent's Senior Product Manager

Senior Tencent product manager He Zhichao shares his experience and methodology for creating a high‑quality data platform, covering the transition from technical roles to product, understanding data users’ needs, the Euler asset‑factory implementation, product‑manager best practices, and solutions to common data‑engineering challenges.

Data EngineeringDataOpsdata governance
0 likes · 16 min read
Building an Effective Data Platform: Insights and Practices from Tencent's Senior Product Manager
DataFunTalk
DataFunTalk
Dec 5, 2023 · Big Data

Design and Practice of Xiaomi’s One‑Stop Data Production Platform

This article presents a comprehensive overview of Xiaomi’s data production platform, detailing the full data lifecycle, the technical‑driven product design methodology, the platform’s architecture and core capabilities, as well as real‑world case studies and a Q&A session that illustrate how the system improves data collection, storage, processing, and usage across the organization.

Data EngineeringETLdata lifecycle
0 likes · 17 min read
Design and Practice of Xiaomi’s One‑Stop Data Production Platform
DataFunSummit
DataFunSummit
Dec 2, 2023 · Artificial Intelligence

OPPO’s Unified Modeling Strategy for App Distribution: Balancing Cost Reduction and User Value

In this interview, OPPO’s senior manager Lai Hongke explains how the company tackles the challenges of sparse, cross‑scenario data in app distribution by deploying a unified modeling framework, MMOE sharing, and the oCPX capability to simultaneously cut costs, improve recommendation performance, and preserve user value across its software store and game center.

AIData EngineeringOPPO
0 likes · 11 min read
OPPO’s Unified Modeling Strategy for App Distribution: Balancing Cost Reduction and User Value
DaTaobao Tech
DaTaobao Tech
Oct 11, 2023 · Big Data

Fundamental Data Skills and Complex Query Techniques in MaxCompute

The article teaches developers essential MaxCompute data‑processing skills—from creating and naming tables, handling strings and dates, and writing basic SELECTs, joins, and aggregations, to employing advanced techniques such as temporary tables, CTEs, partitioning, and map‑join hints for efficient complex queries.

Data EngineeringETLMaxCompute
0 likes · 15 min read
Fundamental Data Skills and Complex Query Techniques in MaxCompute
DataFunSummit
DataFunSummit
Sep 13, 2023 · Artificial Intelligence

Data Engineering, Automated Evaluation, and Knowledge Graph Integration in Large Model Development

This article presents a comprehensive overview of data engineering practices for large model training, reviews current model scales and pre‑training data sources, discusses automated evaluation techniques, and explores how knowledge graphs can be integrated throughout the model lifecycle to improve quality and applicability.

AIData Engineeringautomated evaluation
0 likes · 29 min read
Data Engineering, Automated Evaluation, and Knowledge Graph Integration in Large Model Development
Alibaba Cloud Developer
Alibaba Cloud Developer
Sep 13, 2023 · Big Data

How to Quickly Land as a Data Engineer in a New Company

This guide explains how data engineers can rapidly adapt to a new workplace by mastering business context, data domains, and system architecture, using structured learning, practical case studies, and continuous reflection to earn trust and deliver value efficiently.

Business KnowledgeData EngineeringData Modeling
0 likes · 15 min read
How to Quickly Land as a Data Engineer in a New Company
21CTO
21CTO
Sep 8, 2023 · Big Data

Why Real-Time Data Processing Is the Next Frontier for Data Engineers

Real-time data processing transforms traditional batch pipelines by delivering fresh, low‑latency data to millions of concurrent users, leveraging event‑driven architectures, streaming engines, and real‑time databases, with use cases ranging from fraud detection to personalized e‑commerce and operational dashboards, and includes reference architectures and tool recommendations.

Data Engineeringarchitecturebig data
0 likes · 16 min read
Why Real-Time Data Processing Is the Next Frontier for Data Engineers
DataFunTalk
DataFunTalk
Aug 16, 2023 · Artificial Intelligence

Data Engineering, Automated Evaluation, and Knowledge Graph Integration in Large Model Development

This article presents a comprehensive overview of data engineering practices, pre‑training data composition, automated model evaluation techniques, and the synergistic use of knowledge graphs within large‑scale AI model research, highlighting pipelines, quality criteria, and practical case studies.

Data EngineeringLarge Language Modelsautomation evaluation
0 likes · 29 min read
Data Engineering, Automated Evaluation, and Knowledge Graph Integration in Large Model Development
DataFunTalk
DataFunTalk
Aug 10, 2023 · Big Data

iQIYI Magic Mirror: Evolution of a Big Data Analysis Platform

The article details how iQIYI's Magic Mirror platform evolved from a simple single‑table reporting tool to a multi‑engine, self‑service big data analysis system that improves data access speed, reduces operational costs, and supports comprehensive business analytics across the company.

Data EngineeringMagic Mirrorbig data platform
0 likes · 17 min read
iQIYI Magic Mirror: Evolution of a Big Data Analysis Platform
DataFunSummit
DataFunSummit
Jul 28, 2023 · Big Data

User Path Analysis and SessionAnalytics: Business Practices, Technical Architecture, and Open‑Source Framework

This article introduces user path analysis and the SessionAnalytics open‑source framework, covering business scenarios, data processing techniques, algorithmic mining methods, technical architecture, implementation details, comparisons with event‑based analysis, and a comprehensive Q&A for practical deployment.

Data EngineeringNLPbig data
0 likes · 19 min read
User Path Analysis and SessionAnalytics: Business Practices, Technical Architecture, and Open‑Source Framework
Top Architect
Top Architect
Jul 14, 2023 · Big Data

Lambda Architecture: Real-Time Big Data Processing and Practical Use Cases

This article introduces the Lambda Architecture for billion‑scale real‑time data analysis, explains its three layers—Batch, Speed, and Serving—covers its flexibility, fault tolerance, and scalability, and demonstrates concrete applications such as Twitter hashtag analysis and a smart‑parking recommendation system.

Batch LayerData EngineeringLambda architecture
0 likes · 11 min read
Lambda Architecture: Real-Time Big Data Processing and Practical Use Cases
政采云技术
政采云技术
Jun 15, 2023 · Big Data

Optimizing Data Lineage Extraction Using Spline REST API

This article discusses the practical implementation of extracting table and field lineage information via the Spline REST API, analyzing API call frequency, server load tolerance, and the strategy of re-parsing lineage only when job versions change to optimize performance.

Data EngineeringREST APISpline
0 likes · 5 min read
Optimizing Data Lineage Extraction Using Spline REST API
JD Cloud Developers
JD Cloud Developers
May 30, 2023 · Big Data

ClickHouse & Flink: Choosing Engines, Tuning Queries, and Scaling Concurrency

This article details how JDQ, Flink, and ClickHouse were integrated to replace Elasticsearch for real‑time reporting, covering table‑engine selection, Flink sink implementation, performance bottlenecks, CPU hot‑spots, query optimization techniques, and strategies for handling high concurrency while ensuring data consistency and system stability.

ClickHouseData EngineeringFlink
0 likes · 46 min read
ClickHouse & Flink: Choosing Engines, Tuning Queries, and Scaling Concurrency
Big Data Technology & Architecture
Big Data Technology & Architecture
May 29, 2023 · Big Data

Kuaishou Data Lake Construction with Apache Hudi: Architecture, Challenges, and Solutions

This article explains why Kuaishou built a data lake, describes its lake architecture based on Apache Hudi and Flink, outlines five major production challenges—including ingestion bottlenecks, snapshot queries, update bottlenecks, merge limitations, and operational reliability—and details the practical solutions and future roadmap.

Apache HudiData EngineeringFlink
0 likes · 18 min read
Kuaishou Data Lake Construction with Apache Hudi: Architecture, Challenges, and Solutions
Architects Research Society
Architects Research Society
May 20, 2023 · Cloud Native

Leveraging Software Architecture at Nubank: From Startup to Scale

This article chronicles Nubank’s architectural evolution—detailing how strategic technology choices, cloud‑native platforms, micro‑service design, and data‑engineering practices were leveraged across startup, growth, consolidation, and expansion phases to achieve massive scalability and business agility.

Cloud NativeData EngineeringKubernetes
0 likes · 24 min read
Leveraging Software Architecture at Nubank: From Startup to Scale
DataFunSummit
DataFunSummit
Apr 24, 2023 · Artificial Intelligence

OpenMLDB: A Production‑Grade Feature Platform for Consistent Online and Offline Machine Learning

OpenMLDB is an open‑source machine‑learning database that delivers a production‑grade, consistent online‑offline feature platform for real‑time AI applications such as recommendation, risk control and fraud detection, offering millisecond‑level feature computation, dual SQL engines, extensive ecosystem integration, and a roadmap of new capabilities.

AIData EngineeringFeature Store
0 likes · 13 min read
OpenMLDB: A Production‑Grade Feature Platform for Consistent Online and Offline Machine Learning
Python Programming Learning Circle
Python Programming Learning Circle
Apr 23, 2023 · Big Data

Parallel Processing of Large CSV Files in Python with multiprocessing, joblib, and tqdm

This tutorial demonstrates how to accelerate processing of a 2.8‑million‑row CSV dataset by using Python's multiprocessing, joblib, and tqdm libraries, covering serial, parallel, and batch processing techniques, performance measurements, and best‑practice code examples for efficient large‑scale data handling.

Data EngineeringParallel ProcessingPython
0 likes · 9 min read
Parallel Processing of Large CSV Files in Python with multiprocessing, joblib, and tqdm
DataFunTalk
DataFunTalk
Mar 25, 2023 · Artificial Intelligence

ZhongAn Financial Real‑Time Feature Platform: MLOps Practices, Architecture and Anti‑Fraud Applications

This article presents ZhongAn Financial’s end‑to‑end MLOps workflow and real‑time feature platform architecture, detailing team roles, data pipelines, Flink‑based processing, TableStore storage, anti‑fraud feature design, and answers to common implementation questions, offering a comprehensive guide for building scalable, low‑latency ML services in finance.

Anti-FraudData EngineeringFlink
0 likes · 25 min read
ZhongAn Financial Real‑Time Feature Platform: MLOps Practices, Architecture and Anti‑Fraud Applications
Architecture Digest
Architecture Digest
Mar 22, 2023 · Big Data

Performance Platform: Accelerating Data Production and Consumption

This article details how the Performance Platform at Baidu speeds up data production and consumption across the company's R&D pipelines by introducing five optimization paths, 18 concrete methods, service tiering, compliance measures, and self‑service analytics for both real‑time memory tables and offline disk tables.

Data EngineeringETLSelf-Service Analytics
0 likes · 13 min read
Performance Platform: Accelerating Data Production and Consumption
Huolala Tech
Huolala Tech
Mar 16, 2023 · Big Data

How HuoLala’s YunTai BI Platform Transforms Data Visualization at Scale

The article details HuoLala’s internally built YunTai BI platform, covering its motivation, system architecture, data source integration, zero‑code modeling, visual report and dashboard creation, performance optimizations, and future plans for stability and code design, illustrating a comprehensive big‑data visualization solution.

BIData Engineeringdata visualization
0 likes · 13 min read
How HuoLala’s YunTai BI Platform Transforms Data Visualization at Scale
Baidu Geek Talk
Baidu Geek Talk
Mar 6, 2023 · Big Data

Accelerating Data Production and Consumption in Baidu's Performance Platform

Baidu's Performance Platform speeds data production and consumption by adopting a unified stream‑batch architecture with TM and Spark, leveraging the Turing warehouse, introducing tiered service grading, robust governance and compliance measures, and offering self‑service analytics, cutting latency from minutes or days to milliseconds while handling billions of daily records and boosting SLA adherence, data accuracy, and user satisfaction.

Data Engineeringbig datadata governance
0 likes · 12 min read
Accelerating Data Production and Consumption in Baidu's Performance Platform
DataFunSummit
DataFunSummit
Mar 2, 2023 · Big Data

Huya's Data Self‑Service Product: Challenges, Design, and Practice

The article presents Huya's data‑self‑service product, describing the problems of traditional data services, the principles of a good data service, the MVP implementation, architectural components, project outcomes, and future evolution, while also addressing common Q&A scenarios.

Data Engineeringbig datadata product
0 likes · 12 min read
Huya's Data Self‑Service Product: Challenges, Design, and Practice
Past Memory Big Data
Past Memory Big Data
Feb 23, 2023 · Databases

Optimizing ClickHouse: ByteHouse’s Real‑Time Data Warehouse Breakthrough

ByteHouse, a cloud‑native data warehouse on Volcano Engine, leverages ClickHouse to deliver ultra‑fast real‑time analytics, detailing its business motivations, ROI‑driven evaluation, performance trade‑offs, architectural evolution, and concrete financial‑industry deployments such as real‑time monitoring and risk‑control scenarios.

ByteHouseClickHouseCloud Native
0 likes · 17 min read
Optimizing ClickHouse: ByteHouse’s Real‑Time Data Warehouse Breakthrough
DataFunTalk
DataFunTalk
Feb 18, 2023 · Artificial Intelligence

Building the ATLAS Automated Machine Learning Platform at Du Xiaoman: Architecture, Optimization, and Practical Insights

This article details Du Xiaoman's development of the ATLAS automated machine learning platform, covering business scenarios, AI algorithm deployment challenges, the end‑to‑end production workflow, platform components such as annotation, data, training and deployment, as well as optimization techniques like AutoML, meta‑learning, NAS, and large‑scale parallelism, concluding with lessons learned and future directions.

AI DeploymentAutoMLData Engineering
0 likes · 20 min read
Building the ATLAS Automated Machine Learning Platform at Du Xiaoman: Architecture, Optimization, and Practical Insights
dbaplus Community
dbaplus Community
Feb 15, 2023 · Big Data

How Bilibili Scaled User Behavior Analytics with ClickHouse, Flink, and Iceberg

This article details Bilibili's 北极星 user behavior analysis platform, tracing its evolution from early Spark‑Jar models to Flink‑ClickHouse pipelines and Iceberg‑based full aggregation, and explains the technical solutions for event, retention, funnel, path analysis, data ingestion, cluster rebalancing, and performance optimizations that enable massive real‑time analytics on billions of daily events.

ClickHouseData EngineeringData Modeling
0 likes · 32 min read
How Bilibili Scaled User Behavior Analytics with ClickHouse, Flink, and Iceberg
Kuaishou Big Data
Kuaishou Big Data
Feb 3, 2023 · Big Data

Inside Kuaishou’s Company‑Wide Metric Platform: Architecture, Lessons & Best Practices

This article details Kuaishou’s three‑year evolution of its metric middle platform, covering the data infrastructure, key challenges of data inconsistency and low analysis efficiency, the enterprise‑level OneMetric solution, architectural design, development phases, practical lessons, system implementation, and real‑world applications.

Data EngineeringKuaishoubig data
0 likes · 23 min read
Inside Kuaishou’s Company‑Wide Metric Platform: Architecture, Lessons & Best Practices
DataFunSummit
DataFunSummit
Dec 29, 2022 · Big Data

Understanding Lakehouse Systems: Architecture, Practices, and Innovations by Databricks

This article explains the Lakehouse concept, why it is needed, the limitations of traditional data warehouses and data lakes, and how Databricks’ unified architecture—through open storage formats, fine‑grained governance, and optimized query engines—delivers high‑quality, low‑latency data for BI, analytics, and machine learning workloads.

Data EngineeringDatabricksDelta Lake
0 likes · 21 min read
Understanding Lakehouse Systems: Architecture, Practices, and Innovations by Databricks
Alibaba Cloud Developer
Alibaba Cloud Developer
Dec 26, 2022 · Backend Development

How to Build a Scalable Tag/Profile System for Marketing Automation

This article shares engineering practices for constructing a tag‑profile system, covering core concepts, minimal architecture, technology selection, key modules such as estimation, selection, deployment, and validation, and offers design details and implementation tips for large‑scale marketing scenarios.

Alibaba CloudBackend ArchitectureData Engineering
0 likes · 11 min read
How to Build a Scalable Tag/Profile System for Marketing Automation
Architects Research Society
Architects Research Society
Nov 27, 2022 · Big Data

Building a Data‑Driven Organization: Culture, Structure, and Roles

This article explains the practical steps to transform a company into a data‑driven organization by establishing a self‑service culture, aligning organizational structures, defining key roles such as analysts, engineers, scientists, and CDOs, and addressing common obstacles and best‑practice tips.

CultureData EngineeringSelf-Service
0 likes · 23 min read
Building a Data‑Driven Organization: Culture, Structure, and Roles
Alibaba Cloud Big Data AI Platform
Alibaba Cloud Big Data AI Platform
Nov 25, 2022 · Big Data

What Drives the Next Wave of Open‑Source Big Data? Insights from the 2022 Heat Report

The 2022 Open Source Big Data Heat Report analyzes 102 active projects since 2015, revealing that heat values double every 40 months, highlighting diversification, integration, and cloud‑native trends, and offering guidance for developers, contributors, and project maintainers navigating the evolving big‑data landscape.

Data Engineeringtechnology trends
0 likes · 15 min read
What Drives the Next Wave of Open‑Source Big Data? Insights from the 2022 Heat Report
Tencent Cloud Developer
Tencent Cloud Developer
Nov 7, 2022 · Big Data

Data Engineering and Data Warehouse Design: Principles, Practices, and Governance

The article outlines comprehensive data‑engineering and warehouse‑design principles—covering collection (four Ws and methods like SDK, point‑code, binlog), reporting strategies, source selection, modeling with fact, aggregation, dimension and model tables, quality checks, and governance practices such as standardized SDKs, metric libraries, automated lineage, and cost optimization—to share actionable experience for any organization.

Data EngineeringETLbig data
0 likes · 32 min read
Data Engineering and Data Warehouse Design: Principles, Practices, and Governance
DevOps Cloud Academy
DevOps Cloud Academy
Oct 22, 2022 · Fundamentals

How to Write Your First Apache Airflow DAG (Hello World)

This tutorial walks through creating a simple “Hello World” Apache Airflow DAG by setting up the Python file, importing modules, defining the DAG object, adding a PythonOperator task, writing the callable function, and running the DAG with Airflow’s webserver and scheduler.

Apache AirflowDAGData Engineering
0 likes · 9 min read
How to Write Your First Apache Airflow DAG (Hello World)
Hulu Beijing
Hulu Beijing
Oct 21, 2022 · Big Data

How Hulu Scales Spark on Kubernetes: Cloud‑Native Big Data at Disney‑Scale

Hulu’s data platform team describes how they migrated large‑scale Spark workloads from Yarn to native Spark on Kubernetes, leveraging AWS services such as EKS, S3, and custom operators to achieve dynamic scaling, unified monitoring, cost‑effective resource management, and improved stability for search, recommendation, and advertising pipelines.

AWSCloud NativeData Engineering
0 likes · 18 min read
How Hulu Scales Spark on Kubernetes: Cloud‑Native Big Data at Disney‑Scale
DataFunTalk
DataFunTalk
Aug 31, 2022 · Big Data

Solving Data‑Driven Full‑Link Technical Challenges: A Case Study of the Kai Shu Storytelling App

This article analyzes the technical difficulties of building a data‑driven full‑link system and describes how the Kai Shu Storytelling app overcame them by adopting DataFinder for automated event tracking, metric management, and growth analysis, offering practical guidance for enterprises and developers.

AnalyticsApp DevelopmentData Engineering
0 likes · 7 min read
Solving Data‑Driven Full‑Link Technical Challenges: A Case Study of the Kai Shu Storytelling App
Selected Java Interview Questions
Selected Java Interview Questions
Aug 27, 2022 · Backend Development

Deploying a Cost‑Effective ClickHouse‑Based Backend Data Platform: Comparison with Elasticsearch and Step‑by‑Step Setup Guide

This article compares Elasticsearch and ClickHouse for log analytics, presents cost analysis, and provides detailed deployment instructions for Zookeeper, Kafka, Filebeat, and ClickHouse to build a private, high‑performance backend data platform for SaaS services.

ClickHouseData EngineeringElasticsearch
0 likes · 12 min read
Deploying a Cost‑Effective ClickHouse‑Based Backend Data Platform: Comparison with Elasticsearch and Step‑by‑Step Setup Guide
DataFunSummit
DataFunSummit
Aug 26, 2022 · Big Data

Data Governance Practice and Logical Closed‑Loop at KuaiKan: A Case Study

This article presents KuaiKan's data governance journey, detailing the rapid business expansion challenges, the three‑step planning framework, the logical closed‑loop architecture, practical implementation experiences, cross‑team collaboration techniques, and the evaluation of governance outcomes and future plans.

Data Engineeringdata quality
0 likes · 16 min read
Data Governance Practice and Logical Closed‑Loop at KuaiKan: A Case Study
DataFunSummit
DataFunSummit
Jul 9, 2022 · Big Data

Alibaba's One‑Stop Real‑Time Data Warehouse: Hologres Architecture and CCO Implementation Experience

The article reviews the shift of big‑data computing from batch to real‑time, outlines the evolution of one‑stop real‑time data warehouses, introduces Alibaba's Hologres solution and its technical advantages, and shares the CCO department’s three‑generation architecture upgrades and practical use cases.

AlibabaData EngineeringHologres
0 likes · 16 min read
Alibaba's One‑Stop Real‑Time Data Warehouse: Hologres Architecture and CCO Implementation Experience
DaTaobao Tech
DaTaobao Tech
Jul 8, 2022 · Frontend Development

Alibaba Front‑End Intelligent Technology: PipCook, DataCook, imgcook and Future Directions

Alibaba Front‑End Intelligent Technology combines PipCook, DataCook, and imgcook to enable data‑driven UI generation, on‑device AI inference via WASM‑Rust‑SIMD and WebGPU, and applications such as code IntelliSense and design‑to‑code, while outlining a roadmap toward unified AI‑powered interfaces for commerce.

AIData EngineeringTensorFlow.js
0 likes · 33 min read
Alibaba Front‑End Intelligent Technology: PipCook, DataCook, imgcook and Future Directions
AntTech
AntTech
Jun 29, 2022 · Big Data

YoDA: Reducing Entropy in Ant Financial Risk Data Systems through White‑Box, Logical, and Integrated Approaches

The YoDA project tackles the growing entropy of Ant Financial's risk data platform by introducing white‑box visibility, logical abstraction, and integrated heterogeneous fusion, enabling systematic governance, cost reduction, and consistent decision‑making across online, offline, and near‑line environments.

AIData EngineeringEntropy Reduction
0 likes · 21 min read
YoDA: Reducing Entropy in Ant Financial Risk Data Systems through White‑Box, Logical, and Integrated Approaches
Bilibili Tech
Bilibili Tech
May 31, 2022 · Big Data

Bilibili Offline Computing Platform: Migration from Hive to Spark and Operational Practices

Bilibili migrated its massive offline platform from Hive to Spark using an automated SQL rewrite and dual‑run verification, cutting execution time over 40% and resource use 30%, while introducing small‑file merging, shuffle stability, runtime filters, data‑skipping, lineage tracking, auto‑parameter tuning, and metastore federation for robust large‑scale processing.

Data EngineeringHiveSpark
0 likes · 30 min read
Bilibili Offline Computing Platform: Migration from Hive to Spark and Operational Practices
DataFunSummit
DataFunSummit
May 29, 2022 · Big Data

OPPO Commercial Data System Construction Practice: Platform, Ingestion, Development, Governance, and Analytics

This article presents OPPO's commercial data system construction practice, covering the data platform strategy, ingestion pipelines, development efficiency toolkits, data validation, visualization aids, UDF principles, warehouse architecture, metric systems, dimensional modeling, ETL optimization, governance metadata, quality management, monitoring, attribution services, analytics reporting, and a Q&A session.

AnalyticsData Engineeringdata platform
0 likes · 17 min read
OPPO Commercial Data System Construction Practice: Platform, Ingestion, Development, Governance, and Analytics
dbaplus Community
dbaplus Community
May 21, 2022 · Big Data

5 Trends for 2022: Analytics Engineers, Lakehouse Wars, Real‑Time Pipelines, Cloud Market

The article outlines five major 2022 data trends— the rise of analytics engineers, the intensifying lake‑house competition, the growth of real‑time streaming pipelines and operational analytics, the expanding cloud marketplaces for data tools, and the push toward unified data‑quality terminology—explaining their origins, market impact, and future outlook.

Data Engineeringanalytics engineeringcloud marketplace
0 likes · 21 min read
5 Trends for 2022: Analytics Engineers, Lakehouse Wars, Real‑Time Pipelines, Cloud Market
Alibaba Cloud Developer
Alibaba Cloud Developer
May 18, 2022 · Big Data

Why Delta Lake Is Revolutionizing Data Lakes with ACID Guarantees

This article explains how Delta Lake adds reliability to data lakes by offering ACID transactions, scalable metadata, and unified batch‑and‑stream processing, outlines the challenges it solves, details its implementation principles, and demonstrates a practical demo for building an integrated data warehouse.

ACIDData EngineeringDelta Lake
0 likes · 9 min read
Why Delta Lake Is Revolutionizing Data Lakes with ACID Guarantees
Big Data Technology Architecture
Big Data Technology Architecture
Apr 29, 2022 · Big Data

Halodoc’s Data Platform Evolution: From Redshift to a LakeHouse Architecture with Apache Hudi

This article describes how Halodoc’s data engineering team identified limitations of their Redshift‑based platform, evaluated a LakeHouse design, selected Apache Hudi for mutable data handling, and outlined the challenges and benefits of building a scalable, decoupled storage‑compute architecture for their growing healthcare services.

Apache HudiData Engineeringdata platform
0 likes · 9 min read
Halodoc’s Data Platform Evolution: From Redshift to a LakeHouse Architecture with Apache Hudi
DataFunTalk
DataFunTalk
Apr 19, 2022 · Artificial Intelligence

Intelligent Risk Control Platform: Design Principles, Strategy and Model Lifecycle Management, and Architecture

This article presents a comprehensive overview of an intelligent risk control platform, covering its design background, six core characteristics, the "five‑full double‑core" concept, end‑to‑end strategy and model lifecycle management, business architecture atomization, and real‑world anti‑fraud case studies.

AIData EngineeringRisk Control
0 likes · 13 min read
Intelligent Risk Control Platform: Design Principles, Strategy and Model Lifecycle Management, and Architecture
Big Data Technology & Architecture
Big Data Technology & Architecture
Apr 6, 2022 · Big Data

Data Quality Issues, Causes, and Practices in Big Data Platforms

This article explains the harms and root causes of data quality problems—such as integrity, latency, accuracy, and consistency issues—then outlines systematic prevention methods, baseline monitoring, and concrete NetEase YouShu platform practices, illustrated with real incidents, code snippets, and tag‑monitoring strategies.

Data EngineeringIncident Management
0 likes · 10 min read
Data Quality Issues, Causes, and Practices in Big Data Platforms
58 Tech
58 Tech
Mar 29, 2022 · Big Data

Design and Implementation of the 58 Group Penalty Data Center

This article presents the design, architecture, and implementation of a unified penalty data center for 58 Group, detailing the challenges of heterogeneous data sources, the selection of Flink for real‑time ETL, the use of a DSL and LRU aggregation, and the adoption of MVEL for feature recognition to achieve standardized, high‑performance penalty data processing.

Data EngineeringETLFlink
0 likes · 13 min read
Design and Implementation of the 58 Group Penalty Data Center
Architects Research Society
Architects Research Society
Mar 11, 2022 · Artificial Intelligence

Key Software Industry Trends in 2021 and What to Watch in 2022

The 2021 software industry review highlights the rise of hybrid work, the continued dominance of microservices, emerging data engineering and AI/ML practices, ethical and sustainability concerns, multi‑cloud and cloud‑native adoption, and anticipates further developments in these areas throughout 2022.

AIData EngineeringEthics
0 likes · 14 min read
Key Software Industry Trends in 2021 and What to Watch in 2022
21CTO
21CTO
Feb 24, 2022 · Big Data

5 Data Trends for 2022: Analytics Engineers, Lakehouse Wars, Real‑Time

In 2022 the modern data stack will be driven by the rise of analytics engineers, intensified competition between lakehouse and warehouse solutions, growing demand for real‑time analytics, the explosive growth of cloud marketplaces, and the emergence of unified data‑quality terminology, all reshaping data infrastructure and operational practices.

Data Engineeringanalytics engineeringcloud marketplaces
0 likes · 17 min read
5 Data Trends for 2022: Analytics Engineers, Lakehouse Wars, Real‑Time