BestHub
Sign in
Tagged articles

Data Integration

293 articles · Page 2 of 3
AsiaInfo Technology: New Tech Exploration
AsiaInfo Technology: New Tech Exploration
Nov 4, 2024 · Big Data

How Apache SeaTunnel Redefines Data Integration for Modern Data Platforms

This article reviews the evolution of data‑integration architectures toward EtLT, explains the core capabilities of Apache SeaTunnel, and details how a Chinese data‑platform vendor applied and extended SeaTunnel to simplify batch and streaming ingestion, unify multi‑engine processing, and reduce development and operational costs.

Apache SeaTunnelBig DataConnector Development
0 likes · 17 min read
How Apache SeaTunnel Redefines Data Integration for Modern Data Platforms
DataFunSummit
DataFunSummit
Nov 1, 2024 · Big Data

DataFun Summit Session Overview and E‑book Access Instructions

The article outlines how to obtain the DataFun Summit e‑book by following the public account instructions and provides concise English summaries of twelve technical sessions covering data lineage, integration, AI language models, multimodal content, game AI agents, lake‑warehouse governance, big‑data architecture, and cluster management.

AIBig DataData Integration
0 likes · 5 min read
DataFun Summit Session Overview and E‑book Access Instructions
DataFunSummit
DataFunSummit
Oct 27, 2024 · Artificial Intelligence

How Siemens Harnesses Generative AI to Build the Enterprise Knowledge Chatbot “XiaoYu”

This article describes Siemens' journey in applying generative AI and Retrieval‑Augmented Generation to create an internal knowledge chatbot, detailing the business challenges, technical architecture, data integration, multi‑modal capabilities, deployment outcomes, and strategic lessons for enterprise AI adoption.

AI chatbotData IntegrationEnterprise Knowledge Management
0 likes · 21 min read
How Siemens Harnesses Generative AI to Build the Enterprise Knowledge Chatbot “XiaoYu”
macrozheng
macrozheng
Sep 27, 2024 · Big Data

Master DataX: Efficient Offline Data Sync for Heterogeneous Sources

This guide walks through the challenges of synchronizing massive datasets across heterogeneous databases, introduces Alibaba's open‑source DataX tool, explains its framework‑plugin architecture, and provides step‑by‑step instructions—including environment setup, installation, job configuration, and both full and incremental MySQL synchronization—complete with code examples and performance metrics.

Big DataData IntegrationDataX
0 likes · 15 min read
Master DataX: Efficient Offline Data Sync for Heterogeneous Sources
Data Thinking Notes
Data Thinking Notes
Sep 9, 2024 · Fundamentals

Master the 6‑Step Blueprint for Building an Enterprise Data Middle Platform

This guide outlines a practical six‑step methodology—covering overall planning, data integration, model construction, data development, asset management, and data services—to help enterprises build a robust data middle platform that unlocks business value and supports agile digital transformation.

Data GovernanceData IntegrationData Modeling
0 likes · 10 min read
Master the 6‑Step Blueprint for Building an Enterprise Data Middle Platform
Ops Development & AI Practice
Ops Development & AI Practice
Aug 7, 2024 · Artificial Intelligence

How ChatGPT’s New JSON Output Transforms AI Integration

This article examines OpenAI's recent ChatGPT API update that adds JSON‑formatted responses, detailing the technical background, implementation steps, example requests and responses, and the broader impact on developers, enterprises, and future AI applications.

APIArtificial IntelligenceChatGPT
0 likes · 10 min read
How ChatGPT’s New JSON Output Transforms AI Integration
Data Thinking Notes
Data Thinking Notes
Jul 29, 2024 · Big Data

What Is a Data Middle Platform and How Does It Transform Enterprise Data Management?

This article explains the concept, design principles, and core components of a data middle platform, detailing its overall, functional, layered, logical, and data architectures, as well as the specific platforms for data collection, processing, organization, governance, quality, sharing, and visualization, illustrated with diagrams.

Big DataData GovernanceData Integration
0 likes · 27 min read
What Is a Data Middle Platform and How Does It Transform Enterprise Data Management?
DaTaobao Tech
DaTaobao Tech
Jul 8, 2024 · Big Data

ODPS (MaxCompute) SQL Basics, Data Integration and Hologres Import Guide

This guide provides a comprehensive, beginner‑to‑advanced reference for ODPS (MaxCompute) SQL, covering table creation, DDL/DML commands, query syntax, join hints, MySQL‑to‑ODPS synchronization, one‑click and custom imports into Hologres, and scheduling variables for automated data pipelines.

Data IntegrationETLHologres
0 likes · 37 min read
ODPS (MaxCompute) SQL Basics, Data Integration and Hologres Import Guide
DataFunSummit
DataFunSummit
Jun 14, 2024 · Big Data

JD Logistics One‑Stop Agile BI Solution: Architecture, Challenges, and Product Evolution

This article presents JD Logistics' one‑stop agile BI platform, detailing the complex data sources, rapid business demands, the UData solution architecture, performance and usability improvements, and future upgrade plans that together enable faster data integration, self‑service reporting, and enhanced decision‑making across the organization.

Agile AnalyticsBIBig Data
0 likes · 25 min read
JD Logistics One‑Stop Agile BI Solution: Architecture, Challenges, and Product Evolution
DataFunTalk
DataFunTalk
May 13, 2024 · Big Data

Data Integration Maturity Model: From ETL to EtLT

The article examines the evolution of data integration architectures—from traditional ETL through ELT to the emerging EtLT model—highlighting their advantages, disadvantages, industry trends, maturity stages, and practical guidance for enterprises and professionals navigating modern big‑data pipelines.

Big DataData IntegrationDataOps
0 likes · 31 min read
Data Integration Maturity Model: From ETL to EtLT
DataFunTalk
DataFunTalk
May 8, 2024 · Big Data

Risk Control and Data Application in the Bulk Commodity Industry: Challenges, Solutions, and Core Capabilities

The article presents Ant Group's exploration of applying its data‑driven risk control and credit assessment capabilities to the traditional bulk commodity sector, detailing industry background, data pain points, core technical solutions, and the construction of a secure, explainable data‑model platform for digital transformation.

AIBig DataBulk Industry
0 likes · 13 min read
Risk Control and Data Application in the Bulk Commodity Industry: Challenges, Solutions, and Core Capabilities
21CTO
21CTO
Apr 28, 2024 · Artificial Intelligence

5 Transformative Business Use Cases for Conversational AI

This article explores how conversational AI, powered by large language models, is reshaping enterprise operations across five key scenarios—from customer support assistants and AI‑driven data interfaces to HR bots, unstructured data processing, and multi‑agent digital assistants—highlighting benefits, implementation considerations, and privacy challenges.

Conversational AICustomer SupportData Integration
0 likes · 13 min read
5 Transformative Business Use Cases for Conversational AI
Data Thinking Notes
Data Thinking Notes
Apr 9, 2024 · Big Data

What Is a Data Middle Platform and Why It’s Essential for Modern Enterprises

Data middle platforms transform raw enterprise data into reusable assets by integrating collection, storage, processing, governance, and service layers, enabling faster deployment, consistent metrics, improved data quality, and business value across digital transformation, while addressing challenges like siloed data, low efficiency, and inconsistent standards.

Big DataData GovernanceData Integration
0 likes · 23 min read
What Is a Data Middle Platform and Why It’s Essential for Modern Enterprises
DataFunSummit
DataFunSummit
Apr 7, 2024 · Big Data

Li Auto’s Flink on Kubernetes Data Integration Practice

This article presents Li Auto’s end‑to‑end data integration journey, detailing the evolution of its data platform, the challenges of heterogeneous sources, and how a unified Flink‑on‑K8s solution with cloud‑native architecture, operator management, monitoring, and checkpointing addresses batch‑stream convergence and future scalability.

Batch ProcessingBig DataData Integration
0 likes · 12 min read
Li Auto’s Flink on Kubernetes Data Integration Practice
DataFunTalk
DataFunTalk
Mar 1, 2024 · Big Data

Understanding Data Fabric and Data Virtualization: Concepts, Practices, and Real‑World Case Study

This article explains the fundamentals of Data Fabric and data virtualization, highlights the limitations of traditional centralized data warehouses, describes the three‑layer virtualization architecture, and presents a detailed securities‑industry case study that demonstrates cost, efficiency, and compliance benefits.

Big DataData FabricData Integration
0 likes · 17 min read
Understanding Data Fabric and Data Virtualization: Concepts, Practices, and Real‑World Case Study
DataFunTalk
DataFunTalk
Feb 23, 2024 · Artificial Intelligence

Challenges and Opportunities in Applying Large‑Model AI to Healthcare

The article analyzes how large‑model medical AI is rapidly adopted yet struggles with implementation due to doctor shortages, behavioral resistance, data silos, safety regulations, and the need for strategic alignment, while contrasting the more supportive innovation ecosystem in the United States.

AI adoptionData IntegrationHealthcare Innovation
0 likes · 6 min read
Challenges and Opportunities in Applying Large‑Model AI to Healthcare
DataFunSummit
DataFunSummit
Feb 20, 2024 · Big Data

BitSail Open‑Source Data Integration Engine: Architecture, New Features, CDC Solutions and Future Outlook

This article introduces ByteDance's open‑source data integration engine BitSail, covering its background, layered architecture, recent feature enhancements, automated testing framework, CDC‑based full‑library synchronization solutions, and future development plans for connectors and real‑time data consistency.

Big DataCDCData Integration
0 likes · 12 min read
BitSail Open‑Source Data Integration Engine: Architecture, New Features, CDC Solutions and Future Outlook
DataFunTalk
DataFunTalk
Feb 17, 2024 · Big Data

JD Logistics One‑Stop Agile BI Solution: Architecture, Challenges, and Optimization

This article presents JD Logistics' one‑stop agile BI platform, detailing the complex data sources, rapid requirement changes, and Chinese‑style reporting challenges it addresses, while outlining the UData solution, product methodology, performance enhancements, and real‑world case studies that demonstrate significant efficiency gains.

Agile AnalyticsBIBig Data
0 likes · 26 min read
JD Logistics One‑Stop Agile BI Solution: Architecture, Challenges, and Optimization
DataFunSummit
DataFunSummit
Feb 5, 2024 · Artificial Intelligence

Ant Group's Knowledge Graph: Overview, Construction, Applications, and Integration with Large Models

Ant Group shares its comprehensive knowledge graph initiatives, detailing the fundamentals, construction pipeline, fusion techniques, cognitive representations, diverse business applications, and the emerging synergy between knowledge graphs and large language models, illustrating how graph-based AI enhances accuracy, interpretability, and downstream services.

Artificial IntelligenceData IntegrationGraph Fusion
0 likes · 14 min read
Ant Group's Knowledge Graph: Overview, Construction, Applications, and Integration with Large Models
DataFunTalk
DataFunTalk
Jan 29, 2024 · Big Data

Case Study: Deploying RisingWave for Real-Time Stream Processing in a Large-Scale Quantitative Hedge Fund

An ultra‑large hedge fund with over $10 billion AUM replaced ksqlDB and Flink with RisingWave, leveraging its PostgreSQL‑compatible streaming SQL to achieve sub‑10 ms latency, lower learning and operational costs, rich connectors, advanced operators, and comprehensive observability for real‑time trade data processing.

Data IntegrationRisingWaveSQL
0 likes · 10 min read
Case Study: Deploying RisingWave for Real-Time Stream Processing in a Large-Scale Quantitative Hedge Fund
NetEase LeiHuo UX Big Data Technology
NetEase LeiHuo UX Big Data Technology
Jan 9, 2024 · Artificial Intelligence

Accelerating Recommendation System Development with MindsDB

The article explains how the data team adopted the open‑source machine‑learning platform MindsDB to simplify data integration, enable SQL‑based model training and inference, manage model versions, and dramatically shorten recommendation system development cycles, achieving up to 30% efficiency gains.

Data IntegrationMindsDBModel Management
0 likes · 5 min read
Accelerating Recommendation System Development with MindsDB
Alibaba Cloud Native
Alibaba Cloud Native
Dec 28, 2023 · Cloud Computing

How to Set Up No‑Code Data Dump from Alibaba Cloud Kafka to OSS

This guide explains how to use Alibaba Cloud Message Queue Kafka's no‑code, fully managed, serverless dump feature to transfer data to OSS, covering its benefits, typical scenarios, required prerequisites, step‑by‑step configuration, testing, and verification of the resulting objects.

Alibaba CloudData IntegrationKafka
0 likes · 9 min read
How to Set Up No‑Code Data Dump from Alibaba Cloud Kafka to OSS
Sohu Tech Products
Sohu Tech Products
Dec 27, 2023 · Big Data

Practical Implementation of Data Integration with Flink on Kubernetes at Li Auto

Li Auto built a cloud‑native data‑integration platform by deploying Flink on Kubernetes, unifying batch and streaming workloads with a storage layer (JuiceFS + BOS) and Flink Operator, enabling simple source‑sink pipelines, elastic scaling, automated checkpointing, and centralized monitoring while addressing earlier fragmentation and resource inefficiencies.

Big DataCloud NativeData Integration
0 likes · 11 min read
Practical Implementation of Data Integration with Flink on Kubernetes at Li Auto
DataFunTalk
DataFunTalk
Dec 22, 2023 · Big Data

Practical Implementation of Flink on Kubernetes for Data Integration at Li Auto

This article details Li Auto's end‑to‑end data integration practice using Flink on Kubernetes, covering the evolution of their integration platform, architectural design, cloud‑native deployment, operational challenges, and future roadmap, while highlighting unified batch‑stream processing and resource elasticity.

Batch ProcessingBig DataCloud Native
0 likes · 12 min read
Practical Implementation of Flink on Kubernetes for Data Integration at Li Auto
Baidu Intelligent Cloud Tech Hub
Baidu Intelligent Cloud Tech Hub
Dec 12, 2023 · Databases

Master Database Migration to Cloud: Challenges & Solutions with Baidu DTS

This article examines the rapid growth of China's database market, the technical hurdles of moving databases to public cloud—including engine selection, lengthy migration processes, efficiency, disaster recovery, and data consistency—and explains how Baidu Intelligent Cloud's DTS platform offers a smooth, reliable, high‑availability, and high‑performance one‑stop solution with real‑world use cases.

Baidu CloudCloud databasesDTS
0 likes · 25 min read
Master Database Migration to Cloud: Challenges & Solutions with Baidu DTS
Data Thinking Notes
Data Thinking Notes
Dec 5, 2023 · Big Data

How to Overcome Data Governance Challenges and Unlock Business Value

Enterprises face significant hurdles in data governance and integration, from siloed systems and unclear responsibilities to poor data quality, but by establishing clear rules, fostering user department engagement, and aligning governance with business-driven data applications, they can create a cohesive data asset management framework that drives value.

Big DataData GovernanceData Integration
0 likes · 10 min read
How to Overcome Data Governance Challenges and Unlock Business Value
Alibaba Cloud Native
Alibaba Cloud Native
Nov 23, 2023 · Cloud Native

How CDC + Serverless Functions Enable Real‑Time ETL in Cloud Native Architectures

This article explains how Alibaba Cloud's Serverless Function Compute combined with Database Change Data Capture (CDC) creates a complete, real‑time ETL pipeline, detailing the ETL model, DTS integration, architecture components, event‑driven processing, and practical use cases such as OLTP‑to‑OLAP data flow.

Alibaba CloudCDCData Integration
0 likes · 10 min read
How CDC + Serverless Functions Enable Real‑Time ETL in Cloud Native Architectures
DataFunSummit
DataFunSummit
Oct 24, 2023 · Big Data

Practices of Data Fabric in Data Integration Scenarios

The presentation by Aloudata Vice President Yu Jun introduces his extensive background in large‑scale internet and big‑data platforms and outlines how Data Fabric and data virtualization can be applied to data integration, highlighting the differences from traditional solutions and the business value of logical data warehouses.

Big DataData FabricData Integration
0 likes · 2 min read
Practices of Data Fabric in Data Integration Scenarios
DataFunTalk
DataFunTalk
Sep 30, 2023 · Big Data

Building a Marketing‑Oriented Data Middle Platform: Concepts and Practices

This article outlines how a marketing‑focused data middle platform can be constructed by integrating online and offline behavior data, business data, and third‑party sources, then applying data integration, modeling, processing, and application capabilities to enable data‑driven user journeys and personalized marketing strategies.

Big DataData IntegrationData Modeling
0 likes · 13 min read
Building a Marketing‑Oriented Data Middle Platform: Concepts and Practices
Java High-Performance Architecture
Java High-Performance Architecture
Sep 28, 2023 · Databases

How to Use Debezium for MySQL CDC in Spring Boot Without Adding Extra Middleware

Learn how to capture MySQL data changes using Debezium's CDC capabilities within a Spring Boot application, avoiding heavyweight message brokers by leveraging binlog monitoring, configuring connectors, handling snapshots, and processing change events for use cases like cache invalidation, data integration, and simplifying monolithic architectures.

CDCData IntegrationDebezium
0 likes · 24 min read
How to Use Debezium for MySQL CDC in Spring Boot Without Adding Extra Middleware
Architects Research Society
Architects Research Society
Sep 27, 2023 · Fundamentals

What Is the Common Data Model and Why Use It?

The Common Data Model provides a shared, standardized data language and metadata system that simplifies cross‑application data integration, reduces custom development effort, and enables consistent, extensible data structures for business and analytics scenarios across Microsoft Power Platform and Azure services.

Common Data ModelData IntegrationMicrosoft Power Platform
0 likes · 8 min read
What Is the Common Data Model and Why Use It?
DataFunSummit
DataFunSummit
Sep 8, 2023 · Big Data

Tianqiong OLAP Real‑time Lakehouse Fusion Platform Architecture Practice

This article explains why lake‑warehouse fusion is needed, describes the challenges of integrating real‑time data warehouses with data lakes, introduces a new StarRocks‑based architecture that supports real‑time ingestion, cooling, offline loading, and adaptive hot‑cold query rewriting, and outlines future plans and Q&A.

Big DataData IntegrationData Warehouse
0 likes · 21 min read
Tianqiong OLAP Real‑time Lakehouse Fusion Platform Architecture Practice
DataFunSummit
DataFunSummit
Aug 13, 2023 · Big Data

KwaiBI: Evolution of Kuaishou’s One‑Stop Business Intelligence Platform from 1.0 to 2.0

The article details Kuaishou’s KwaiBI business intelligence platform evolution, covering its 1.0 tool‑based implementation, the 2.0 standardized architecture built on an indicator middle‑platform, core processes, data integration, self‑service features, and future directions for self‑service and intelligent analytics.

BIBig DataData Integration
0 likes · 22 min read
KwaiBI: Evolution of Kuaishou’s One‑Stop Business Intelligence Platform from 1.0 to 2.0
DataFunSummit
DataFunSummit
Aug 10, 2023 · Databases

ClickHouse Deployment in Lenovo Manufacturing: Architecture, Data Integration, and Performance Optimization

This article details Lenovo's implementation of ClickHouse in a manufacturing environment, covering the current data landscape, cluster architecture, integration challenges, performance enhancements, and solutions such as Seatunnel and query pre‑aggregation, illustrating how OLAP engines can address real‑time analytics and concurrency issues in production data pipelines.

ClickHouseData IntegrationSeaTunnel
0 likes · 11 min read
ClickHouse Deployment in Lenovo Manufacturing: Architecture, Data Integration, and Performance Optimization
Data Thinking Notes
Data Thinking Notes
Aug 2, 2023 · Fundamentals

Mastering Enterprise Data: A Practical Guide to Master Data Management

This article explains why fragmented data hampers business insight in large enterprises and provides a comprehensive overview of master data concepts, governance structures, standards, processes, and step‑by‑step implementation practices to achieve consistent, high‑quality enterprise data.

Data GovernanceData IntegrationMDM
0 likes · 18 min read
Mastering Enterprise Data: A Practical Guide to Master Data Management
Architects Research Society
Architects Research Society
Aug 2, 2023 · Fundamentals

Data Fabric Architecture: Three Patterns, Core Technical Components, and Inherent Limitations

The article explains data fabric architecture as a promising approach for enabling data exchange across distributed systems, outlines its three design patterns, describes key technical components such as data virtualization, data catalog, and knowledge graphs, and discusses the trade‑offs, costs, and limitations that organizations must consider.

Data FabricData IntegrationKnowledge Graph
0 likes · 17 min read
Data Fabric Architecture: Three Patterns, Core Technical Components, and Inherent Limitations
Didi Tech
Didi Tech
Jul 31, 2023 · Big Data

Data Serviceization at Didi: Architecture, Phases, and Standard Metric Service

Didi’s data serviceization converts raw business data into consumable services through a four‑stage pipeline—integration, development, production, and back‑flow—while the Data Dream Factory and Shu‑Chain platform automate synchronization, provide a unified access gateway for thousands of APIs, and introduce a standard metric service that abstracts storage complexities and ensures high‑performance, secure data delivery.

Data Integrationdata platformdata serviceization
0 likes · 16 min read
Data Serviceization at Didi: Architecture, Phases, and Standard Metric Service
Inke Technology
Inke Technology
Jun 28, 2023 · Big Data

Extending Apache Seatunnel for Trino and Kyuubi Integration: A Practical Guide

This article outlines the challenges of scaling data integration platforms, proposes a comprehensive solution using Apache Seatunnel and Dinky, details the implementation of Trino and Kyuubi JDBC support, and describes the platform's architecture, task publishing workflow, logging, monitoring, resource management, and future enhancements.

Apache SeaTunnelData IntegrationData Pipeline
0 likes · 16 min read
Extending Apache Seatunnel for Trino and Kyuubi Integration: A Practical Guide
Architects Research Society
Architects Research Society
Jun 21, 2023 · Fundamentals

The Strategic Role of Enterprise Architects: Five Strategic and One Tactical Focus Areas

Enterprise architects align IT strategy with business goals by overseeing application portfolio management, technology and risk, IT operations, security and privacy, integration and data, and finance, defining roadmaps for 1‑3‑5 year plans while balancing strategic and tactical responsibilities in a rapidly changing environment.

Application PortfolioData IntegrationIT Operations
0 likes · 6 min read
The Strategic Role of Enterprise Architects: Five Strategic and One Tactical Focus Areas
21CTO
21CTO
Jun 20, 2023 · Fundamentals

ETL vs ELT: Which Data Integration Method Wins for Your Business?

ETL extracts, transforms, then loads data, while ELT extracts, loads, and transforms later, each offering distinct advantages; the article compares their processes, key differences, and factors such as data volume, complexity, latency, and cost to help businesses choose the optimal integration approach.

Data IntegrationData WarehousingELT
0 likes · 12 min read
ETL vs ELT: Which Data Integration Method Wins for Your Business?
360 Tech Engineering
360 Tech Engineering
Jun 2, 2023 · Big Data

Overcoming Challenges in User Profiling: A Big Data‑Driven Framework for Precise Marketing

The article outlines how a unified, big‑data‑based user profiling platform addresses traditional data silos, high costs, and limited functionality by standardizing tags, integrating Spark and RHadoop processing, and enabling a closed‑loop marketing workflow that improves accuracy and operational efficiency.

Big DataData IntegrationMarketing Automation
0 likes · 7 min read
Overcoming Challenges in User Profiling: A Big Data‑Driven Framework for Precise Marketing
StarRocks
StarRocks
May 26, 2023 · Big Data

How SeaTunnel’s StarRocks Connector Enables High‑Performance Data Sync

This article explains SeaTunnel’s architecture and its StarRocks connector, detailing source and sink features such as field projection, predicate push‑down, parallel reading, state recovery, data type mapping, Stream Load writes, CDC support, configuration examples, and future roadmap for exactly‑once semantics.

Big DataConnectorData Integration
0 likes · 16 min read
How SeaTunnel’s StarRocks Connector Enables High‑Performance Data Sync
Top Architect
Top Architect
May 4, 2023 · Big Data

Data Middle Platform: General Architecture and Core Components

The article explains the concept, benefits, and detailed modular architecture of a data middle platform, covering data storage, acquisition, processing, governance, security, and operation frameworks, and illustrates how enterprises can build and evolve such platforms to turn data into valuable services.

Big DataData GovernanceData Integration
0 likes · 19 min read
Data Middle Platform: General Architecture and Core Components
ITPUB
ITPUB
Apr 26, 2023 · Databases

Mastering Change Data Capture: Open‑Source Tools and How to Choose the Right One

This article explains the concept of Change Data Capture (CDC), outlines its common use cases, compares the main technical approaches—including timestamps, data diff, triggers, and log‑based methods—and reviews popular open‑source CDC solutions and their database‑specific configuration requirements.

CDCChange Data CaptureData Integration
0 likes · 15 min read
Mastering Change Data Capture: Open‑Source Tools and How to Choose the Right One
ITPUB
ITPUB
Apr 25, 2023 · Big Data

Top 8 Open‑Source ETL Tools for Data Migration and Integration

This article reviews eight widely used ETL and data‑migration tools—including Kettle, DataX, DataPipeline, Talend, DataStage, Sqoop, FineDataLink, and Canal—detailing their core features, architectures, supported data sources, and typical usage scenarios to help practitioners choose the right solution.

Big DataData IntegrationData Migration
0 likes · 13 min read
Top 8 Open‑Source ETL Tools for Data Migration and Integration
DataFunSummit
DataFunSummit
Apr 9, 2023 · Big Data

Expert Interview: Architecture and Trends of Big Data Platforms

This article presents a comprehensive interview with several big‑data platform experts, outlining the core components such as data integration, storage and computation, distributed scheduling, and query analysis, while also highlighting current challenges, best‑practice tools, and future trends in big‑data architecture.

Big DataData IntegrationDistributed Computing
0 likes · 10 min read
Expert Interview: Architecture and Trends of Big Data Platforms
DataFunTalk
DataFunTalk
Apr 4, 2023 · Big Data

Upgrading Hangzhou Bank Consumer Finance Big Data Platform with Apache Doris 1.2: Architecture, Performance Gains, and Integration

This article details how Hangzhou Bank Consumer Finance modernized its big‑data platform by introducing Apache Doris 1.2, replacing the original Greenplum + CDH architecture, unifying data sources via Multi‑Catalog, achieving second‑level query latency, reducing storage and compute costs, and outlining the integration workflow with DolphinScheduler, SeaTunnel, and Spark.

Apache DorisBig DataData Integration
0 likes · 20 min read
Upgrading Hangzhou Bank Consumer Finance Big Data Platform with Apache Doris 1.2: Architecture, Performance Gains, and Integration
HomeTech
HomeTech
Mar 31, 2023 · Artificial Intelligence

Digital Transformation of Used‑Car Buying: Integrated Data, AI Valuation, and VR Visualization

The article describes how a comprehensive digital platform combines structured, semi‑structured, and panoramic data with machine‑learning valuation models, natural‑language processing, and VR technology to make used‑car condition information transparent, improve estimation accuracy, and enhance user decision‑making in the Chinese second‑hand car market.

AI valuationBig DataData Integration
0 likes · 15 min read
Digital Transformation of Used‑Car Buying: Integrated Data, AI Valuation, and VR Visualization
macrozheng
macrozheng
Mar 27, 2023 · Big Data

Top 8 Open-Source ETL Tools for Efficient Data Migration

This guide reviews eight popular ETL and data migration tools—including Kettle, DataX, DataPipeline, Talend, DataStage, Sqoop, FineDataLink, and Canal—detailing their core features, architectures, and use cases to help engineers choose the right solution for reliable data integration.

Big DataData IntegrationData Migration
0 likes · 14 min read
Top 8 Open-Source ETL Tools for Efficient Data Migration
Su San Talks Tech
Su San Talks Tech
Mar 24, 2023 · Big Data

Top 8 Open-Source ETL Tools You Should Know for Efficient Data Migration

Explore a comprehensive overview of eight popular ETL and data migration tools—including Kettle, DataX, DataPipeline, Talend, DataStage, Sqoop, FineDataLink, and Canal—detailing their features, architectures, and use cases to help you choose the right solution for efficient data integration.

Big DataData IntegrationData Migration
0 likes · 13 min read
Top 8 Open-Source ETL Tools You Should Know for Efficient Data Migration
Architects Research Society
Architects Research Society
Mar 5, 2023 · Big Data

Best Open‑Source and Commercial ETL Tools: Detailed Comparison

This article introduces the concept of ETL, explains its importance for modern data‑driven applications, and provides a comprehensive comparison of the most popular open‑source and commercial ETL platforms—including their key features, supported data sources, and deployment options—helping readers choose the right tool for their data integration needs.

Big DataCommercialData Integration
0 likes · 19 min read
Best Open‑Source and Commercial ETL Tools: Detailed Comparison
HomeTech
HomeTech
Mar 1, 2023 · Backend Development

Overview of the Wenjie Low-Code Platform: Architecture, Technologies, and Use Cases

The article presents a comprehensive overview of the Wenjie low-code platform, detailing its motivation, front‑end React framework, back‑end Spring Cloud micro‑services architecture, PowerJob scheduler, custom ORM, various data‑modeling and data‑processing scenarios, dashboard visualizations, monitoring and alerting features, as well as future plans and a concluding summary.

Data IntegrationLow-codePlatform
0 likes · 11 min read
Overview of the Wenjie Low-Code Platform: Architecture, Technologies, and Use Cases
DataFunTalk
DataFunTalk
Feb 27, 2023 · Big Data

Comprehensive Overview of Data Middle Platform Architecture and Its Core Frameworks

This article provides a detailed overview of data middle platform concepts, describing a decoupled six‑subsystem architecture—including storage, collection, processing, governance, security, and operation frameworks—while illustrating typical enterprise implementations, industry‑specific solutions, and best‑practice considerations for building scalable, secure, and value‑driven data platforms.

Big DataData GovernanceData Integration
0 likes · 25 min read
Comprehensive Overview of Data Middle Platform Architecture and Its Core Frameworks
Aikesheng Open Source Community
Aikesheng Open Source Community
Feb 24, 2023 · Databases

SQLE 2.2302.0 Release Notes – New Features, Enhancements, and Bug Fixes

The SQLE 2.2302.0 release introduces data source import from external platforms, operation‑record viewing, improved CloudBeaver integration, manual deployment support, numerous UI optimizations, and a long list of bug fixes, providing a more seamless and secure SQL auditing experience for both enterprise and community users.

Bug FixesCloudBeaverData Integration
0 likes · 8 min read
SQLE 2.2302.0 Release Notes – New Features, Enhancements, and Bug Fixes
DataFunTalk
DataFunTalk
Feb 2, 2023 · Big Data

SeaTunnel: Design Goals, Current Status, Architecture, and Future Roadmap

This article provides a comprehensive overview of Apache SeaTunnel, covering its design objectives, current capabilities such as multi‑engine support and extensive connector ecosystem, detailed architecture including engine‑independent APIs and execution flows, and outlines the upcoming roadmap to expand connectors, launch a visual web UI, and introduce a dedicated SeaTunnel Engine.

Batch ProcessingBig DataConnector
0 likes · 12 min read
SeaTunnel: Design Goals, Current Status, Architecture, and Future Roadmap
Data Thinking Notes
Data Thinking Notes
Jan 31, 2023 · Fundamentals

Mastering Data Governance: From Metadata to ETL in One Guide

This comprehensive guide walks you through the entire data governance ecosystem, covering metadata fundamentals, classification, maturity models, data standards, modeling, integration, lifecycle management, quality assurance, security, and ETL processes, all illustrated with clear diagrams and practical steps.

Data GovernanceData IntegrationData Modeling
0 likes · 13 min read
Mastering Data Governance: From Metadata to ETL in One Guide
DataFunTalk
DataFunTalk
Jan 31, 2023 · Big Data

SPI Refactoring Practice in Apache InLong Manager to Reduce Maintenance Cost and Enhance Extensibility

This article presents the SPI-based refactoring of Apache InLong Manager, describing the project's background, existing maintenance challenges, the concept of Java Service Provider Interface, the concrete implementation steps, code restructuring, and the resulting benefits such as higher code reuse, easier extension, and reduced DDL changes.

Apache InLongBig DataCode Refactoring
0 likes · 10 min read
SPI Refactoring Practice in Apache InLong Manager to Reduce Maintenance Cost and Enhance Extensibility
Data Thinking Notes
Data Thinking Notes
Jan 29, 2023 · Big Data

How to Turn Data Assets into Business Value: A Roadmap for Enterprises

Enterprises must shift their perception of data assets and embed data‑value into every digital process, establishing governance, unified asset catalogs, operational metrics, security controls, integration, services, and visualization to transform raw data into strategic business outcomes.

Big DataData GovernanceData Integration
0 likes · 12 min read
How to Turn Data Assets into Business Value: A Roadmap for Enterprises
Code Ape Tech Column
Code Ape Tech Column
Jan 28, 2023 · Big Data

Using Alibaba DataX for Offline Data Synchronization and Incremental Sync

This article introduces Alibaba DataX, explains its architecture and role in offline heterogeneous data synchronization, provides step‑by‑step Linux installation, demonstrates full‑load and incremental MySQL‑to‑MySQL sync with JSON job templates, and shares practical tips for handling large data volumes.

Data IntegrationDataXETL
0 likes · 15 min read
Using Alibaba DataX for Offline Data Synchronization and Incremental Sync
ITPUB
ITPUB
Jan 26, 2023 · Big Data

How NetEase’s Arctic Unifies Streaming and Batch with Iceberg for Real‑Time Lakehouse

This article explains the challenges of a Lambda‑architecture data pipeline, introduces NetEase’s Arctic lakehouse built on Apache Iceberg, details its table‑store design, optimization cycles, consistency mechanisms, real‑time features, practical use cases, and future roadmap, highlighting its advantages over similar solutions.

ArcticData IntegrationFlink
0 likes · 14 min read
How NetEase’s Arctic Unifies Streaming and Batch with Iceberg for Real‑Time Lakehouse
DataFunSummit
DataFunSummit
Jan 24, 2023 · Big Data

Building a Real-Time Data and User Profiling Architecture with Apache Doris at Zhihu

The article details Zhihu's data empowerment team's design and implementation of a low‑cost, high‑response real‑time data platform built on Apache Doris, covering real‑time business metrics, algorithm features, and user profiling, and explains the challenges, architectural choices, tooling, performance gains, and future directions.

Apache DorisData IntegrationLambda Architecture
0 likes · 22 min read
Building a Real-Time Data and User Profiling Architecture with Apache Doris at Zhihu
JD Tech
JD Tech
Jan 13, 2023 · Big Data

UData: Solving the Last Mile of Data Usage – Architecture, Query Engine Design, and Federated Query Enhancements

This article introduces the UData platform, explains its data‑integration architecture, details the StarRocks‑based query engine workflow from SQL parsing to distributed execution, and describes recent optimizations such as computation push‑down, support for JSF/HTTP/ClickHouse external tables, and a proxy‑based federated query framework.

Big DataData IntegrationFederated Query
0 likes · 20 min read
UData: Solving the Last Mile of Data Usage – Architecture, Query Engine Design, and Federated Query Enhancements
Alibaba Cloud Big Data AI Platform
Alibaba Cloud Big Data AI Platform
Jan 12, 2023 · Operations

What Is DataOps and How Can It Transform Your Data Management?

DataOps, the data‑centric counterpart of DevOps, combines agile principles, standardized tools, and cross‑team collaboration to manage the full data lifecycle—from integration and development to storage, governance, and service—enabling organizations to handle massive, diverse datasets efficiently, reduce silos, and turn data into actionable value.

Big DataData GovernanceData Integration
0 likes · 15 min read
What Is DataOps and How Can It Transform Your Data Management?
DataFunTalk
DataFunTalk
Jan 6, 2023 · Big Data

ZhongAn's Hundred‑Billion‑Scale Data Integration Service: Architecture, Business Support, and Evolution

This article presents the architecture and practical experience of ZhongAn's hundred‑billion‑scale data integration service, covering common integration technologies, business support scenarios for offline and real‑time data, technical challenges, evolution from single‑machine to service‑oriented designs, and future directions using Flink and DataX.

Data IntegrationDataXETL
0 likes · 31 min read
ZhongAn's Hundred‑Billion‑Scale Data Integration Service: Architecture, Business Support, and Evolution
DataFunTalk
DataFunTalk
Nov 29, 2022 · Big Data

Summary of Flink Forward Asia 2022: Keynotes, Technical Innovations, and Industry Deployments of Apache Flink

The 2022 Flink Forward Asia conference highlighted Apache Flink’s rapid growth, showcased major technical advances such as upgraded checkpointing, cloud‑native state storage, Hybrid Shuffle, Flink CDC 2.0, and Flink ML 2.0, and presented real‑world deployments from Alibaba, Midea, miHoYo, and Disney.

Apache FlinkData IntegrationReal-time Streaming
0 likes · 25 min read
Summary of Flink Forward Asia 2022: Keynotes, Technical Innovations, and Industry Deployments of Apache Flink
Alibaba Cloud Big Data AI Platform
Alibaba Cloud Big Data AI Platform
Nov 29, 2022 · Big Data

How Flink’s Stream‑Batch Fusion Is Transforming Real‑Time Big Data

The article explores Apache Flink’s eight‑year journey to becoming a top‑level Apache project, Alibaba’s extensive contributions, the rise of stream‑batch unified computing, its impact on real‑time data integration, cloud‑native deployment, and the emerging Flink‑based data‑warehouse and serverless solutions.

Apache FlinkBig DataCloud Native
0 likes · 15 min read
How Flink’s Stream‑Batch Fusion Is Transforming Real‑Time Big Data
DataFunTalk
DataFunTalk
Nov 6, 2022 · Big Data

BitSail: ByteDance’s Open‑Source Unified Data Integration Engine – Architecture, Evolution, and Capabilities

BitSail, an open‑source data integration engine from ByteDance, provides a unified solution for batch, streaming, full‑load, and incremental data synchronization across heterogeneous sources, detailing its background, technical evolution, architecture, low‑cost co‑building features, compatibility strategies, and future roadmap.

CDCData IntegrationFlink
0 likes · 18 min read
BitSail: ByteDance’s Open‑Source Unified Data Integration Engine – Architecture, Evolution, and Capabilities
IT Services Circle
IT Services Circle
Oct 26, 2022 · Databases

Debezium: Open‑Source Change Data Capture Platform – Overview, Architecture, Use Cases, and Installation Guide

This article introduces Debezium, an open‑source low‑latency change data capture platform that streams database row changes via Kafka, explains its architecture and common scenarios such as cache invalidation and CQRS, and provides step‑by‑step Docker commands to install ZooKeeper, Kafka, MySQL and the Debezium connector.

CDCData IntegrationDebezium
0 likes · 15 min read
Debezium: Open‑Source Change Data Capture Platform – Overview, Architecture, Use Cases, and Installation Guide
Big Data Technology Architecture
Big Data Technology Architecture
Oct 25, 2022 · Big Data

Rebuilding Shopee's Data Integration Platform with Apache SeaTunnel

Shopee faced fragmented data‑ingestion pipelines, limited source support, and high maintenance overhead, so it evaluated open‑source tools and adopted Apache SeaTunnel to unify batch and streaming data transfers, simplify ETL workflows, and provide a scalable, extensible solution for its multi‑TB daily data processing needs.

Data IntegrationETLSeaTunnel
0 likes · 17 min read
Rebuilding Shopee's Data Integration Platform with Apache SeaTunnel
Alibaba Cloud Native
Alibaba Cloud Native
Sep 29, 2022 · Cloud Native

Why Use RocketMQ Connect for Scalable Data Pipelines?

This article explains the challenges of point‑to‑point data sync, introduces RocketMQ Connect as a cloud‑native solution that decouples upstream and downstream, details its architecture, connectors, REST API, metrics, deployment modes, and provides a step‑by‑step guide to building custom connectors for use cases such as CDC, data lakes, and system migration.

CDCCloud NativeConnector
0 likes · 19 min read
Why Use RocketMQ Connect for Scalable Data Pipelines?
HomeTech
HomeTech
Sep 13, 2022 · Big Data

Integrating Heterogeneous Data Sources with openLooKeng and Upgrading the Apache Kylin Connector at AutoHome

This article describes how AutoHome tackled the complexity of managing multiple relational, NoSQL, and Hive data stores by adopting openLooKeng for unified, cross‑source SQL queries, outlines its key features such as ANSI‑SQL support, diverse connectors, and query optimizations, and details the custom enhancements made to the Apache Kylin connector to better serve their commercial data analysis workloads.

Big DataConnectorsData Integration
0 likes · 13 min read
Integrating Heterogeneous Data Sources with openLooKeng and Upgrading the Apache Kylin Connector at AutoHome
Top Architect
Top Architect
Sep 11, 2022 · Frontend Development

Optimizing Frontend‑Backend Collaboration with Data Direct Access at Baidu Commercial Frontend Team

This article describes how Baidu's commercial frontend team improved front‑end and back‑end collaboration efficiency by introducing a BFF layer, data‑direct capabilities, staged data tiering, and fragment‑based batch editing, ultimately reducing delivery time by over 50% while maintaining quality.

BFFData IntegrationEfficiency
0 likes · 9 min read
Optimizing Frontend‑Backend Collaboration with Data Direct Access at Baidu Commercial Frontend Team
Architects Research Society
Architects Research Society
Sep 6, 2022 · Fundamentals

Understanding Microsoft’s Common Data Model: Components, Benefits, and Real‑World Use

The article explains how Microsoft’s Common Data Model provides a shared metadata system and standardized, extensible data schemas that simplify integration across Power Apps, Power BI, Dynamics 365 and Azure, enabling consistent data semantics, easier app development, and scalable enterprise solutions.

Common Data ModelData IntegrationDynamics 365
0 likes · 7 min read
Understanding Microsoft’s Common Data Model: Components, Benefits, and Real‑World Use
Architect
Architect
Sep 2, 2022 · Frontend Development

Optimizing Front‑End and Back‑End Collaboration with Data Direct Capability at Baidu Commercial Front‑End Team

The article describes how Baidu's commercial front‑end team introduced a data‑direct capability and BFF layer to streamline front‑end/back‑end cooperation, reduce environment‑maintenance overhead, enable parallel development, and improve overall delivery efficiency across multiple project phases.

BFFData IntegrationFrontend
0 likes · 9 min read
Optimizing Front‑End and Back‑End Collaboration with Data Direct Capability at Baidu Commercial Front‑End Team
Shopee Tech Team
Shopee Tech Team
Sep 2, 2022 · Big Data

Shopee Data System Challenges and Apache Hudi Practices

Shopee tackled its data‑system bottlenecks by customizing Apache Hudi to provide unified stream‑batch integration, efficient state‑detail snapshots, and low‑latency wide‑table generation, using CDC‑based bootstrapping, COW/MOR tables, savepoints and partial updates, which cut latency to ten minutes, lowered resource use, and yielded several community‑backed enhancements.

Apache HudiBig DataData Integration
0 likes · 18 min read
Shopee Data System Challenges and Apache Hudi Practices
DevOps Cloud Academy
DevOps Cloud Academy
Aug 8, 2022 · Operations

Understanding DataOps ETL: Benefits, Automation, and Implementation Guide

This article explains DataOps and its role in modern ETL pipelines, outlines the benefits of DataOps for efficiency and reliability, and provides a detailed roadmap and best‑practice guidelines for planning, implementing, and optimizing DataOps‑driven ETL in cloud‑native environments.

AutomationData EngineeringData Integration
0 likes · 13 min read
Understanding DataOps ETL: Benefits, Automation, and Implementation Guide
Big Data Technology & Architecture
Big Data Technology & Architecture
Aug 4, 2022 · Big Data

Comprehensive Guide to DataX: Introduction, Architecture, Usage, and Deployment

This article provides a detailed overview of DataX, covering its purpose, framework design, core architecture, scheduling process, practical examples of MySQL-to-MySQL synchronization, step‑by‑step installation and configuration of DataX‑WEB, UI usage, routing strategies, task types, and advanced task building techniques.

Big DataData IntegrationDataX
0 likes · 14 min read
Comprehensive Guide to DataX: Introduction, Architecture, Usage, and Deployment
DataFunTalk
DataFunTalk
Jul 30, 2022 · Databases

StarRocks-Based Unified Data Service and Analytics Platform at JD Logistics

JD Logistics leverages StarRocks to create the Udata unified query engine, addressing data silos, low performance, and high maintenance costs by integrating data services and analytics, enabling low‑code data service generation, high‑speed federated queries, real‑time updates, and future data‑lake and resource isolation capabilities.

Data IntegrationDatabaseReal-Time Analytics
0 likes · 14 min read
StarRocks-Based Unified Data Service and Analytics Platform at JD Logistics
Big Data Technology Architecture
Big Data Technology Architecture
Jul 15, 2022 · Big Data

Using and Designing the Apache SeaTunnel Examples Module

This article introduces Apache SeaTunnel's Examples module, compares SeaTunnel with DataX, explains its multi‑engine design, demonstrates Flink and Spark example implementations, and shares the speaker's experiences contributing to the open‑source community, providing practical guidance for big‑data integration projects.

Apache SeaTunnelData IntegrationFlink
0 likes · 10 min read
Using and Designing the Apache SeaTunnel Examples Module
dbaplus Community
dbaplus Community
Jul 13, 2022 · Big Data

Unpacking the Core Technologies Behind Modern Big Data Platforms

From data ingestion to real‑time analytics, this guide breaks down the essential layers of a typical big‑data platform—covering collection methods, HDFS storage, Hive/Spark analysis, data sharing mechanisms, application use‑cases, streaming with Spark Streaming, and the need for robust scheduling and monitoring.

Big DataData IntegrationData Warehouse
0 likes · 9 min read
Unpacking the Core Technologies Behind Modern Big Data Platforms
JD Retail Technology
JD Retail Technology
Jun 10, 2022 · Big Data

Design and Implementation of an International Business Data Platform for JD.com's 618 Promotion

The article details JD International's challenges and solutions in building a unified, real‑time data platform for its multi‑regional 618 promotion, covering business characteristics, data distribution, team organization, dashboard architecture, integration strategies, and short‑ and long‑term technical plans.

Data IntegrationFlinkInternational Business
0 likes · 8 min read
Design and Implementation of an International Business Data Platform for JD.com's 618 Promotion
Big Data Technology Architecture
Big Data Technology Architecture
Jun 9, 2022 · Databases

Building a Real‑Time Data Warehouse with Apache Doris: Architecture, Benefits, and Lessons Learned

This article details how a fast‑growing supply‑chain platform migrated from MySQL and Hive to Apache Doris for real‑time analytics, describing the architectural evolution, the advantages of the new design, practical implementation steps, encountered challenges, and the performance and cost benefits achieved.

Apache DorisData IntegrationFlink CDC
0 likes · 12 min read
Building a Real‑Time Data Warehouse with Apache Doris: Architecture, Benefits, and Lessons Learned
IT Architects Alliance
IT Architects Alliance
Jun 7, 2022 · Databases

Introduction to Change Data Capture (CDC) Practices

This article introduces the concept and practice of Change Data Capture (CDC), explaining how it captures database changes to provide real‑time incremental data for analytics and reporting without impacting source performance, and outlines modern CDC methods, challenges, and production‑ready system requirements.

CDCChange Data CaptureData Integration
0 likes · 8 min read
Introduction to Change Data Capture (CDC) Practices
Top Architect
Top Architect
Jun 7, 2022 · Databases

An Introduction to Change Data Capture (CDC) Practices and Modern Approaches

This article introduces the concept of Change Data Capture (CDC), explains why traditional batch reporting strains resources, describes how CDC captures only data changes to keep source databases performant, and outlines modern CDC architectures, production‑ready considerations, and best‑practice guidelines for building reliable data pipelines.

CDCChange Data CaptureData Integration
0 likes · 16 min read
An Introduction to Change Data Capture (CDC) Practices and Modern Approaches