1 views collected around this technical thread.
This article explains how Baidu's vertical search platform tackled massive data growth and scaling challenges by redesigning its data management system, introducing elastic scheduling, decoupling ETCD access, implementing auto‑scaling, and advancing shard expansion to improve performance, stability, and cost efficiency.
This article examines Alibaba Cloud’s Qi Tian integrated operation‑management platform, detailing the challenges of massive cloud network management and the innovative data‑fusion, automated change, intent‑aware monitoring, and multi‑plane self‑healing technologies that enable secure, high‑performance operation at million‑device scale.
The article details a large‑scale Chinese New Year interactive activity’s front‑end architecture, describing a layered system for business logic, data abstraction, and animation engines, unified data handling, dynamic animation rendering with downgrade paths, high‑concurrency QPS reduction, resilience measures, and extensive performance and workflow optimizations.
The article introduces Easy DataSet, an open‑source tool that streamlines the creation of domain‑specific datasets by aggregating public data sources, chunking Markdown documents, generating and managing QA pairs with configurable LLM endpoints, and exporting them in common formats, while outlining its architecture and future roadmap.
This article outlines five common IT architecture design errors—neglecting connectivity, postponing security, poor compatibility, uncontrolled data duplication, and unsynchronized environments—illustrated with real cases and provides practical strategies to prevent each pitfall and build resilient, efficient systems.
This article presents the background, design principles, architecture, and practical deployment of Xiaomi's AI Data Management Platform, highlighting how unified cataloging, Fileset integration, and notebook‑based development address AI data governance, cost reduction, and workflow efficiency for both structured and non‑structured data.
This article details JD's large‑scale HDFS unified storage implementation, covering cross‑region storage challenges, topology design, asynchronous block replication, flow‑control mechanisms, tiered storage strategies, automatic hot‑cold data migration, and the resulting performance and cost improvements for big‑data workloads.
This article examines how data quality governance is applied in the financial sector, covering regulatory background, key challenges, management system design, practical methodologies, and evaluation standards to improve data assets and support digital transformation.
This article presents JD Retail's large‑scale HDFS deployment, detailing its unified storage architecture, cross‑region data replication challenges and solutions, tiered storage strategies for hot, warm and cold data, and the operational modules that together improve performance, reliability and cost efficiency in a big‑data environment.
This article surveys data management for large language model training, covering an overview, pre‑training data composition, scaling‑law‑driven quantity control, quality filtering, deduplication, harmful‑content removal, instruction fine‑tuning strategies, dynamic data selection, and emerging research challenges such as bias mitigation, multimodal data handling, and synthetic‑data filtering.
This article reflects on the challenges of data governance, emphasizing that effective governance involves not only technical data handling but also managing people, aligning responsibilities, fostering cooperation between leadership and business units, and establishing clear ownership and incentive mechanisms.
This article provides a comprehensive overview of object storage, covering its origins, historical development, fundamental operations, key advantages over block and file storage, major application scenarios, and predicted future trends in cloud and big‑data environments.
This article details JD Retail’s large‑scale HDFS deployment, describing how cross‑region storage challenges were solved with a full‑copy topology, asynchronous block replication, flow‑control mechanisms, and a tiered storage strategy that automatically moves hot, warm, and cold data among SSD, HDD, and high‑density HDD nodes to improve performance and cut costs.
This article shares Ant Group's extensive experience in big data governance, detailing the overall data governance framework, data quality management, data storage governance, and future considerations, illustrated with practical cases and strategies for ensuring compliance, reliability, and cost efficiency.
This article presents a comprehensive overview of the Apache‑incubated Paimon project, covering its evolution from Flink Table Store, the current features of primary‑key and log tables, management tools such as snapshots, tags and branches, performance optimizations for Flink and Spark, and a detailed roadmap of upcoming functionalities.
This article provides a comprehensive overview of AI large‑model development, defines what constitutes a large model, discusses current challenges such as cost, privacy and safety, and examines how these models can improve efficiency across automotive marketing, sales, service, data management, infrastructure building, and future automation stages.
The author recounts taking the Elastic Certified Engineer exam in March 2024, outlines the official syllabus, shares ten hands‑on exam tasks with code snippets, and provides practical tips and personal reflections on preparation and execution.
This article explains how agile methodologies can be applied to data management, covering the need for agile data practices, core principles, iterative modeling, governance, CI/CD pipelines, tooling, metrics, security, case studies, challenges, and future outlooks in a comprehensive, step‑by‑step guide.
This article introduces the background, challenges, and cloud‑native solutions of lakehouse architecture, explains Apache Iceberg’s open table format and its cloud‑native features, details Amoro’s management and self‑optimizing capabilities, showcases three real‑world cloud migration cases, and outlines future development plans.
The article explains how the explosive growth to the zettabyte scale reshapes enterprise data strategy, emphasizing business‑driven value, big‑data management practices, and integrated processes that turn massive data into actionable insights for competitive advantage.