The article describes a two‑part mobile app performance monitoring system that automatically instruments code to capture method execution times, ANR and frame stalls, then processes, cleans, aggregates, and visualizes the data on a backend platform to generate alerts, trend reports, and guide optimization across versions.
This guide shows how to use the jq command‑line tool to elegantly format, query, and extract data from JSON responses across Linux, Windows, and macOS, providing practical examples, syntax basics, and integration with curl for API testing.
The article describes how high‑frequency LiDAR scanners on precision mapping vehicles suffer from Multi‑Time‑Around (MTA) errors—mis‑assigning distant returns to near ranges—and explains both internal laser strategies (continuity assumption and variable‑period emission) and a four‑step neighborhood‑weighting algorithm that reliably corrects these artifacts, restoring accurate point‑clouds for automated map generation.
This article walks through reading monthly rainfall data with pandas, cleaning missing values, calculating rainfall deviations, generating descriptive paragraphs, and rendering a formatted Word report using docxtpl, providing complete code snippets and example outputs for each step.
Apache Beam 2.28.0 introduces extensive Parquet support, new hash functions in BeamSQL and ZetaSQL, ApproximateDistinct via HLL, enhanced I/O connectors including SpannerIO for Numeric fields, ParquetIO schema support, KafkaTableProvider thrift, HadoopFormatIO key/value cloning skip, and various other improvements.
This article provides PHP code examples for reading data from a CSV file and writing data to a new CSV file, including handling of locale settings, skipping headers, constructing headers, and appending rows using built‑in functions such as fgetcsv, fopen, and fwrite.
This article details the implementation of a real-time push system for new house listings based on user search subscriptions.
This article presents ten big‑data interview problems, each describing a scenario such as finding the most frequent IP, top‑K queries, word frequency counting under memory limits, and techniques like hashing, bitmap, trie, heap, and external sorting to solve them efficiently.
This article introduces twelve popular open‑source Python frameworks—including Django, Tornado, Twisted, Pulsar, Bottle, Diesel, NumPy, Scrapy, Cubes, Falcon, Web2py, and Zerorpc—detailing their key features, typical use cases, and providing direct project URLs for developers seeking robust solutions.
This article walks you through creating a Python Tkinter application that fetches a waste‑classification dataset, processes it with pandas, and provides interactive learning and testing modules so users can practice sorting recyclable, hazardous, wet, and dry waste.
This article introduces Alink, Alibaba's open‑source machine‑learning platform built on Flink, detailing its core algorithms, performance advantages over Spark ML, version evolution, Maven and PyAlink installation steps, data‑source integrations, FM algorithm support, and unified file‑system operations for both batch and streaming workloads.
This article explains Youku's Android responsive solution, covering the responsive SDK, loading flow, architecture, data reprocessing, page and container responsiveness, and control size adaptation, providing practical guidelines for building a single app that adapts to diverse device sizes and form factors.
The article details Amap’s (Gaode) technical approach to building and deploying high‑precision maps for autonomous driving, covering accuracy requirements, data collection, point‑cloud alignment, AI‑driven perception and map‑update pipelines, and the challenges of scale, cost, and freshness.
This tutorial walks you through installing openpyxl, understanding Excel's workbook‑sheet‑cell hierarchy, and provides step‑by‑step code examples for reading, writing, styling, and manipulating Excel files using Python, serving as a handy reference for developers.
This tutorial explains how to construct a recommendation system by loading transaction data, creating sparse user‑item and item‑item matrices, applying Log‑Likelihood Ratio for item similarity, and indexing the results into ElasticSearch for real‑time serving, using Python and open‑source big‑data tools.
This article demonstrates how to leverage Java 8 Stream API to perform common collection operations such as defining a data class, creating test data, grouping by fields, converting lists to maps, filtering, summing numeric fields, finding max/min values, removing duplicates, and explains the Collectors utility methods.
This article demonstrates how to implement an advertising business data statistics pipeline using Spark and Kudu, detailing metric requirements, Scala processing code, complex SQL aggregations, schema design, and data sinking for verification.
This article walks through the complete implementation of an advertising statistics pipeline using Spark and Kudu, covering requirement analysis, Scala code development, SQL queries, schema definition, and data sinking, with full code snippets and execution results.
This article walks through a Spark and Kudu advertising project, explaining the refactoring approach, Scala trait usage, implementation of ETL and province‑city statistics processors, and shows the complete Spark application entry point with full code examples.
This article explains how to use JsonPath in Java to filter JSON data, covering operators, functions, and practical examples such as checking values in arrays, subsets, size constraints, and empty checks, with complete code snippets for each case.
Grid Studio is a web‑based spreadsheet that integrates Python, allowing users to read, write, and visualize Excel data with just a few lines of code, offering seamless installation, custom functions, and powerful data‑science tools like Plotly and Matplotlib for efficient data processing.
This article surveys the rapidly evolving big data landscape by reviewing a wide range of Apache projects—including Hadoop, Spark, Flink, HBase, Kudu, Impala, Kafka, and others—detailing their core components, architectures, strengths, and typical use‑cases for building distributed data platforms.
The voting service for 'You Are My Youth 2' was re‑architected using Docker‑based QAE and the Skywalker microservices platform, adding containerized one‑click scaling, cross‑data‑center MySQL/Couchbase/HBase high availability, and Hive/Impala real‑time processing, which doubled performance, cut preparation from 30 days to 12 hours, and incorporated third‑party audit verification.
This tutorial demonstrates how to combine Excel’s interactive features with Python’s Pandas library to perform comprehensive data operations—including reading, generating, filtering, sorting, handling missing values, deduplication, merging, grouping, calculation, statistics, visualization, sampling, pivot tables, and VLOOKUP—showing when each tool excels.
This article demonstrates how to scrape bullet comments from iQiyi for the first episode of a popular mystery series, decode the binary files, extract the text, and use Python's jieba and wordcloud libraries to clean the data and generate a visual word cloud of audience sentiments.
This tutorial demonstrates how to use Python's xlrd library to read an Excel file containing student records and retrieve a specific student's information by name or ID, covering installation, code walkthrough, and sample output.
This article explains how to combine the Flink data‑processing engine with TensorFlow to create a unified, end‑to‑end machine‑learning workflow, covering background, challenges, the Flink‑AI‑extended architecture, ML framework and operator abstractions, and both batch and streaming training and prediction modes.
This article demonstrates how Python's pandas library can replace manual Excel operations by reading, calculating, sorting, filtering, and visualizing data, offering efficient solutions for large datasets and showcasing code examples for automatic column filling, price adjustments, and student score analysis.
This article provides a comprehensive introduction to Apache Beam, covering its unified batch‑and‑stream processing architecture, programming model, workflow patterns, Lambda and Kappa architectures, the characteristics of PCollection, pipeline construction, core transforms, I/O handling, and includes practical code examples.
The article explains why loading all log data into a database is impractical, outlines three drawbacks—volatile requests, data bloat, and cost—and introduces the lightweight awk tool with concrete command examples to filter and analyze network logs efficiently without a database.
This article demonstrates how to use Python's pandas library to read CSV and XLS files, filter and merge data based on group assignments, compute derived columns, and export the results to a new Excel workbook, providing a complete automation workflow for Excel data processing.
This article compiles the 15 most popular open‑source Python frameworks—from full‑stack web solutions like Django and Flask to specialized tools for event I/O, OLAP, distributed computing, and continuous integration—providing concise descriptions to help developers choose the right library for their projects.
This article reviews recent progress in Apache Flink's AI ecosystem, explaining how Flink unifies batch and stream processing for machine‑learning pipelines, introduces the Flink ML Pipeline and Alink library, describes the AI Flow framework for end‑to‑end ML workflows, and presents a novel mini‑batch streaming iteration mechanism to support both offline and online learning scenarios.
This article explains why monitoring is essential for operations, outlines the four‑layer monitoring standard (infrastructure, liveliness, performance, business), breaks down functional modules and data flows, and showcases JD Cloud's practical design, alarm‑convergence project, and future AI‑driven observability directions.
This article details the design and implementation of ADMonitor, a high‑availability, low‑latency advertising monitoring platform built on OpenResty, covering its architecture, data collection, processing, archiving, and performance outcomes.
The article outlines how OpenWhisk’s serverless execution model supports diverse use cases—including microservices, web and mobile back‑ends, IoT pipelines, API services, data processing, and cognitive applications—highlighting its modularity, language flexibility, automatic scaling, and integration with cloud services.
This article introduces Java 8's Stream API, explains why functional streams improve code readability and performance, and provides detailed examples of common operations such as filter, map, flatMap, reduce, collect, Optional handling, parallel processing, and debugging techniques for efficient data processing.
This tutorial walks through loading order data into pandas and SQL, then demonstrates how to retrieve current dates, extract date components, convert between readable dates and Unix timestamps, transform between 10‑digit and 8‑digit date formats, and perform date arithmetic using pandas, MySQL, and Hive.
The article details the technical evolution of 58.com’s real-time computing platform—from Storm and Spark Streaming to a Flink‑based one‑stop solution called Wstream—covering use cases, architecture, stability measures, migration from Storm, operational diagnostics, and future development plans.
This article presents a comprehensive collection of Flink interview questions covering fundamental concepts, advanced topics, and source‑code details to help candidates prepare effectively for Flink‑related technical interviews.
This article introduces Apache Beam as a unified programming model for batch and streaming data processing, explains its architecture, core components, advantages, extensibility, and demonstrates practical usage with KafkaIO, BeamSQL, and AIoT scenarios across multiple runners.
This article examines the technical challenges of large‑scale data processing, compares the classic Lambda and Kappa architectures, introduces the unified stream‑batch Lambda+ design built on Tablestore and Blink, and outlines suitable scenarios and practical solutions for modern big‑data systems.
The paper presents a high‑performance real‑time data processing pipeline that collects, transports, preprocesses, and computes service logs and metrics using Alibaba Logtail, LogHub, and an enhanced Flink (Blink) engine, persisting root‑cause graphs in Lindorm, achieving sub‑3‑second latency for tens of millions of events per second and cutting diagnosis time to about five seconds.
This tutorial walks through creating a desktop weather application with Python 3, PyQt5, and the requests library, covering environment setup, city code data preparation, UI design with Qt Designer, API querying, JSON parsing, widget handling, shortcut keys, and final packaging with PyInstaller.
This article explains the concept of Spark Shuffle, details its two-phase write and read processes, describes the evolution from Hash‑based to Sort‑based and Tungsten‑based shuffles across Spark versions, and outlines the relevant source‑code components in Spark 2.1.
This guide explains what jq is, how to install it on various Linux distributions, and provides step‑by‑step examples for pretty‑printing JSON, extracting specific fields, handling arrays, and using built‑in functions like keys and has, all with clear command‑line snippets.
The article examines common pitfalls when using NumPy arrays and matrices for data manipulation in machine learning, highlighting chaotic data structures, inefficient filtering, confusing arithmetic syntax, and unintuitive code patterns compared to MATLAB/Octave, and concludes with a critique of Python’s ergonomics.
This article provides a comprehensive overview of big data fundamentals, including the 4V characteristics, the Hadoop 2.0 layered architecture, a comparison between Hadoop and Spark, classification of common big‑data tools, and the typical offline and real‑time data processing workflows.
At the 2018 Hangzhou Yunqi Conference, Alibaba's CTO Zhang Jianfeng outlined DAMO Academy’s expansive AI strategy, unveiling new research labs, a semiconductor venture, quantum computing initiatives, and a youth science award aimed at accelerating data, algorithm, and computing breakthroughs.
At the 2018 Hangzhou Cloud Expo, Alibaba’s CTO and Damo Academy director Zhang Jianfeng unveiled the institute’s global expansion, new semiconductor venture, AI research pillars, upcoming NPU chip, quantum computing initiatives, and the youth-focused Damo Academy Green Orange Award, highlighting a comprehensive strategy for data, algorithms, and computing power.
The article presents a comprehensive overview of 360's Big Data Center evolution, covering business background, platform‑as‑a‑service architecture, data asset management, user‑profile unification, platform milestones, technical architecture, performance optimizations, online query capabilities, future plans, and a Q&A session.
Meituan’s in‑store dining team demonstrates how R’s open‑source packages, powerful data manipulation, rich visualization libraries, and reproducible reporting can be engineered into scalable, parallelized workflows that turn secondary data processing into fast, interactive dashboards and analytics, proving R’s enterprise‑grade performance and adoption.
This article introduces Hadoop’s origins and core challenges, then presents thirteen essential open‑source tools spanning resource scheduling, real‑time query engines, and additional processing frameworks, detailing each project's purpose, key features, and repository locations to help practitioners choose the right component for big‑data workloads.
This article uses a human‑computer analogy and a playing‑card counting example to explain the fundamentals of distributed computing, why single machines cannot handle massive data, and how the MapReduce model’s four steps—split, transform, shuffle, and merge—solve big‑data problems.
The author, a Sun Yat‑sen University undergraduate and repeat weekly champion, shares practical tips on handling large datasets, effective feature engineering, and combining GBDT with a custom deepFFM model to achieve top scores in the Tencent advertising algorithm competition.
A comprehensive catalog of Python libraries covering network communication, web crawling frameworks, HTML/XML parsing, text manipulation, file format handling, natural language processing, browser automation, concurrency, cloud services, email processing, URL manipulation, multimedia extraction, WebSocket support, DNS resolution, computer vision, proxy servers, and other useful tools for developers.
This guide explains how DataMagic leverages Spark on YARN for fast, scalable offline analytics—covering Spark’s core role, four steps to master its terminology, configurations, parallelism, and code modification, plus practical deployment scripts, dynamic resource tuning, MongoDB export, job troubleshooting, and cluster upkeep for trillion‑record workloads.
The article presents Hermes, Meituan’s O2O performance management platform for travel, detailing its six‑module, four‑engine architecture—including data, incentive, rule, calculation, and scheduling engines—while highlighting technical innovations such as two‑level caching, work‑stealing producer‑consumer processing, Map‑Reduce‑style calculations, and future AI‑driven enhancements.
Learn how to construct a scalable data processing platform using the SMACK stack—Spark, Mesos, Akka, Cassandra, and Kafka—covering storage design, processing workflows, resource management, deployment options, and fault‑tolerant task execution for both batch and streaming workloads.
This article summarizes Wu Xiaoguang's talk at Gdevops 2017, detailing how Vipshop integrates data processing, analysis, and mining technologies—such as Flume, Kafka, Spark, and custom scheduling—to improve operational decision‑making, performance monitoring, root‑cause analysis, and predictive modeling across its e‑commerce platform.
This article examines four typical traps when using NumPy for matrix and vector operations—confusing array and matrix shapes, inefficient data filtering, ambiguous multiplication syntax, and cumbersome syntax—offering examples, explanations, and comparisons with MATLAB/Octave to help Python users write clearer, more reliable code.
This article explains Meitu's development of a distributed bitmap system that leverages the speed and storage efficiency of bitmap structures to handle massive user data, detailing its evolution, architectural choices, implementation practices, and lessons learned to inspire similar big‑data solutions.
The article traces Tencent Cloud Storage's evolution from basic data access in the early 2010s, through a data‑processing phase driven by video and image workloads, to the current data‑activation era focused on big‑data analytics and cost‑effective cloud migration, highlighting technical features and real‑world use cases.
This article explains what MapReduce is, when to use it, and how to implement a PHP word‑count and a gold‑price average calculation on an Apache Hadoop cluster, covering installation hints, mapper and reducer scripts, testing commands, and visualizing results with gnuplot.
This article introduces the evolution of big‑data processing from Google’s MapReduce concept to modern open‑source frameworks, defines big data and its 3V characteristics, outlines typical processing pipelines, and compares batch, stream, and hybrid systems such as Hadoop, Storm, Samza, Spark, and Flink.
This comprehensive guide lists essential Python libraries for network operations, asynchronous programming, web crawling frameworks, HTML/XML parsing, text handling, data conversion, slug creation, office document manipulation, PDF processing, markdown rendering, YAML handling, CSS utilities, feed parsing, SQL tools, HTTP clients, microformats, executable analysis, PSD handling, natural language processing, browser automation, headless tools, multiprocessing, queues, cloud execution, email handling, URL manipulation, web content extraction, video downloading, wiki archiving, WebSocket communication, DNS queries, computer vision, proxy services, and miscellaneous utilities.
The nju_newbiew team, winners of the weekly champion in Tencent Social Ads University Algorithm Competition, recount their data processing, offline validation, feature engineering, and model strategies, highlighting practical machine‑learning lessons while also providing competition announcements and contact information.
Facing ever‑growing complexity, Java developers can streamline their projects by mastering a curated list of essential data storage and processing tools—including MongoDB, Elasticsearch, Cassandra, Redis, Hazelcast, EHCache, Hadoop, Solr, Spark, and Memcached—each offering distinct strengths for modern big‑data applications.
This article introduces Hadoop as a widely used big‑data framework, explains its core components HDFS and MapReduce, describes the cluster node roles, presents typical command‑line usage and a sample MapReduce workflow, and offers guidance for further learning.
This tutorial quickly introduces the MapReduce model, explains its core principles and execution flow, and guides you through seven practical examples—from basic WordCount to custom serialization, partitioning, joins, and friend‑recommendation—while providing test data and an optional ready‑made Hadoop environment for hands‑on practice.
This article provides a comprehensive overview of Hadoop as the leading open‑source platform for big‑data processing, detailing its core components HDFS and MapReduce, the evolution to Hadoop 2.0/YARN, and the extensive ecosystem of tools and commercial solutions that enable scalable storage, analysis, and machine‑learning on massive data sets.
Apache Beam is an open‑source, unified programming model for distributed data processing that lets developers write pipelines once and run them on multiple execution engines such as Spark, Flink, or Dataflow, simplifying code reuse and easing migration between frameworks.
This article introduces Apache Beam, its Beam Model, and how the Beam SDK enables developers to write unified, flexible pipelines for both bounded batch jobs and unbounded streaming workloads, illustrating concepts with mobile‑gaming examples and detailed code snippets.
This article introduces Apache Spark as a fast, general‑purpose big‑data engine, explains its ecosystem, compares HDFS with MongoDB, and demonstrates how Spark can be combined with MongoDB through the Mongo‑Spark connector, including real‑world case studies and sample code.
Qiniu Cloud transformed its high‑traffic data processing platform by containerizing services with Docker, addressing challenges such as massive request volume, CPU‑intensive workloads, IO bottlenecks, and burst traffic through architecture evolution, queueing, rate limiting, auto‑scaling, and secure, isolated custom processing pipelines.
This tutorial guides readers through using Apache Spark and MongoDB to compute daily active user statistics, covering Spark fundamentals, a Spark‑vs‑Hadoop comparison, MongoDB use cases, environment setup, Scala code workflow, Maven compilation, and job submission on a YARN cluster.
This article provides a comprehensive overview of Hadoop and its surrounding ecosystem, detailing core components, storage principles, key algorithms, and a wide range of modern big‑data technologies such as Spark, Flink, Kafka, NoSQL databases, and cloud‑based processing platforms.
The author reflects on a programmer's focus on web, distributed systems, and data processing, arguing that deep, continuous investment in fundamentals—such as algorithms, networking, and OS concepts—drives lasting skill growth, better project outcomes, and a healthier professional mindset.
The article explains how Qiniu's data processing platform has evolved from a simple real‑time URL‑based model to a more complex architecture featuring separate caching, agent services, discover monitoring, and container‑based elastic scaling to handle massive unstructured data workloads.
Discover a comprehensive collection of Python libraries covering network requests, web crawling frameworks, HTML/XML parsing, text manipulation, file format handling, natural language processing, browser automation, asynchronous programming, and more, providing developers with essential tools for efficient web scraping and data processing tasks.
This article presents a detailed, categorized catalogue of more than fifty open‑source big‑data projects—including Hadoop‑related utilities, analytics platforms, databases, BI solutions, data‑mining packages, query engines, programming languages, search tools, and in‑memory technologies—highlighting their primary functions, supported operating systems, and official links.
The author recounts delivering a MapReduce talk, then uses a vivid pizza sauce preparation story to illustrate how mapping chops ingredients and reducing blends them, effectively explaining distributed data processing concepts to a non‑technical audience.
This article provides a comprehensive overview of the Hadoop family—detailing more than twenty open‑source projects, their core functions, and a structured learning roadmap to help developers master Hadoop, Hive, Pig, HBase, Zookeeper, Mahout, and related tools.
The article explains Apache Spark’s memory‑based distributed computing model, its advantages over Hadoop’s MapReduce, key features, fault tolerance, deployment modes, ecosystem components, and the scenarios where Spark is most effective for large‑scale data analytics.
This article examines Baidu’s home‑grown real‑time big‑data platforms Dstream and TM, detailing their architectures, performance metrics, key features, and practical use cases such as log ETL and real‑time bidding, while highlighting how they meet millisecond‑level processing demands.
This article presents a curated list of fifteen popular Python open‑source frameworks—including web, API, OLAP, concurrency, and web‑crawling tools—explaining their core features and typical use cases for developers seeking robust, community‑backed solutions.
