PsiBot demonstrates that, with a 100,000‑hour human‑operation dataset captured via exoskeleton gloves and ego‑vision, a world‑model (W0) and reinforcement‑learning policy (R2) can bridge the gap to robot control, offering a scalable alternative to costly teleoperation pipelines.
The article argues that successful speech model training starts with understanding user scenarios, then selecting appropriate data, and finally choosing metrics, detailing six key questions, data sourcing strategies, evaluation criteria, and compliance considerations to avoid the misconception that sheer data volume guarantees performance.
The article analyzes how Kepler Robotics’ dual‑path, six‑degree‑of‑freedom force‑tactile data collection system overcomes the scaling bottleneck of embodied AI, enabling a VTLA model that integrates vision, language, action and tactile feedback to achieve near‑perfect industrial assembly performance.
Qianxun Intelligent secured ¥30 billion in funding within a month, leveraged a scaling‑law data engine and the Spirit v1.5 VLA model to achieve breakthrough robot performance, and demonstrated the commercial loop through deployments at JD.com retail and CATL battery lines.
HumDex combines wireless inertial motion capture, a learning‑based hand‑redirection network, and a two‑stage pre‑training/fine‑tuning pipeline to deliver portable, high‑precision teleoperation for humanoid robots, cutting data‑collection time by 26% and raising remote‑operation success to 91.7% while enabling zero‑shot generalization across unseen objects and scenes.
This survey systematically reviews how foundation models reshape embodied navigation, covering problem definition, taxonomy of tasks and robot forms, system architecture from perception to control, data sources and training strategies, edge deployment techniques, benchmark metrics, and future research directions.
In multi‑cloud environments, enterprises face high outbound traffic fees for unified observability, but by routing logs through a CDN and using the high‑performance LoongCollector agent, they can reduce cross‑cloud transfer costs by up to 70%, improve throughput, and simplify deployment.
In the era of massive AI model training, this article presents a step‑by‑step technical guide—covering the full data‑collection pipeline, three acquisition modes, IP‑type choices, bandwidth savings, path and mixed‑request optimizations, and business‑level cost controls—to reduce web‑scraping expenses by more than 60% while maintaining data quality.
This article outlines the evolution from traditional AI to embodied intelligence, explains the Vision‑Language‑Action (VLA) paradigm, highlights data‑collection bottlenecks, introduces the UnrealZoo simulation platform built on Unreal Engine, and showcases real‑world case studies and future challenges for embodied AI research.
Spider_XHS is an open‑source Xiaohongshu crawler that automates note, user, comment, and message extraction, offers watermark‑free media downloads, exports structured Excel/JSON data, integrates with the creator platform, and includes proxy and anti‑ban features, enabling marketers and researchers to cut weeks of manual work into hours.
This article explains how, amid the rapid rise of AI‑assisted programming, a scientific and actionable R&D efficiency metric framework was designed, detailing core indicators such as AI code adoption rate, data collection methods, platform architecture, and practical insights from a large‑scale implementation.
Mobile developers often struggle with unreproducible crashes and lag reported by users, spending days sifting through logs and isolated stack traces; this article explains how a cloud‑native iOS SDK links performance metrics, error logs, and user behavior through systematic data collection to dramatically speed up issue diagnosis.
LoongCollector, the core component of Alibaba Cloud's LoongSuite, delivers zero‑intrusion, multi‑tenant, high‑performance data collection and processing for AI services, integrating logs, metrics, traces, events, and profiles into a unified, programmable pipeline that scales elastically across heterogeneous GPU clusters.
This article examines Ctrip's Hetu visual event‑tracking system, comparing it with traditional code‑based tracking, detailing its architecture, SDK modes, configuration workflow, and best‑practice implementations that improve efficiency, reduce coupling, and enhance data reliability across web, app, and mini‑program platforms.
This article introduces ten practical Python-based web scraping methods—including requests, Selenium, Scrapy, Crawley, PySpider, aiohttp, asks, vibora, Pyppeteer, and Fiddler‑plus‑Node reverse engineering—explaining their use cases, advantages, and code examples for efficiently gathering e‑commerce and app data.
This article explains what constitutes high‑quality LLM training data, why large datasets are essential, outlines the step‑by‑step process for collecting, preprocessing, and fine‑tuning models, and highlights the best data sources—including web content, books, code repositories, and news—while noting available free datasets.
This comprehensive guide walks through every stage of building a large‑language‑model pretraining pipeline—from data sourcing, cleaning, and deduplication, to tokenizer design, model architecture choices, training framework selection, optimization tricks, and evaluation methods—providing actionable tips and pitfalls to avoid for both newcomers and seasoned practitioners.
This article compiles twenty‑three Python web‑scraping projects, each described with its purpose, key features, and a direct GitHub repository link, offering developers a ready‑made toolbox for data collection across platforms such as WeChat, DouBan, Zhihu, Bilibili, and more.
This article explains how Arkit, a Go‑based unified monitoring agent, collects and parses data from multiple sources, leverages Go plugins for flexible, high‑performance processing, provides custom plugin development guidelines, and discusses the performance benefits and limitations of the plugin system.
This article details Ctrip's approach to weak network detection, covering background, data collection, processing, dynamic weighting algorithms, result output, deployment effects, and future plans, and provides practical code examples and threshold settings for improving mobile network performance.
This guide explains why data analysis is essential, breaks it into three core stages—data collection, data mining, and data visualization—offers practical tool recommendations, and presents principles for efficient learning and skill development.
This comprehensive guide walks through every stage of LLM pretraining—from data sourcing, cleaning, and deduplication, to tokenizer design, model architecture choices, training framework selection, optimization tricks, and evaluation methods—offering actionable tips and pitfalls to avoid.
This article outlines the design, architecture, and practical applications of a quality inspection data collection system, covering data point structures, reporting mechanisms, compliance analysis, intelligent strategy iteration, and BI dashboards, illustrating how big‑data techniques enable digital transformation of inspection processes.
This article presents a detailed case study of Ctrip's large‑scale desktop IT management solution, describing the challenges of handling tens of thousands of office PCs, the full‑link architecture built with Rust, Tauri, SpringBoot and Django, automated health monitoring, fault detection, remediation workflows, security measures, performance optimizations, and the measurable operational improvements achieved.
This article explains why interface analysis is essential, how to collect request data from web, mini‑programs, and Flutter using hooks and overrides, and how to process, store, and visualize metrics with Redis, InfluxDB, MySQL, and Elasticsearch while managing concurrency and flow control.
This article introduces ten practical Python techniques—including requests, Selenium, Scrapy, Crawley, PySpider, aiohttp, asks, vibora, Pyppeteer, and Fiddler‑based reverse engineering—to efficiently collect e‑commerce and app data while addressing common challenges such as IP blocking, captchas, and authentication.
This article explores browser extension use cases—from product capabilities and data‑collection tools to daily utilities—provides detailed implementation ideas for each scenario, and walks through concrete code examples such as traffic recording, behavior analysis, and proxy plugins, helping developers build practical, production‑ready Chrome extensions.
This article details the necessity of establishing a no-buried-point scheme mindset, key advantages over traditional methods, mobile client no-buried-point collection implementation mechanisms, data processing, and the benefits of this approach.
This article outlines practical steps—understanding problem background, gathering quality data, selecting appropriate models, solving and analyzing them, and applying results—to ensure mathematical modeling moves beyond theory and effectively addresses real-world issues.
This article details the design and implementation of a road‑testing platform that improves navigation accuracy by integrating data collection, real‑time video streaming, WebSocket communication, and automated reporting, resulting in faster issue detection, reduced manual effort, and measurable efficiency gains across multiple cities.
This article explains the motivations behind iLogtail's architectural redesign, details the evolution from a single‑file C++ collector to a modular pipeline with Golang plugins, outlines the refactor goals and implementation practices, and reflects on the challenges and outcomes of the six‑month effort.
The visual event tracking solution replaces costly manual code instrumentation with a Data‑Trackid and relative Xpath system, a VSCode plugin for automatic ID generation, and an SDK that captures clicks and exposures, dynamically loads third‑party analytics, and provides validation, monitoring, and future decoupling for scalable, real‑time product analytics.
This article explains what frontend event tracking (埋点) is, why it is essential for product analytics, when and how to implement it, the different tracking models and reporting methods, as well as practical tips, iteration processes, and common pitfalls for developers and product teams.
This article provides a comprehensive guide to front‑end event tracking, covering its definition, motivations, scenarios, various tracking types, data models, reporting mechanisms, implementation steps, security considerations, and practical tips for ensuring accurate and non‑blocking data collection in web applications.
This article explains how to collect images from Zhihu's "beauty" topic using Python's Requests and lxml libraries, filter them with Baidu AI's AipFace face detection service, and store the qualified pictures locally, detailing the required environment, logic, and preparation steps.
This article explains what event tracking (埋点) is, its role in collecting user behavior data, the real‑time and offline data pipelines, the responsible teams, the end‑to‑end implementation process, common quality issues, detailed test cases, and the Track platform used for verification.
This article presents a comprehensive overview of Zhihu's event‑tracking system, covering its motivation, toolset, demand‑management platform, verification workflow, data‑collection pipeline, query service architecture, cloud‑native data service design, and practical Q&A on best practices and optimization strategies.
The article explains how cloud‑native observability systems gather massive telemetry from infrastructure, containers, middleware and services, compares direct push and file‑based collection approaches, and details head, tail and local sampling methods to optimize data completeness and performance.
The article explains that while observability is not new, cloud‑native systems have driven rapid development of observable platforms, detailing data collection architectures, direct push versus file‑based approaches, and various sampling techniques (head, tail, and local sampling) to balance completeness, real‑time reporting, and performance impact.
This article walks through building a Python web scraper that automatically logs in, fetches daily free reports from a registration‑required site, saves each as a PDF with appropriate filenames, and includes complete source code with detailed explanations for handling retries, cookies, and file management.
Vivo’s Points Task System evolved from a simple configuration‑driven task model into a scalable, multi‑source behavior incentive platform that uses an AviatorScript engine, unified SDK, and three isolated services—event collection, computation, and task handling—to deliver configurable tasks, real‑time rewards, and flexible user notifications while ensuring stability and extensibility.
This article presents a comprehensive overview of EMQ's Neuron industrial IoT data collection platform, detailing the lessons learned from version 1.x, the redesigned v2.0 architecture, core modules, plugin mechanisms, data‑tag management, eKuiper integration, and two real‑world case studies in oil‑field and smart‑factory environments.
This article explores the challenges of end‑to‑end tracing on mobile apps, explains why issues are hard to reproduce, and presents a four‑step solution using a unified OpenTelemetry standard, automated data linking, performance optimizations, and machine‑learning‑driven root‑cause analysis.
This article discusses how to improve the reliability of online A/B experiments by designing robust architecture, evaluating group balance with orthogonal testing, ensuring consistent front‑end/back‑end logging, automating statistical significance checks, reducing group imbalance, and interpreting results using causal trees.
This article explains the concept of data tracking (埋点), outlines its key application scenarios such as exposure, click, and page‑event tracking, describes the end‑to‑end workflow from requirement gathering to deployment and post‑analysis, and summarizes the practical “seven‑word” checklist for successful implementation.
This article examines the evolving security risks of cloud‑native architectures, explains why traditional perimeter defenses are insufficient, introduces zero‑trust principles for Kubernetes, outlines common K8s threat vectors, and presents a complete data‑collection and monitoring solution based on the open‑source iLogtail agent.
In this talk, Wu Yakun from Guance Cloud explains the shortcomings of traditional operations, introduces intelligent, data‑driven approaches for the cloud‑native era, and outlines how unified data collection, observability, and SLO‑based monitoring can dramatically improve fault detection and system reliability.
This article describes how the author collected a watermelon dataset, defined measurable features such as size, color, sugar content, seed count, and texture, and documented the process with photos, tables, and a brief discussion of data characteristics for future machine‑learning analysis.
This article introduces a data‑collection SDK with a real‑time visualization feature, explains the shortcomings of traditional packet‑capture and log‑based methods, describes the underlying architecture—including a new SDK entry, encrypted reporting, WebSocket communication and Elasticsearch storage—and provides step‑by‑step usage instructions for developers.
This article provides a comprehensive overview of the big data ecosystem, detailing nine essential technology categories—including data collection, storage, computation, analysis, resource management, retrieval, underlying infrastructure, and cluster installation—while comparing popular tools and illustrating their typical use‑cases with diagrams.
This article explores frontend data collection and monitoring systems, covering why they are needed, what they can achieve, existing implementation schemes, and future evolution directions including performance data reporting and interface data monitoring.
This article explains the main layers of a typical big data platform—data collection, storage and analysis, sharing, and application—detailing common tools such as Flume, DataX, Hive, Spark, SparkSQL, Impala, and Spark Streaming, and discusses task scheduling and monitoring in the ecosystem.
This article walks through the four essential stages of building a data project—data collection, modeling, analysis, and application—explaining key principles, common models such as 3NF, star/snowflake, cube, and wide tables, and comparing offline versus real‑time pipelines.
The Database Autonomy Service (DAS) is a platform that uses big‑data, machine‑learning, and expert knowledge to automatically collect, compress, and analyze MySQL metrics, providing self‑service fault detection, root‑cause diagnosis, and security management, thereby reducing manual effort, shortening MTTR, and supporting Meituan’s rapid database growth.
This tutorial walks you through creating a Python web scraper that logs into Weibo, extracts the real‑time hot search list and top comments using requests, lxml and pandas, saves the data to Excel, and schedules the script to run hourly with Windows Task Scheduler.
This article describes the end‑to‑end design, metric hierarchy, data collection methods, visualization dashboards, and alerting mechanisms used to build a robust monitoring system for a large‑scale big‑data platform, covering physical hosts, Hadoop components, business services, and data layers with tools such as Telegraf, Prometheus, and Grafana.
This article explains the concept of event tracking (埋点), its importance for capturing user behavior, outlines the four‑module architecture of a tracking system, compares code‑based, visual and full tracking methods, describes data models, storage, management, and presents a practical case study with analysis techniques.
This tutorial explains how to use Python 3 with Requests, lxml, and Baidu's AipFace SDK to crawl images from Zhihu's "beauty" topic, filter them by face detection and gender criteria, and store the qualified pictures locally.
This article introduces a Python-based project that demonstrates how to log into and scrape data from 18 major websites—including Facebook, Twitter, Zhihu, and Bilibili—using methods such as Selenium, direct HTTP requests, and cookie management, providing code examples and future improvement plans.
This tutorial walks you through using Python Selenium to automate QQ Music, scroll through the infinite‑scroll comment section, extract user avatars, names, timestamps and comment text, save the data to CSV, and finally visualize the comments with a word‑cloud image.
Gaode’s high‑precision map platform, described by GM Xiang Zhe, details a three‑stage production pipeline, multi‑layer map architecture, and tiered data‑collection strategy that together address city‑road challenges, ensure map freshness, advance positioning and perception algorithms, and support commercial Level‑4 autonomous‑driving deployments.
The platform implements comprehensive metadata governance by extracting, standardizing, and ingesting basic, trend, resource, lineage, and task metadata from offline and real‑time systems via a Kafka‑based SDK, enabling unified storage, monitoring, alerts, and future automation to improve data asset visibility and quality.
This article explains how to locate Weibo comment APIs, work around rate limits by using the mobile site, extract required parameters, and implement a Python script that handles cookies, pagination, emoji cleaning, deduplication, and scheduled execution to collect comments efficiently.
This article details Meituan Waimai's traffic data collection history, the design and implementation of its large‑scale data warehouse—including ODL, IDL, CDL, MDL, and DIM layers—along with attribution modeling, data governance, and practical applications for analytics and product development.
This article outlines how to design and implement a comprehensive mobile data collection and analysis system using the open‑source Countly platform, covering background requirements, solution selection, architecture, customizations for client, server and dashboard, SDK integration for Android and H5, and practical data mining insights.
Learn how to quickly create a questionnaire using SeaTable, automatically collect responses, and run a custom script to randomly select winners, providing a fast, low‑code solution for urgent business needs without a full development cycle.
This article chronicles the rapid evolution of JD’s 福礼 employee‑benefits platform, detailing its Vue‑based frontend architecture, custom build tools, NutUI component library adoption, data‑collection strategies, multi‑device integration, development‑efficiency hacks, and collaborative processes that together drove a 265% YoY active‑user growth.
This article explains the complete Prometheus data collection workflow, covering key concepts such as targets, samples, and meta labels, detailing the relabeling steps, configuration options, example use‑cases, and the final scrape and storage phases for effective monitoring.
The article explains how to build a unified user‑behavior data collection platform that standardizes event definitions, front‑end reporting, and back‑end storage using Kafka pipelines and Elasticsearch, enabling comprehensive analysis of user interactions across Android, iOS, and web clients.
The article introduces ZLog, a unified data‑service solution that offers end‑to‑end collection, reporting, processing, and analysis for mobile (Android/iOS) and PC applications, explains its layered architecture, SDK design, verification of data accuracy, and showcases real‑world applications that improve debugging, security, and user‑behavior insights.
Tencent’s rapid‑deployment project creates a volunteer‑hosted, low‑cost atmospheric monitoring network using LoRa, NB‑IoT and Wi‑Fi, with edge‑agnostic data collection, cloud‑based device normalization, and business processing modules that manage devices, analyze PM2.5 sensor data, and visualize real‑time air quality across five community sites.
This article describes how the frontend team designed and implemented a comprehensive user behavior data collection and analysis platform, covering its business value, overall architecture, SDK-based data gathering, event interception, processing pipelines, analytics dashboards, and practical insights for product and operations teams.
This article explains the design of a standardized, simple, and efficient front‑end A/B testing pipeline, covering experiment configuration, data models, platform architecture, runtime JSSDK, traffic‑splitting strategies, and data back‑flow to enable reliable, data‑driven product decisions.
This article outlines a comprehensive, standardized front‑end AB testing architecture that separates experiment configuration and data chains, introduces a JSSDK with Core and Coupler packages, and explains traffic‑splitting models, data back‑flow, and extensibility across multiple front‑end DSLs.
Xianyu created a highly extensible user‑behavior collection framework that standardizes data into a common ODPS schema, uses JavaScript Proxy to intercept navigation and API calls, maps business metrics via JSON, aggregates reports to cut dataset‑creation effort from days to minutes while avoiding heavy full‑tracking overhead.
The Wuhan New Coronavirus Prevention Information Collection Platform aggregates hospital, hotel, factory, logistics, donation, and treatment data through manual reporting and automated processing, enabling real‑time information sharing and efficient allocation of social resources during the pandemic.
The guide explains how comprehensive data collection in big‑data platforms relies on a standardized event model, passive and code‑based embedding, multi‑platform SDKs, a log‑middleware layer, precise location tracking, and an embedding management platform that supports workflow, testing, quality monitoring, and scalable infrastructure for future enhancements.
The article outlines how to automate the end‑to‑end mobile app packaging workflow—from code submission and continuous integration to data collection, automated testing, and release management—highlighting the benefits of reducing manual effort, improving reliability, and enabling comprehensive historical package tracking.
The article outlines systematic approaches and practical tips for conducting industry research, covering the overall research framework, step‑by‑step information‑gathering methods, categorisation of data sources, efficiency‑boosting tactics, and best practices for deep interviews to derive actionable business insights.
The article explains how data serves as a critical asset for recommendation systems, outlining the necessary steps from understanding business problems and data dimensions to collection, cleaning, integration, and analysis, while distinguishing explicit and implicit user feedback and emphasizing data quality, timeliness, and relevance.
This article explains why front‑end monitoring is essential for user experience, outlines the key data types to collect, and provides practical AOP‑based implementations for route changes, JavaScript errors, performance metrics, resource failures, API calls, and reliable log reporting.
This article describes how the author built a Python web crawler to automatically generate numeric URLs, fetch TV drama pages from the 天天美剧 site, extract ed2k download links using regular expressions, and save them into organized text files, streamlining the download process with Thunder.
The article examines three typical data collection problems faced by startups—unclear collection methods, chaotic tracking points, and poor collaboration between data and engineering teams—and offers practical strategies such as adopting full‑event models, appointing data architects, and securing top‑down support to achieve reliable, comprehensive analytics.
This article compares probe‑based agents and wire‑data collection for application performance monitoring, detailing their architectures, advantages, drawbacks, and cost implications, and concludes which approach best supports modern, real‑time operational intelligence.
Meitu's data platform, serving dozens of apps and handling billions of daily events, combines custom log collection, multi‑layer storage, offline and real‑time processing, and open data services to support personalization, anti‑fraud, analytics, and business growth.
The article outlines comprehensive automated performance testing approaches for Android and iOS applications, covering challenges of data accuracy, reliability and volume, and describing configurable UI automation, remote device management, data collection, and reporting mechanisms to enable scalable, low‑effort mobile testing.
This article introduces vehicle networking concepts, explains Huawei's OceanConnect solution, and details how data open and data collection are implemented using platforms like DAP, Kafka, and Hadoop to provide reliable, real‑time vehicle information for various applications.
This article details the architecture and practical techniques behind WeChat's large‑scale monitoring system, covering lightweight data collection, classification of real‑time, non‑real‑time and user‑specific metrics, anomaly detection algorithms, automated configuration, and high‑performance storage solutions for billions of events per minute.
At Meitu’s seventh Internet Technology Salon in Xiamen, over a hundred experts discussed recommendation algorithms and big‑data solutions, with talks on the Arachnia log‑collection system, the Naix distributed bitmap service, Meitu’s personalized recommendation pipeline challenges, and novel data‑missing‑theory models for improved performance.
The article details WMDA, a no‑code and manual‑code data collection platform for PC, mobile and app that supports real‑time and offline user behavior analysis, describing its functional model, behavior taxonomy, five‑layer architecture, tracking techniques, circle‑selection, data services, streaming and batch processing pipelines, and related technologies such as Storm, Spark, Druid and Roaring Bitmap.
The article explains how to move beyond macro app metrics by collecting offline and real‑time user data, storing it in HDFS, processing it with Spark, visualizing behavior paths as state‑machine trees, and performing branch‑funnel analysis to uncover conversion bottlenecks and improve product quality.
This tutorial walks through using Python and BeautifulSoup to crawl user profiles, fan lists, and song information from the Quanmin K‑Song app, clean and store the data in MongoDB, handle pagination, and prepare the dataset for further analysis.
This article details Qunar's hardware automation operations platform, covering the hardware scope, pain points of manual processes, a five‑stage lifecycle, automated testing, data collection, fault handling, and the underlying Mesos‑Marathon‑Docker infrastructure that together improve efficiency, reliability, and cost control.
This article explains how to design and implement a comprehensive operations monitoring platform by integrating Ganglia for data collection and Centreon for alerting, detailing a six‑layer architecture, data flow, seamless integration, and practical Q&A for real‑world deployment.
This article details a step‑by‑step process for crawling all popular Ximalaya channels, extracting each audio's metadata, and storing roughly 700,000 records in MongoDB, while also showing how to speed up the crawl with asynchronous requests.
This article details the design, technology selection, architecture, encryption, compression, and performance evaluation of Ctrip's real-time user data collection system, which leverages Java, Netty, Kafka, and Avro to achieve high throughput, low latency, and robust fault tolerance for mobile and web applications.
By defining data collection (event tracking) requirements in a structured JSON format and storing them in Git with a web interface that abstracts version control, teams can standardize identifiers, validate data formats automatically, track changes via commit logs, and streamline collaboration between product managers, developers, and testers.
This article explains what user feedback is, why it should be analyzed, and provides a step‑by‑step methodology—including channel setup, data collection, coding, categorization, and statistical analysis—along with key performance indicators for monitoring feedback handling in product management.
