In the big‑data era, high‑quality data are essential for reliable analytics, and this article explains data‑quality concepts, key dimensions, analysis methods for missing values, outliers, inconsistencies and duplicates, as well as practical management practices to ensure data assets become a competitive advantage.
Version 1.5.3 of the Python pandas library introduces optional pip extras, expanded Index support for NumPy dtypes, a new dtype_backend parameter, improved write copying, fixes for GroupBy cumulative operations overflow, several backward‑incompatible API changes, and multiple deprecations aimed at enhancing data analysis workflows.
SCQL, an open‑source Secure Collaborative Query Language built on multi‑party computation, enables SQL‑style privacy‑preserving data analysis for small‑to‑medium organizations by offering easy integration, fine‑grained column‑level access control, broad data‑source support, and optimized performance for collaborative queries.
This article outlines the background, risk scenarios, and technical design of Ziroom's risk control platform, describing how decision‑engine calculations, data‑driven feature extraction, and rule‑based strategies are used to identify and mitigate fraudulent account activities and reduce bad‑debt losses.
This presentation explains how to construct a metric system, identify bottlenecks, design targeted growth strategies, and apply A/B testing through detailed examples from Volcengine's DataFinder and DataTester platforms, culminating in practical case studies from Douyin Group and other enterprises.
This article shares practical experiences and insights on building an experiment platform, covering A/B testing applications, platform selection, broader business use cases, rapid system integration, and robust experiment analysis techniques such as pre‑AA checks, SRM, Welch’s T and Z tests.
This article introduces the entropy weight method for multi‑criteria evaluation, explains its theoretical basis, outlines the calculation steps, presents a supplier‑selection case study, and provides a complete Python implementation to compute indicator weights and overall scores.
This article explains how to conduct data analysis by accurately interpreting numbers and context, objectively assessing problems with standards, selecting appropriate analytical methods, and adopting structured thinking frameworks such as business‑process, department‑function, and strategy‑goal orientations to derive actionable insights.
The presentation outlines Xiaohongshu’s comprehensive community anti‑cheat strategy, defining cheating risks across industries, mapping the black‑gray ecosystem, and detailing a five‑module framework—risk perception, capability building, identification, mitigation, and evaluation—implemented via layered data architecture and multi‑stage detection to protect platform integrity.
This article demonstrates how to download the Google 1‑gram dataset, load the ~1.4 billion rows with Python and NumPy (using the PyTubes library), compute yearly word frequencies, visualize the rise of "Python" and compare it with Pascal and Perl, while discussing performance challenges and future improvements.
This article explains the Principal Component Analysis (PCA) method, outlines its step‑by‑step weighting algorithm, and demonstrates a complete Python implementation that converts supplier metrics into objective scores using scikit‑learn.
This article explains the concept of A/B testing, details the architecture and implementation of an AB testing platform—including experiment, metric, whitelist, and traffic services—provides practical guidelines for experiment design, data reporting, statistical evaluation, and outlines future enhancements for product optimization.
This article explains the Massey ranking method, shows how to build the required matrices and vectors from World Cup match data, implements the model in Python, and compares three scoring strategies to forecast whether Argentina or France will win the tournament.
This article presents a detailed case study of how a video app team used two rounds of A/B testing with different swipe‑up guide designs to diagnose retention issues, refine the user onboarding experience, and ultimately achieve significant improvements in new‑user retention and engagement metrics.
This article explains the theory behind simple linear regression, demonstrates how to generate random data and fit a model using R's lm() function, and interprets the statistical output including coefficients, significance tests, and goodness‑of‑fit measures.
This article explains the theory behind hypothesis testing, demonstrates one‑sample and two‑sample t‑tests using R’s t.test() function with agricultural yield examples, and shows how to interpret one‑sided versus two‑sided test results for decision making.
This article explains how various continuous and discrete probability distributions—such as normal, binomial, Poisson, and negative binomial—are used in real‑world contexts, introduces R’s naming conventions for distribution functions, and provides code examples for computing densities, CDFs, quantiles, and random samples.
This guide explains how to install, update, load, and use R packages—including CRAN and Bioconductor sources—by demonstrating key commands such as install.packages(), BiocManager::install(), library(), and help(), while also covering library paths and package management best practices.
This article explains the fundamentals of statistics with a model‑centric approach, covering data collection, exploratory data analysis, descriptive statistics, visualization, and the three‑step Bayesian modeling process—including hypothesis formulation, model fitting, and evaluation—while emphasizing simplicity and practical programming tools such as Python.
This article demonstrates how to perform multiple linear regression using sklearn's LinearRegression and the statsmodels library in Python, covering both formula‑based and array‑based approaches, complete with example data, code snippets, and model evaluation details.
This article walks through a Python‑pandas solution for marking rows in an Excel sheet where two string columns share common characters, presenting three concise code approaches and visual results to help readers quickly apply the technique to their own data.
This article explains the method of moments for estimating population parameters, outlines its step‑by‑step derivation, and demonstrates a Python implementation that estimates a basketball player's shooting odds from binary outcome data using.
This article introduces xlite, a Rust‑based SQLite extension that lets you import Excel files as virtual tables and run full SQL queries for data lookup and analysis, outlining a simple three‑step workflow and noting current read‑only limitations.
The article introduces the business background, architectural design, evolution challenges, and step‑by‑step methodology for building and operating a risk‑control experiment platform that supports online and offline A/B testing, data analysis, and strategy iteration in internet finance.
This article explains the four types of averages—arithmetic, geometric, weighted, and harmonic—along with median, mode, and various quantiles, showing how each is calculated and when to use them for reliable data analysis.
This article introduces major economic and financial data analysis tools—including Python with its scientific libraries, R, Stata, Matlab, EViews, SAS, and SPSS—detailing their origins, key features, typical applications, and where to find more information.
The article examines the Global Hunger Index’s methodology and recent ranking, highlighting India’s severe hunger classification, the index’s three‑step calculation process, and the broader significance of composite evaluation models for assessing nutrition, health, and development across nations.
This article introduces the GoPUP Python library, explains how to install it, demonstrates retrieving Weibo index data with code examples, shows how to visualize the results using Pandas, Jupyter and Matplotlib, and lists the wide range of public data APIs the library supports.
The 2022 Open Source Big Data Heat Report, jointly launched by the Open Atom Open Source Foundation, X‑Lab, and Alibaba Open Source Committee, analyzes GitHub and Jira data from 2015‑2022 to map heat values across big‑data project categories, publishes a list of 92 selected projects, and invites additional contributions before its official release at the 2022 Cloud Expo.
The article guides readers through practical Excel functions—such as REPT, SUMPRODUCT, FREQUENCY, and text extraction—and advanced chart types like grouped bar, butterfly, and dual‑pie charts, while offering visualization best practices that prioritize clarity, efficient data processing, and effective communication of insights.
This article demonstrates how to use Python libraries such as itchat, jieba, matplotlib, snownlp, and TencentYoutuyun to retrieve WeChat friend information and perform data analysis on gender distribution, avatar usage, signature sentiment, and geographic location, presenting results with charts and word clouds.
This article outlines the DevMind system—a comprehensive, data‑driven framework that turns R&D efficiency measurement into an online, low‑threshold, scalable practice, covering four best‑practice pillars, technical architectures, product modules, and real‑world impact within large organizations.
This article explains how to calculate the number of bins for a frequency distribution histogram using Sturges' formula, illustrating the process with a sample of 50 data points where the maximum value is 980 and the minimum is 500, and shows how to derive the bin width.
This article introduces the fundamentals of statistics, covering its purpose, types, data collection methods, data organization steps, graphical representation, measures of central tendency and dispersion, probability concepts, parameter estimation, hypothesis testing, and the distinction between correlation and regression analysis.
This article explains the differences between diagnostic, descriptive, predictive, and prescriptive analytics, showing when each approach is useful and how businesses can choose the right analytical solution to turn massive data streams into actionable insights.
Despite the rise of newer data tools, SQL continues to dominate the developer landscape in 2022, offering universal applicability, high demand, strong career prospects, and a rich history that makes it an essential skill for data analysts, developers, and high‑paying tech roles.
This article compares Elasticsearch and ClickHouse on write throughput, query speed, and cost, then provides a detailed deployment guide for Zookeeper, Kafka, Filebeat, and ClickHouse, including configuration snippets and solutions to common setup issues.
This article describes how the author collected a watermelon dataset, defined measurable features such as size, color, sugar content, seed count, and texture, and documented the process with photos, tables, and a brief discussion of data characteristics for future machine‑learning analysis.
This article provides a comprehensive, visual guide to essential Pandas operations—including sorting, column selection, grouping, multi‑aggregation, filtering, dropping, joining, merging, pivot tables, melting, stacking, unstacking, and index manipulation—illustrated with clear code examples for data analysis in Python.
Learn how Pearson, Spearman, and Kendall correlation coefficients measure linear and monotonic relationships between variables, explore their mathematical properties, interpret their value ranges, and see practical Python examples using Pandas to compute each coefficient on generated data.
Principal Component Analysis (PCA) reduces high‑dimensional data to a few uncorrelated components by maximizing variance, enabling noise reduction, visualization, and efficient modeling, with practical steps—including data standardization, covariance matrix computation, eigenvalue extraction, and component selection—illustrated through a clothing‑size measurement case study.
This article introduces ten Python packages that automate exploratory data analysis, explaining each library's capabilities, providing concise usage examples, and showing how they can generate comprehensive data summaries and visualizations with just a few lines of code.
This article outlines thirteen key statistical techniques—including descriptive and inferential methods, hypothesis testing, reliability analysis, contingency tables, regression, clustering, discriminant, factor, and time‑series analysis—explaining their purposes, assumptions, and typical applications for researchers and data analysts.
Learn how to use pandas' query() function to filter DataFrames with single or multiple conditions, combine logical operators, handle string queries, and perform inline arithmetic and built‑in functions, illustrated with a synthetic dataset and step‑by‑step code examples.
This article outlines a five‑step framework for designers to demonstrate that their solutions address genuine product and experience problems, using qualitative user research, quantitative data analysis, stakeholder alignment, KPI impact assessment, and iterative validation to build trust and effectiveness.
This article explains why Python is an ideal language for data analysis, covering its simplicity, extensive libraries, compatibility with big‑data platforms, and a step‑by‑step workflow that includes data acquisition, storage, preprocessing, modeling, and visualization, while also highlighting useful tools and resources.
This article provides a comprehensive overview of the big data ecosystem, detailing nine core technology categories—from data collection and storage to computation, analysis, scheduling, and underlying infrastructure—along with tool comparisons, selection guidelines to help readers quickly build a complete big data knowledge system.
This tutorial walks through building a simple Python web crawler for the QDaily website, covering target analysis, environment setup, SQLite database creation, data extraction with requests and BeautifulSoup, storing articles and comments, performing basic analysis, and visualizing results with a word cloud.
This article presents ten crucial intermediate‑to‑advanced SQL techniques—including CTEs, recursive CTEs, temporary functions, CASE‑WHEN pivots, EXCEPT vs NOT IN, self‑joins, ranking window functions, delta calculations, cumulative sums, and date‑time manipulation—to help data professionals ace interview questions and write cleaner, more powerful queries.
This article demonstrates how to perform multiple linear regression using scikit‑learn's LinearRegression and the statsmodels library, covering data preparation, model fitting, code examples, and interpretation of statistical results for a cement heat‑release case study.
This guide explains how to use Alibaba Cloud Function Compute asynchronous tasks for large‑scale data analysis, database autonomous services, Kafka‑based ETL pipelines, and high‑performance video transcoding, highlighting architecture migration, cost reduction, deployment steps, and observable serverless task capabilities.
This tutorial demonstrates multiple pandas techniques—including datetime mapping, MonthBegin/MonthEnd offsets, floor operations, period conversion, and asfreq—to accurately retrieve the first and last day of a month for any given date, while highlighting common pitfalls and performance considerations.
This article presents eight practical Python data‑analysis techniques—including one‑line list definitions, lambda expressions, map and filter functions, NumPy arange/linspace, Pandas axis handling, DataFrame concatenation/merging/joining, the apply method, and pivot tables—each illustrated with clear code examples and explanations.
This article presents Vivo Game Center's end‑to‑end intelligent gray‑release data system, detailing its experimental mindset, statistical methods, data models, and product solutions that ensure scientific version evaluation, project progress, and rapid issue closure through root‑cause analysis and full‑process automation.
This tutorial explains how data analysts can automate the creation, processing, and distribution of statistical reports using Python by reading various data sources, performing DataFrame calculations, generating visualizations, and sending the results via email, thereby saving time and reducing errors.
This tutorial shows how to use Python libraries such as itchat, jieba, matplotlib, snownlp, and Tencent Youtu to fetch WeChat friend data and then visualize gender distribution, detect face usage in avatars, generate word clouds and sentiment analysis for signatures, and export location information for mapping.
This article demonstrates how to calculate similarity between rows in a pandas DataFrame using both a Cartesian‑product approach and vectorized broadcasting, providing complete Python code, performance notes, and step‑by‑step explanations.
This article demonstrates how to deduplicate invoice numbers and compute monthly order counts in retail data using pandas, comparing three approaches—unique with explode, groupby with value_counts, and drop_duplicates—while highlighting performance differences and providing complete code examples.
This tutorial demonstrates how to generate random product usage data, compute per‑product usage‑duration percentages with pandas, calculate cumulative sums, and create a horizontal stacked bar chart using Matplotlib, offering multiple pandas techniques such as groupby‑value_counts, unstack, and crosstab for clear visual analysis.
This article provides a visual guide to using Pandas for data manipulation, covering sorting with sort_values, column selection, grouping with groupby and various aggregations, filtering rows and columns, dropping, joining, merging, pivot tables, melting, stacking, unstacking, and index operations, illustrated with code snippets and screenshots.
This article demonstrates how to use pandas' groupby and diff functions to calculate daily differences for numeric columns within each category, offering four implementation methods—including apply, direct diff, transform, and a custom function—complete with code examples and visual results.
This article outlines the complete procedure of Grey Relational Analysis, from preprocessing evaluation indicators and constructing the evaluation matrix to calculating relational coefficients, grey relational degrees, and ranking objects based on their relational scores, providing a clear framework for multi‑criteria decision making.
This article explains why designers must master data analysis, defines what “design data analysis” means, and walks through a three‑step framework—data splitting, tracking, and analysis—illustrated with practical e‑commerce and recruitment case studies to boost product metrics and retention.
This article walks through a Python data‑analysis task where a fan asks how many teachers come from the United States, demonstrating two solutions with pandas—using merge() and join()—including code examples, explanations, and how to adjust the output to meet changing requirements.
This article compiles thirteen concise pandas techniques—including missing‑value calculation, group‑by max row extraction, multi‑column aggregation, string filtering, memory optimization, and more—to help Python users perform data analysis more efficiently and effectively.
This article analyzes the performance challenges of Taobao Mini Programs, defines a multi‑dimensional experience metric, builds a standardized ops data pipeline, introduces the T2 first‑screen algorithm and a three‑stage performance model, and shares concrete optimization practices that reduced T2 from 2.7 s to 1.9 s while improving business metrics.
This comprehensive tutorial walks you through pandas fundamentals—including reading CSV/Excel files, creating Series and DataFrames, performing basic operations, cleaning data, indexing, grouping, concatenation, merging, and handling time series—using clear examples and code snippets.
This article presents eight practical pandas data‑selection techniques—including boolean indexing, loc/iloc, isin, str.contains, where/mask, query, filter, and any/all—illustrated with code examples and visual outputs to help Python users efficiently extract and analyze data.
This tutorial shows how to calculate upward and downward differences of numeric columns for each category in a pandas DataFrame by sorting, grouping with groupby, and applying diff through several concise methods, including a custom helper function.
This article demonstrates how to conduct various hypothesis tests—including Z‑test, one‑sample and two‑sample t‑tests, F‑test, as well as distribution fitting and outlier detection—using Python libraries such as Statsmodels, SciPy, and pandas on both the Iris dataset and a human temperature dataset.
HaLao’s precise marketing system combines Pu‑Learning lookalike models, graph‑embedding user similarity analysis, and TSA/EM optimization within a robust feature‑engineering and deployment framework, delivering over 20% ROI gains and 3‑10× user growth while addressing low‑ROI and targeting inefficiencies.
MDAP is a multi‑dimensional real‑time monitoring platform for mobile apps that gathers metrics, logs, and traces via lightweight SDKs, processes data through micro‑service back‑ends using Flink, Spark, and ClickHouse, applies intelligent analysis for smoothness scoring, memory‑snapshot optimization, stack de‑obfuscation, crash clustering, and URL templating, and aims to extend end‑to‑end observability and predictive issue detection.
By photographing and labeling every garment, the author uses data‑analysis metrics such as quantity and elimination rate to reveal seasonal imbalances, high discard rates for online and flashy items, and then recommends buying more denim, trying winter clothes in‑store, and regularly updating the wardrobe dataset.
This article demonstrates how to use Python, the PyTubes library, and NumPy to load, process, and visualize the massive Google Ngram 1‑gram dataset—over 1.4 billion records—showing performance considerations, data‑cleaning steps, and comparative language trends for Python, Pascal, and Perl.
This article examines how Taobao mini programs tackled stability and speed challenges by redefining user experience metrics, standardizing operational data, analyzing performance stages, and implementing best‑practice solutions such as engine reuse, data prefetching, and template snapshot rendering, resulting in measurable improvements.
Data analysis must be tightly linked to business needs, as illustrated by a Pareto analysis of a food specialty store that reveals a 55% user contribution to 80% sales, prompting strategic decisions on customer segmentation, value distribution, and growth tactics beyond traditional 20/80 expectations.
This article explains the three progressive stages—giving data, giving conclusions, and giving viewpoints—that data analysts should follow to turn raw metrics into actionable business recommendations, and highlights common pitfalls such as unclear objectives, vague advice, and unimplementable suggestions, offering practical guidance for developing a data‑thinking mindset.
The article outlines a comprehensive technical growth mindset for engineers, describing seven progressive thinking modes—problem, tool, product, data, risk, production‑relationship, and craftsmanship/evangelism—plus brand and team mindsets, emphasizing lifelong learning and systematic self‑development.
This article explores how enterprises can overcome common data‑analysis frustrations, explains why quantitative data is essential for effective management, outlines methods to boost data‑driven management efficiency through analysis and productization, and presents a detailed second‑hand car platform case study illustrating these concepts in practice.
This article demonstrates how to download Google’s massive N‑gram dataset, load the 1.4 billion 1‑gram records with Python and the PyTubes library, use NumPy to efficiently compute yearly word frequencies, and reproduce Google Ngram Viewer charts for Python and other programming languages.
The article explains Gartner’s augmented analytics paradigm—automating data preparation, insight discovery, and sharing via AI/ML—and details Alibaba Mama’s AutoInsight system, whose four‑layer architecture delivers automated anomaly detection, metric contribution analysis, personalized recommendations, and real‑time warnings, illustrated through advertising and mobile placement use cases, highlighting its potential to boost marketing efficiency and insight automation.
This tutorial walks you through a real‑world example of constructing e‑commerce user profiles using Python, pandas, and matplotlib, covering data loading, preprocessing, feature engineering, behavior tagging, activity analysis, and RFM segmentation with clear code snippets and visualizations.
The author describes how Baidu Mobile Assistant’s massive storage costs were slashed by analyzing billions of objects, identifying low‑value incremental patch files, and applying a three‑step “elephant‑in‑the‑fridge” cleanup that reduced petabytes to a few hundred terabytes while highlighting the need for ROI‑driven update policies and ongoing documentation.
This article explains how to retrieve specific values from a pandas DataFrame with a MultiIndex by resetting the index, using query, chaining .loc calls, tuple indexing, and swapping index levels, providing clear code examples and useful tips for efficient data extraction.
This article curates a wide range of high‑quality video tutorials, books, and online resources covering Python fundamentals, web crawling, data analysis, machine learning, Go language development, and front‑end technologies, offering practical recommendations and rating scores to help learners choose suitable study material.
This talk explains the background of Alipay client compatibility issues, describes a four‑step solution that runs detection scripts on the client, presents data‑driven findings about feature support, and outlines future directions for a business‑specific “Can I Use” platform.
This article explains the wide applicability of causal inference, outlines three major challenges—correlation vs. causation, confounding factors, and selection bias—and demonstrates a scientific three‑level approach using examples such as smoking and a real‑world deployment of an intelligent client‑management tool at Beike, including experimental designs, results, and lessons learned.
This case study details 58.com’s 11.0 product overhaul, covering six months of research, data analysis, live‑stream and short‑video integration, AI/AR/VR applications, redesign of the homepage layout, and the evolution of the LOCO Design System to improve user efficiency and service reach.
This article introduces survival analysis, explains how to model user churn by defining purchase and cancellation times as birth and death events, describes data formatting, presents descriptive Kaplan‑Meier results, and shows how Cox regression quantifies the impact of factors such as membership and activity on user survival.
This article compiles a categorized collection of Python packages for quantitative finance, covering scientific computation, pricing, technical indicators, backtesting, risk analysis, data sources, Excel integration, and visualization, with brief descriptions and reference links for each tool.
This article presents a thorough collection of frequently used pandas commands for importing data, preprocessing, cleaning, transforming, analyzing, and visualizing datasets, complete with code examples and explanations to help Python developers perform efficient data analysis tasks.
The article analyzes four fundamental limitations of SQL—lack of stepwise execution, incomplete set semantics, missing ordered operations, and no object reference mechanism—illustrates them with real queries, and introduces SPL as a more intuitive language that overcomes these issues.
The article presents a data‑driven framework for technical support to boost its autonomous handling ratio, covering knowledge‑base creation, diagnostic tool mastery, standardized issue templates, controlled feedback channels, self‑service infrastructure, detailed analytics, and disciplined retrospectives to reduce developer support time and improve efficiency.
This case study details how JD.com’s 11.11 mobile commerce event combined commercial‑driven visual concepts, interactive micro‑games, and front‑end performance optimizations to achieve over 200% growth in dwell time, 330% higher click‑through rates, and a 106% increase in order value.
This tutorial walks through a complete pandas‑based workflow for analyzing a concrete‑strength dataset, covering data loading, cleaning, exploratory visualizations, correlation analysis, and targeted sub‑group investigations to uncover factors influencing product strength and suggest improvement measures.
The article explains how marketers can move beyond last‑click attribution by using touchpoint path and combination models to quantify each channel’s role, visualize journeys, assess overlap and efficiency, and analyze frequency effects, enabling data‑driven channel‑mix optimization and budget allocation.
This guide outlines six Python advancement directions—full‑stack development, web scraping, data analysis & mining, AI & machine learning, DevOps automation, and testing—detailing essential skills, typical salary ranges, and how each role fits the modern job market.
This article explains how designers can shift from macro metrics like DAU to analyzing raw user interaction data, infer user psychology, reconstruct real usage scenarios, and identify optimization opportunities through concrete examples and step‑by‑step methods.
This article walks through scraping Bilibili documentary data with Python, processing it using pandas, and visualizing regional distributions, genre comparisons, yearly trends, episode lengths, popularity metrics, and comment timing to uncover insights about Chinese, UK, and US documentary patterns.
This article demonstrates how to scrape cat breed information from a dedicated website, store the data in Excel, and perform comprehensive analysis and visualizations—including relationship graphs, geographic distribution, size ratios, price extremes, and word clouds—using Python libraries such as requests, lxml, pandas, pyecharts, and stylecloud.
