This article analyzes two financial risk control lakehouse architectures: DiDi uses StarRocks to unify real-time and batch feature serving with primary-key tables and bucket pruning, while Ant Group replaces HBase and Elasticsearch with Apache Paimon's first-row, partial-update, and sort-compaction mechanisms for long-window deduplication, fast reconciliation, and historical lookback.
Ant Group's OmniTable system, awarded VLDB 2026 Best Industrial Paper, uses a unified logical wide table to manage 35PB and 305B records for LLM training, reducing end-to-end data preparation from 14 days to 2.5 days via logical-physical separation, feature dependency DAGs, record-level fault tolerance, and operator fusion.
Two senior undergraduates won the Scaling Law Original Breakthrough Award and 10th place in the academic track at the 2026 Tencent Advertising Algorithm Competition by running nearly 1000 experiments, discovering that increasing model width alone reduces AUC unless accompanied by new informative tokens, leading to their CoSFormer architecture that scales capacity and information jointly.
The article explains how to build an AI‑driven predictive‑maintenance pipeline that collects multi‑dimensional metrics, engineers features, trains a two‑stage anomaly‑detection and failure‑probability model, and automates alerting and remediation to warn of impending system crashes before they happen.
The 2026 China University Computer Competition's Big Data Challenge revealed the monthly star award winners, each receiving a prize, and the top teams detailed their feature engineering, model selection, training validation, and portfolio combination tactics that led to their success.
The 2026 China University Computer Competition Big Data Challenge announced its Monthly Star winners, and the top teams detailed their data preparation, StockTransformer and LightGBM modeling pipelines, feature engineering, validation strategies, ensemble techniques, and key lessons learned from the competition.
Big Data CompetitionCross-Stock AttentionEnsemble Modeling
The 2026 China University Big Data Challenge announced its Monthly Star winners, each receiving a prize, and the top three teams detailed their data processing, feature engineering, model design, training strategies, and post‑processing techniques for cross‑sectional stock ranking.
Big Data CompetitionQuantitative FinanceTime Series
This article details the end‑to‑end engineering of a recommendation system's content asset pipeline—from content ingestion and storage in a Content Pool, through basic, behavior, and quality feature generation, to feature versioning, lifecycle management, and the engineering challenges of recognition accuracy, cold‑start, hotspot handling, deduplication, and multimodal understanding.
The article explains how prediction, reasoning, and inference differ, why a pure prediction model leaves the decision chain broken, and how a dynamic ontology‑driven feature framework—temporal, functional, and relational features—creates explainable, verifiable, and iterative decision loops.
Over the past four decades AI has undergone five fundamental shifts—from manual feature engineering, through deep neural networks and prompt engineering, to autonomous agents and finally the Harness era—each redefining core technology, scarce talent, and value creation, with the 2026 Harness era emphasizing system‑level scalability over model size.
The project consolidates fragmented user data across departments into a unified OneID system, builds a comprehensive tagging and feature platform, and leverages real‑time and offline signals for algorithmic push, custom models, and crowd selection, enabling precise, data‑driven growth initiatives.
The article outlines ten practical data‑cleaning methods—covering missing‑value imputation, duplicate handling, outlier detection, normalization, discretization, text cleaning, type conversion, multi‑source alignment, feature engineering, and sensitive‑data masking—explaining why each step matters for reliable AI model training.
This article walks through the entire feature‑engineering pipeline—data cleaning, missing‑value imputation, encoding, outlier handling, scaling, feature construction, and selection—using Pandas and Scikit‑learn, and shows how to wrap the steps into a reproducible Scikit‑learn Pipeline.
The 2026 China University Computer Competition – Big Data Challenge reveals the Monthly Star award winners, each receiving 800 RMB, and presents detailed experience reports from the top teams covering feature engineering, model selection, training validation, and ensemble strategies for stock prediction.
This article walks through a full feature‑engineering pipeline using Pandas and Scikit‑Learn, covering data inspection, missing‑value imputation, categorical encoding, outlier handling, scaling, feature construction, selection, and a final Pipeline that prepares clean, predictive features for a logistic‑regression model.
The article explains why algorithm engineers resent product managers who treat models as black‑boxes, make vague data‑blind demands, and ignore experimental cycles, and it offers three concrete practices—feature‑focused communication, metric quantification, and respecting experiment timelines—to become a trusted teammate.
This article experimentally evaluates a privacy‑preserving LLM security pipeline that discards raw dialogue after extracting 28 telemetry features, showing that using only 11 text‑independent signals retains about 98.5% of detection performance while reducing false‑positive rates.
This article presents a detailed case study of how a large‑scale API security team built an automated, self‑learning classification system that tags tens of thousands of APIs with business labels, improves model accuracy by five points, and maintains high precision through a confidence‑driven feedback loop.
This article details a real‑world case study of how a large logistics platform created an automated, self‑evolving API asset‑classification pipeline—covering data collection, feature engineering, model training with CatBoost, confidence‑based label feedback, and lessons learned—to improve API security monitoring and reduce manual labeling effort.
The article explains why good features matter more than fancy algorithms and walks through practical techniques—scaling, log transforms, binning, interaction, various encoding schemes, datetime extraction, text statistics, geospatial distances, aggregation, feature selection, and automated feature generation—illustrated with concrete pandas and scikit‑learn code examples.
This article explains how open‑source test prediction analytics can transform limited testing resources into smarter, data‑driven test selection, detailing the end‑to‑end stack, real‑world case study results, common pitfalls, and steps toward autonomous testing.
Algorithm engineers often resent product managers who treat models as black‑box wish‑lists, ignore data feasibility, and issue vague demands, leading to broken trust, wasted effort, and stalled projects, but adopting feature‑focused communication, concrete metrics, and realistic experiment timelines can turn a problematic partnership into a productive collaboration.
The Tsinghua University Big Data Capability Project applied GRU‑based deep learning, pulse‑event encoding, and advanced feature engineering to transform discrete refinery energy data into continuous sequences, achieving prediction accuracies of 84.2%, 82.7% and 81.6% for fuel gas, medium‑pressure and low‑pressure steam respectively.
Over ten years Pinterest transformed a fragmented machine‑learning stack into a unified AI platform, iterating through stages from early ad‑hoc pipelines to scalable GPU‑accelerated services, while learning that timing, organization alignment, and efficiency are crucial for lasting impact.
Facing PB‑scale user behavior data and millions of feature dimensions, the platform transformed its search, advertising, and recommendation pipelines by adopting a distributed, configurable‑service architecture that delivers high‑throughput streaming, elastic storage, rapid feature iteration, and robust fault‑tolerance for AI‑driven personalization.
This article shares a three‑year journey of building a data‑virtualization‑based, multi‑environment feature management framework for real‑time risk decision platforms, detailing challenges like heterogeneous storage, cold‑start, and operational stability, and presenting a unified architecture that decouples physical storage from business logic.
This article walks through the complete machine‑learning workflow for predicting user actions—covering core concepts, data collection, preprocessing, feature engineering, model training, evaluation, hyper‑parameter tuning, deployment, and future directions—using Python and popular AI libraries.
The author shares a detailed account of their experience in the 2025 China University Big Data Competition, describing the team’s top national ranking, the shift from absolute stock price prediction to robust ranking learning, extensive feature engineering, and reflections on balancing technical ambition with real‑world constraints.
Our team "Stay Overnight" from Chongqing University of Posts and Telecommunications placed second nationally in the 2025 China University Computer Competition Big Data Challenge, navigating volatile financial data, shifting from time‑series to supervised learning, and emphasizing feature engineering to boost model performance.
The article shares a top‑5 team's experience in the 2025 China University Big Data Challenge, detailing their roster, competition rules, four key technical insights on data pitfalls, model alignment, generalization, and leveraging SOTA models, plus reflections on the event's excellent support and collaborative atmosphere.
Learn how to transform raw data into model-ready features through comprehensive preprocessing, variable derivation, dimensionality reduction, and the application of IV, PSI, and WOE metrics, with practical guidelines, thresholds, and visual examples to enhance risk model performance.
A trio of freshly graduated undergraduates formed a team just two days before the deadline, overcame early setbacks by reordering feature engineering, leveraged score‑driven decisions, and ultimately surged to fourth place in Tencent's highly competitive algorithm contest, illustrating resilience and practical AI teamwork.
This tutorial walks developers through the complete workflow of building a house‑price regression model—from problem definition, data collection and preprocessing, feature engineering, and model selection, to training, hyper‑parameter tuning, evaluation, optimization, deployment as a Flask service, and ongoing monitoring—using Python, pandas, scikit‑learn, and visualisation libraries.
This article explains JD's complex recommendation system data pipeline—from indexing, sampling, and feature engineering to explainability and real‑time metrics—highlighting challenges such as data consistency, latency, and the use of Flink for massive, low‑latency processing.
The article details how edge intelligence is applied to the Vivo official app to improve product recommendation on the smart‑hardware floor by abstracting the problem, designing feature engineering pipelines, training TensorFlow models, converting them to TFLite, and deploying inference on mobile devices, while also covering monitoring and performance considerations.
This tutorial explains how to construct state feature vectors for reinforcement‑learning value‑function approximation, covering linear, polynomial, Fourier, and radial‑basis representations, as well as state aggregation techniques such as coarse coding and tile coding, and discusses non‑parametric approaches like kernel methods.
This article details how Vivo’s official app implements edge intelligence to dynamically rank and recommend hardware products on its homepage, covering problem abstraction, data collection, feature engineering, model design, TensorFlow‑Lite conversion, on‑device inference, and monitoring for a personalized user experience.
The article examines Databricks' acquisition of the incremental computation engine Fennel, detailing how its unified batch‑stream processing, incremental updates, Python‑native development, and built‑in data governance can eliminate data silos, cut costs by up to 90 % and accelerate real‑time feature engineering for AI models, while also discussing industry impact and future roadmap.
This article presents a detailed collection of built‑in pandas aggregation methods and numerous custom Python functions for time‑series feature engineering, offering beginners practical tools to enhance data preprocessing and model performance in machine‑learning projects.
Large language models can assist and enhance each stage of traditional machine learning—including sample generation, data cleaning, feature engineering, model selection, hyper‑parameter tuning, and workflow automation—by generating synthetic data, refining features, selecting models, and orchestrating pipelines, though challenges such as bias, privacy, and noise remain.
This article explains why feature engineering is crucial for NLP tasks, introduces n‑gram enhancements, provides Python implementations for generating bi‑gram and higher‑order features, demonstrates dynamic padding for text length standardization, and offers practical deployment tips such as feature dimension control and monitoring.
This comprehensive guide walks readers through the evolution of machine learning, starting with basic linear models and feature engineering, progressing through logistic regression, decision trees, and deep learning architectures like MLPs, CNNs, RNNs, and transformers, and demonstrates practical implementations with code examples and evaluation metrics.
Deep LearningEvaluation Metricsfeature engineering
This comprehensive guide explains logistic regression fundamentals, the role of the Sigmoid function, loss and optimization methods, step‑by‑step Python implementation with data preparation, model training, evaluation, hyper‑parameter tuning, handling over‑ and under‑fitting, multi‑class extensions, and diverse application scenarios across medicine, finance, e‑commerce, and text analysis.
Chronon is an open‑source framework that centralizes feature definitions to guarantee training‑inference consistency, eliminates complex ETL pipelines, and supports real‑time and batch processing across diverse data sources, cutting feature‑development cycles from months to under a week, as demonstrated by Airbnb’s 40,000‑feature deployment.
The article outlines Bilibili’s revamped recommendation data‑flow architecture that eliminates timing and calculation inconsistencies by snapshotting online features, unifying feature computation in a single C++ library accessed via JNI, and orchestrating label‑join and sample extraction through near‑line Kafka/Flink pipelines, with further performance gains and Iceberg‑based future extensions.
Tencent's advertising team replaced a traditional HDFS‑Hive warehouse with an Apache Iceberg‑based data lake, adding primary‑key tables, multi‑stream merging, adaptive compaction, and Spark SPJ optimizations to achieve minute‑level feature update latency, 10× back‑fill speed, and up to 60% storage savings.
This tutorial walks through consolidating the first ten days of learning by covering data preprocessing, feature engineering, model training with linear regression, decision tree, and random forest, model evaluation using cross‑validation, and finally saving and loading the best model, all illustrated with complete Python code examples.
In a panel discussion, experts explain how data‑lake‑warehouse integration, columnar formats like Apache Iceberg, and emerging variant types enable efficient feature engineering, support large‑language‑model workloads, and provide flexible vector storage, thereby driving the evolution of AI from traditional ML to the GenAI era.
In a panel discussion, experts from Kuaishou, Ping An, and Datastrato explain how data lake architectures, columnar storage formats like Apache Iceberg, and vector‑enabled lake formats are enhancing feature management, supporting generative AI workloads, and accelerating machine‑learning pipelines.
This article provides a comprehensive guide to data preprocessing using Pandas, covering essential steps like data cleaning, feature engineering, and data transformation for machine learning projects.
This article introduces a cloud‑based feature production platform that simplifies feature engineering for recommendation, risk control and machine learning, explains its component‑based modeling framework, and answers common questions about deployment, performance, and customization, highlighting cross‑platform compatibility and optimization techniques.
This article explains how to extract a variety of date‑time based features—including date, time, lag, rolling, expanding, and domain‑specific attributes—from a time‑series dataset using pandas, and discusses proper validation strategies for building reliable forecasting models.
This article details an open‑source Binance trading bot that uses machine‑learning models and extensive feature‑engineering pipelines to generate and broadcast automated cryptocurrency buy‑sell signals via a Telegram channel, covering data acquisition, offline training, hyper‑parameter tuning, and live service deployment.
The article outlines how internet and financial risk control are categorized into anti‑cheat, anti‑fraud, and content security, describes full‑scenario cheating types, and presents a three‑step joint defense framework using perception, identification, and mitigation with feature‑based analysis.
This article explains the concept of vector components, how they are represented as column or row vectors, zero and unit vectors, basic operations, and their crucial role in machine‑learning tasks such as feature representation, distance measurement, PCA, and gradient descent.
This article demonstrates how to perform data mining with scikit-learn by detailing the full workflow—from data acquisition and feature engineering, through parallel and pipeline processing, to automated hyper‑parameter tuning and model persistence—using the Iris dataset as an example.
This article explains how A/B testing and model grayscale are applied in credit risk control, discusses the specific requirements for effective testing, compares upstream and risk‑system traffic splitting methods, and proposes an integrated all‑in‑one solution to simplify feature engineering, model evaluation, and deployment.
This article explores the concept of intelligent risk control in banking, detailing its AI‑driven architecture, current challenges such as external data costs and model‑deployment friction, and proposes an integrated operational framework that leverages big data, knowledge graphs, and MLOps to enhance risk detection and decision‑making.
Feature engineering, the process of creating input variables for machine learning models, is crucial for banking risk control; this article explains the concepts of features, variables, and metrics, outlines challenges in real‑time feature pipelines, and proposes a practical architecture and best practices for building an efficient, low‑code feature platform.
This article provides a comprehensive guide on building user portrait tags—from basic attribute tags to business and strategy tags—detailing data collection methods, feature engineering techniques such as cleaning, time decay, and smoothing, and evaluation metrics for cohesion and stability, aimed at data product managers and analysts.
The article explores the concept, challenges, and future directions of intelligent banking risk control, emphasizing data integration, AI-driven modeling, feature engineering, MLOps, knowledge graphs, and large‑model applications to create a unified, automated risk management platform.
Opal is iQIYI's end‑to‑end machine‑learning platform that integrates feature production, sample construction, model training, and deployment with big‑data services, addressing duplicated effort, weak data processing, and fragmented pipelines to boost efficiency across recommendation, advertising, and risk‑control scenarios.
AI operationsMachine Learning Platformbig-data-integration
This article presents a detailed compilation of pandas built‑in aggregation methods and a wide range of custom Python functions for time‑series feature engineering, providing ready‑to‑use code snippets that cover statistical descriptors, drawdown metrics, peak detection, and more for data science practitioners.
This article details the challenges of maintaining separate real‑time and batch risk‑control features, evaluates Lambda and Kappa architectures, explores storage‑unified and compute‑unified alternatives, compares Hologres, StarRocks and ClickHouse, and presents a validated StarRocks‑based solution that dramatically reduces feature delivery latency and improves accuracy.
This article analyzes the challenges of building real‑time and batch risk‑control features, compares Lambda and Kappa architectures, evaluates storage‑unified and compute‑unified solutions, and details how StarRocks was selected, validated, and deployed to achieve high‑performance, low‑latency feature serving in a financial context.
This article systematically explains the fundamentals of user profile tags, covering basic attribute tags, business and strategy-oriented tags, detailed feature processing methods such as anomaly handling, time decay, and smoothing, and provides evaluation metrics for cohesion and stability, supplemented by a practical Q&A session.
Feature engineering transforms existing data into new predictive variables through manual analysis or automated pipelines, encompassing single‑variable encoding, pairwise arithmetic, group‑statistics, multi‑variable combinations, time‑series and text derivations, with tools like Deep Feature Synthesis and beam‑search to generate and select useful features.
This article explains how applying machine learning to database operations—specifically AIOps for MySQL—can automate index recommendation by parsing SQL, extracting semantic and statistical features, generating candidate index combinations, and training an XGBoost model to predict optimal indexes, reducing reliance on manual DBA work.
This article explains how to build user portrait tags—from basic attribute tags to business and strategy tags—covers methods for data collection, anomaly handling, time decay, smoothing, and evaluates tag quality using cohesion, stability, and AUC-related metrics to support data‑driven product decisions.
This article explains how to construct risk control feature portraits across four key scenarios—marketing, pre‑loan, in‑loan, and post‑loan—by selecting appropriate data dimensions, describing usable customer, behavior, and ID‑linked data, and illustrating each portrait with visual examples to guide accurate risk assessment.
Baidu’s recommendation ranking system handles billions of daily impressions and millions of users by combining discrete and cross features, bias mitigation, and long‑short sequence modeling within a multi‑stage funnel and hierarchical architecture, while planning to integrate large language models for generative, interpretable, and decision‑oriented recommendations.
This article presents Baidu's comprehensive approach to feed recommendation ranking, covering business and data background, feature engineering principles, core algorithmic strategies, system architecture design, and upcoming plans to integrate large language models for more intelligent and fair recommendations.
This article explains bot fundamentals, classifies bot types, analyzes the rising threat of malicious bot traffic, compares major vendor solutions, and outlines a four‑layer architecture with data, feature, model, and policy layers for robust bot management in modern web services.
This article explores how large language models can be incorporated into recommender systems, discussing background challenges, specific integration points across the recommendation pipeline, practical implementation methods, experimental results, and future research directions, while highlighting industrial considerations and potential improvements.
This article presents practical feature‑engineering methods for ten common business domains, explaining the purpose of each feature, the extraction technique, and providing ready‑to‑run Python code snippets to help build more accurate predictive models.
Business Analyticsdata preprocessingfeature engineering
This article surveys how large language models can be incorporated into recommender systems, discussing their strengths and limitations, outlining where and how they can be applied across the recommendation pipeline, presenting recent research examples, and highlighting challenges and future directions for industrial deployment.
This article presents a complete workflow for predicting whether users will purchase a membership using logistic regression, covering data collection, feature selection, handling imbalanced samples, model training, hyper‑parameter tuning, threshold optimization, evaluation metrics such as accuracy, precision, recall, AUC, lift, and finally deployment on a big‑data platform with PySpark.
This guide walks through ten core data preprocessing methods—including handling missing values, type conversion, standardization, encoding, smoothing, outlier treatment, text cleaning, word‑frequency counting, sentiment analysis, and topic modeling—each illustrated with concise Python code examples.
This article details how AutoHome's homepage pop‑up leverages precise targeting, extensive feature engineering, and multi‑stage DeepFM‑based models with attention and LHUC modules to accurately identify car‑buying users, improve vehicle‑series recommendations, and achieve a 355% conversion rate increase.
The Haro AI platform automates customer‑service ticket routing by applying a four‑step pipeline—feature processing, model training, evaluation, and deployment—using BERT/ALBERT‑based intent recognition, configurable feature storage, AutoML or expert modes, and Faas‑style deployment, as demonstrated in the Universal Ticket System case study, dramatically improving accuracy and efficiency.
The Bilibili Game Center recommendation system combines a unified feature platform, multi‑stage recall, ranking and re‑ranking models, online services, and AB experimentation to deliver personalized game suggestions, resulting in up to 78% higher click‑through, 76% higher conversion, and substantial increases in user engagement and revenue.
This article explains why and how to evaluate machine learning model quality through a structured framework that covers data validation, feature checks, and algorithm testing, helping ensure accuracy, reliability, and maintainability before deployment.
The article presents a comprehensive one‑stop AI platform that unifies training, model, feature, and decision services across cloud, edge, mobile, Flink, and application environments, detailing its architecture, the limitations of cloud‑centric inference, the advantages of localized inference, and the challenges and solutions for model and feature localization, SDK design, and future AutoML enhancements.
The team embedded a full machine‑learning ranking pipeline as an Elasticsearch plug‑in—combining real‑time and offline feature stores, hot‑loadable model jars via Dragonfly, an MLeap execution engine, and a DSL for feature definition—replacing the coarse‑ranking logistic‑regression with a tree model that adds ~10 ms latency but yields a 1.2 % AB‑test lift, while maintaining high throughput, low CPU usage, and supporting future batch deep‑learning rescoring.
This article details Zhihu's Data Management Platform (DMP), covering the business problems it solves, the end‑to‑end workflow, feature taxonomy, system architecture, data pipelines for batch and streaming, audience targeting processes, performance challenges, and future technical directions.
This article details a joint research effort between Meituan and East China Normal University that combines cost‑based methods with AI‑driven, data‑centric models to automatically generate and evaluate missing indexes for billions of daily slow queries, improving recommendation accuracy and query performance.
Tencent’s Taiji machine learning platform, a cloud‑native, distributed parameter‑server system, provides end‑to‑end MLOps for advertising by integrating data ingestion, feature engineering, model training, evaluation, deployment, and monitoring, supporting massive models up to billions of parameters while improving efficiency, scalability, and resource management.
MLOpsMachine Learning Platformdistributed training
This article introduces FeatHub, an open‑source feature‑store project from Alibaba Cloud that provides a Python SDK, flexible architecture, and execution engines such as Flink and Spark to simplify the development, deployment, monitoring, and sharing of real‑time and offline machine‑learning features across multi‑cloud environments.
The article outlines ten core components—data and feature pipelines, feature store, training and retraining pipelines, metadata store, serving infrastructure, production monitoring, reusable ML pipelines, workflow orchestration, CI/CT/CD, and end‑to‑end quality control—that together form a scalable, reliable architecture for modern machine‑learning systems.
The article presents an AI‑driven pipeline that creates, delivers, and optimizes 3D e‑commerce content by leveraging diffusion‑based generation, txt2img/img2img style transfer, Shapley‑value interpretability, and a multi‑level traffic amplification framework to overcome modeling efficiency, asset scarcity, and production cost challenges.
The article describes WeChat Games’ scalable recommendation system, detailing its four‑component architecture—offline ML platform, unified management, online DAG‑based engine, and peripheral services—along with a hybrid algorithm library, feature engineering, real‑time monitoring, and solutions that boost engagement across diverse game recommendation scenarios.
This article introduces the architecture of intelligent risk control, detailing its four-layer structure, the underlying data, feature, model, and decision components, platform interactions, and future development trends, highlighting how AI and big data enhance risk management efficiency and accuracy.
This interview compiles expert opinions on the end‑to‑end recommendation system pipeline—including architecture, data collection, user profiling, content structuring, feature engineering, recall strategies, ranking algorithms, multi‑objective optimization, multi‑modal fusion, re‑ranking, cold‑start solutions, evaluation metrics and real‑world applications—highlighting the technical challenges and practical solutions.
This article outlines the architecture of an intelligent financial risk control platform, detailing data sources, big‑data processing, feature engineering, decision engines, model types, and real‑world application scenarios, while highlighting expert‑identified challenges such as compliance, data quality, real‑time performance, and fraud detection.
The Xianyu recommendation system, built by backend expert Wan Xiaoyong, evolved from offline scoring to a full‑graph, serverless recall‑ranking pipeline that tackles C2C uncertainties through centralized feature engineering, model compression, staged deployment, flexible experimentation, robust governance, and plans for automated attribution and interpretability.
This article details how to gather FIFA and match data, engineer predictive features, build and evaluate Random Forest and Gradient Boosting models, and simulate the 2022 World Cup to forecast match outcomes and the eventual champion using Python and scikit‑learn.