This article walks through the entire feature‑engineering pipeline—data cleaning, missing‑value imputation, encoding, outlier handling, scaling, feature construction, and selection—using Pandas and Scikit‑learn, and shows how to wrap the steps into a reproducible Scikit‑learn Pipeline.
This article walks through a full feature‑engineering pipeline using Pandas and Scikit‑Learn, covering data inspection, missing‑value imputation, categorical encoding, outlier handling, scaling, feature construction, selection, and a final Pipeline that prepares clean, predictive features for a logistic‑regression model.
The article explains that setting the mean to zero recenters data around the origin, making optimization algorithms converge faster, while scaling variance to one equalizes feature scales so no single feature dominates, illustrated with examples and visualizations of how standardization improves machine‑learning models.
The article explains various learning paradigms (supervised, unsupervised, self‑supervised, semi‑supervised, and reinforcement), describes dataset types and quality considerations, outlines preprocessing steps like filtering, deduplication, and tokenization, and discusses scaling laws linking model size, data volume, and compute resources, with concrete examples and code.
This article outlines eight practical techniques—including data augmentation, handling missing values, feature engineering, algorithm selection, hyperparameter tuning, ensemble methods, and cross‑validation—to systematically improve the accuracy of Python machine‑learning models, supported by explanations, examples, and code snippets.
This article details how Pinterest’s senior staff engineer Dr. Luo leveraged the open‑source Ray framework to overcome LLM data‑preprocessing bottlenecks, describing the system’s architecture, key features such as map_batches, Carry‑Over Columns and Accumulators, and the dramatic performance and cost improvements achieved.
This article presents 19 practical Sklearn functions—ranging from outlier detection to hyper‑parameter search—that replace manual data‑science steps, each illustrated with concise code examples and performance comparisons.
Data‑Juicer is an open‑source, one‑stop multimodal data processing system that provides fine‑grained operators, scalable pipelines, and ready‑made recipes to deliver high‑quality, diverse, and model‑friendly data for large language model pre‑training, fine‑tuning, and multimodal applications.
This article presents ten concise, practical one‑line Python code snippets—ranging from loading CSV data with Pandas to building sophisticated Scikit‑learn pipelines—that streamline common machine‑learning tasks such as data cleaning, encoding, splitting, scaling, model training, evaluation, cross‑validation, and prediction.
This article provides a comprehensive guide to decision tree algorithms, covering their theoretical foundations, key components, construction workflow—including data preprocessing, feature selection, tree growth, stopping criteria, and pruning—followed by an overview of popular variants like ID3, C4.5, CART, practical advantages, applications, and a complete Python implementation using scikit-learn.
This tutorial demonstrates how to convert a JSON lyric database into Excel, filter Jay Chou songs, perform Chinese word segmentation with Jieba, compute word frequencies, and create visualizations such as word clouds using Python code and online tools.
This tutorial walks through a complete house price prediction project, covering data collection from Kaggle, preprocessing with pandas and scikit‑learn, model training using RandomForestRegressor, evaluation, and deployment of a Flask API for real‑time predictions, providing full code examples.
This tutorial walks through consolidating the first ten days of learning by covering data preprocessing, feature engineering, model training with linear regression, decision tree, and random forest, model evaluation using cross‑validation, and finally saving and loading the best model, all illustrated with complete Python code examples.
This guide walks through using Pandas for data cleaning, feature engineering, and preparation, then demonstrates building, evaluating, and persisting a machine‑learning model with Scikit‑learn's pipeline and RandomForestClassifier in Python.
This article provides a comprehensive, step‑by‑step guide to training customized large language models, covering industry‑specific needs, data privacy, meticulous data cleaning, optimal data‑ratio balancing, token budgeting, GPU memory accounting, LoRA fine‑tuning techniques, and practical evaluation metrics for robust AI deployment.
This article introduces the Daily_and_Sports_Activities sensor dataset, details its structure and characteristics, provides a Python preprocessing pipeline with sliding‑window segmentation and Z‑score normalization, and reports CNN training results achieving 87.93% accuracy on activity classification.
This article explains the importance of data dimensionality reduction and feature extraction in machine learning, and provides a step‑by‑step guide with PHP code examples—including library installation, data preprocessing, PCA‑based reduction, and feature selection techniques—demonstrating how to handle large datasets efficiently.
This article provides a comprehensive guide to data cleaning in Python, covering common data issues, methods for handling missing values, duplicates, categorical inconsistencies, and text normalization, illustrated with 21 detailed code examples using pandas and matplotlib.
Feature engineering transforms existing data into new predictive variables through manual analysis or automated pipelines, encompassing single‑variable encoding, pairwise arithmetic, group‑statistics, multi‑variable combinations, time‑series and text derivations, with tools like Deep Feature Synthesis and beam‑search to generate and select useful features.
This article presents ten essential data preprocessing methods—including handling missing values, type conversion, standardization, encoding, smoothing, outlier treatment, text cleaning, word frequency counting, sentiment analysis, and topic modeling—each explained with clear Python code snippets.
This article walks through cleaning irregular fan data—removing spaces, Chinese characters, asterisks, and missing brackets—using Python's pandas library, provides the full code snippet, demonstrates the resulting DataFrame, and shares practical tips for preparing data before analysis.
This article presents a comprehensive collection of Python code snippets demonstrating essential data cleaning methods—including handling missing values, outlier detection, type conversion, formatting, duplicate removal, normalization, one‑hot encoding, text preprocessing, and dataset merging—providing practical guidance for preparing data for analysis or machine‑learning tasks.
This tutorial demonstrates how to preprocess a Chinese lyrics JSON dataset, extract Jay Chou's songs using Python, perform word segmentation with Jieba, compute word frequencies, and create visualizations such as word clouds both programmatically and with online tools.
This article presents practical feature‑engineering methods for ten common business domains, explaining the purpose of each feature, the extraction technique, and providing ready‑to‑run Python code snippets to help build more accurate predictive models.
This guide walks through ten core data preprocessing methods—including handling missing values, type conversion, standardization, encoding, smoothing, outlier treatment, text cleaning, word‑frequency counting, sentiment analysis, and topic modeling—each illustrated with concise Python code examples.
RFM analysis is a powerful tool for understanding customer value and behavior, enabling businesses to optimize marketing strategies, improve customer satisfaction, and increase business revenue through data-driven customer segmentation.
This article analyzes upgrade paths of major LLMs—ChatGLM, LLaMA, Baichuan—detailing performance, context length, and architectural changes, then examines essential capabilities, data cleaning, tokenizer and attention design, and offers practical guidance for balanced scaling and efficient model construction.
This article outlines essential data cleaning steps—including handling missing and duplicate values, type conversion, outlier treatment, text processing, standardization, sampling, and merging—providing concise Python pandas code snippets for each technique to improve data quality for analysis.
GoldMiner, a new system from Alibaba Cloud’s PAI platform, elastically scales deep learning data pre‑processing pipelines, dramatically improving training performance up to 12.1× and GPU cluster utilization by 2.5×, and its underlying research was accepted at SIGMOD 2023.
This article demonstrates how to use Python's concurrent.futures module to parallelize image resizing, turning a single‑process script into a multi‑core solution with just three additional lines of code, achieving up to a six‑fold speed‑up on typical CPUs.
Before training large language models like LLaMA, MetaAI applies a multi‑stage CCNet pipeline that crawls web data, stores it in WET format, deduplicates paragraphs, detects and filters languages using fastText, and further refines content by similarity to Wikipedia and citation‑based linear models.
Factor analysis, a statistical technique for uncovering underlying common factors among variables, is explained alongside its distinction from PCA, detailed procedural steps, adequacy tests, and a hands‑on Python implementation using the factor_analyzer library with visualizations and factor rotation methods.
This tutorial walks beginners through the essential steps of data preprocessing for any machine learning model, covering library imports, dataset loading, handling missing values, encoding categorical features, splitting into train‑test sets, and applying feature scaling using Python’s scikit‑learn.
This article details a data‑mining project that uses historical World Cup match data, extensive feature engineering, and various machine‑learning algorithms—including neural networks, logistic regression, SVM, decision trees, and random forests—to predict the champion of the 2022 tournament, while analyzing model errors and proposing improvements.
Before statistical analysis in R, you need to preprocess data by sorting vectors with sort(), rank(), order() or arrange(), merging datasets horizontally with merge() or cbind() and vertically with rbind(), and handling missing values using NA, NaN, na.rm, and na.omit functions.
This guide walks through a comprehensive set of feature‑selection strategies—from removing unused or missing columns to handling multicollinearity, low‑variance features, and using PCA—complete with Python code examples and visualizations to help you build leaner, more interpretable machine‑learning models.
In the era of deep learning, feature engineering remains crucial for recommendation and search advertising because it bridges raw relational data and models, improves performance, reduces complexity, and handles high‑cardinality, large‑scale, and time‑sensitive scenarios with robust transformations and statistical encoding.
This article explains how feature engineering in game data involves classifying structured and unstructured, quantitative and qualitative features, and details practical methods for handling missing values and outliers to improve machine‑learning model performance.
This article explains why meticulous feature engineering remains crucial in recommendation and search advertising, outlines what constitutes good features, describes common transformation techniques such as scaling, binning, and encoding, and provides practical examples and Q&A for practitioners.
This article explains why discretizing continuous variables improves model stability, introduces three common binning techniques—equal-width, equal-frequency, and clustering—and demonstrates how to implement each using scikit‑learn's KBinsDiscretizer with Python code examples on a synthetic score dataset.
This article demonstrates four Pandas techniques—between with loc, cut, qcut, and value_counts—to discretize numeric variables into bins, assigning grades A, B, C to exam scores, and shows how to generate synthetic data, define bin boundaries, and count records per bin.
This article explains the GM(1,1) grey prediction model, covering data preprocessing, model construction, parameter estimation, error testing methods, and how to generate forecasts, providing a practical guide for applying the technique to time‑series data.
This article explains essential time series preprocessing techniques—including data sorting, handling missing values with interpolation methods, applying rolling averages, Fourier transform denoising, and detecting anomalies using rolling statistics, isolation forests, and K‑means clustering—illustrated with Python code on the AirPassengers and Google stock datasets.
This article presents a comprehensive, practical methodology for feature engineering that combines brute‑force AutoML‑style dictionary searches, business‑logic‑driven feature creation, and feature‑importance‑guided refinement, illustrating each approach with real Kaggle competition examples and concrete code snippets.
This article explains how a five‑line pandas script can generate frequency‑based encodings for categorical columns, why such encodings help low‑frequency categories, and how they integrate with feature crossing and industry use cases like ad exposure, PV/UV counting, and fraud detection.
This article explains the concept of feature engineering, illustrates it with a height‑weight classification example, compares kernel‑enhanced models to handcrafted features like BMI, discusses its impact on model performance, and highlights practical tips and domain‑specific considerations.
This guide explains why data cleaning is essential, categorizes common problems such as missing values, noise/outliers, and inconsistent records, and provides step‑by‑step procedures and Pandas code snippets to detect, diagnose, and remediate each issue in real‑world datasets.
This article presents a collection of straightforward yet practical Python functions for data cleaning tasks—including dropping columns, changing data types, converting categorical variables, handling missing values, removing unwanted characters, trimming whitespace, conditional concatenation, and converting string timestamps—designed to streamline preprocessing in data analysis projects.
This article details how the logistics HR data preprocessing platform processes around 20 million daily records by adopting HBase for high‑performance, scalable, column‑oriented storage, covering its architecture, read/write mechanisms, best practices, and performance considerations.
This article details the background, MMoE model architecture, baseline setup, environment configuration, data preprocessing, training process, evaluation results, and department information for the 58.com recruitment recommendation AI competition using the WPAI platform.
This article introduces the HR performance data preprocessing platform’s requirements, explains why HBase was selected as the storage solution, details its core concepts, architecture, data write/read processes, best practices, limitations, and presents performance metrics demonstrating its suitability for large‑scale, high‑throughput workloads.
This article explains how HBase’s distributed column‑oriented architecture, high‑performance read/write capabilities, and flexible schema make it a cost‑effective solution for handling massive, unstructured HR performance data, covering its core concepts, cluster operation, best practices, and performance metrics.
The article reviews the most important additions in scikit‑learn 0.24, including faster hyper‑parameter search methods, ICE plots, histogram‑based boosting improvements, new feature‑selection tools, polynomial‑feature approximations, a semi‑supervised classifier, MAPE metric, enhanced OneHotEncoder and OrdinalEncoder handling, and a more flexible RFE interface.
This tutorial explains why and how to reduce the number of features in the Kaggle Mushroom Classification dataset with Python, covering preprocessing, various feature‑selection methods (filter, wrapper, embedded), code examples, model training, performance impact, and visualisation of results.
This article explains how feature engineering transforms raw data into numerical representations for machine‑learning models, covering mapping of numeric and categorical values, one‑hot and multi‑hot encoding, sparse representations, scaling, handling outliers, binning, data quality checks, and feature interactions to capture non‑linear relationships.
This article explains the purpose of data exploration and cleaning, outlines core analysis tasks, details missing‑value and outlier handling techniques—including various imputation methods—and illustrates the complete workflow with example images and a histogram‑based distribution analysis.
This article walks through five common data‑cleaning scenarios—dropping/renaming columns, handling duplicates and nulls, string manipulation, merging tables, and window‑function ranking—showing practical SQL and Python code that can be applied to large‑scale data warehouses.
This guide explains the fundamentals of tensors in TensorFlow.js—including scalars, vectors, and higher‑dimensional tensors—demonstrates how to convert real‑world data such as the Titanic dataset into tensors, and shows how to build, compile, and train a simple neural network model using appropriate layers, loss functions, and optimizers.
This article provides an extensive overview of feature engineering, covering feature understanding, cleaning, construction, selection, transformation, and dimensionality reduction techniques, illustrated with Python code using the Titanic dataset, and offers practical guidelines for improving data quality and model performance in machine learning projects.
This tutorial walks through the complete feature‑engineering pipeline for the Alibaba Tianchi “Tmall User Repeat Purchase Prediction” competition, covering data acquisition, memory‑efficient preprocessing, multi‑entity feature construction, statistical aggregations, text vectorisation, embedding generation and stacking‑based model features, all illustrated with Python code and diagrams.
This tutorial walks through using pandas for data import, inspection, preprocessing, analysis, and visualization, providing practical code snippets and explanations to help readers understand essential data manipulation techniques in Python.
This article presents a hand‑drawn, illustrated walkthrough of the entire machine‑learning pipeline—from dataset definition, exploratory data analysis, preprocessing, and data splitting to model building, algorithm selection, hyper‑parameter tuning, feature selection, and evaluation for both classification and regression tasks.
This tutorial walks through collecting and cleaning the MovieLens dataset, constructing rating and record matrices, normalizing ratings, defining a collaborative‑filtering model in TensorFlow, training it with Adam optimizer, evaluating performance, and finally generating personalized movie recommendations for a chosen user.
Feature engineering, which occupies the majority of data scientists' time, is essential for building high‑performing machine‑learning models and involves careful data quality control, diverse construction techniques, rigorous selection, and emerging automation efforts, all of which demand domain expertise and systematic practice.
This article explains why feature engineering consumes most of a data scientist's time, outlines its critical steps—including data observation, cleaning, transformation, selection, and reduction—covers practical issues such as missing‑value handling, data leakage, and feature stability, and discusses both manual and automated approaches for building effective machine‑learning models.
The article presents a complete workflow for predicting next‑day API request volumes by exploring per‑minute traffic data, handling missing values, applying seasonal decomposition, training an ARIMA model on the trend component, and generating confidence intervals to flag anomalous spikes.
This article provides a comprehensive, step‑by‑step guide to using pandas for data preprocessing, merging, indexing, sorting, grouping, extraction, filtering, sampling, and statistical analysis, complete with clear code examples and visual results to help readers master essential data manipulation in Python.
This article provides a beginner-friendly overview of machine learning fundamentals, covering the definition of data, the distinction between features and labels, types of features, dimensionality, training and test datasets, normalization, supervised and unsupervised learning methods, algorithm selection, development workflow, and recommended Python libraries such as NumPy.
This article describes how a GBDT‑based prediction model was built to forecast file release application parameters such as volume ratio, target audience, and gray level, covering data collection, feature engineering, model training, service deployment, and practical considerations for handling bad cases.
Effective short‑video recommendation relies on meticulous feature engineering that transforms raw signals—numerical counts, categorical IDs, content and user embeddings, context and session data—through bucketization, scaling, crossing, and smoothing, then selects and evaluates them via filtering, wrapping, regularization, and importance analysis to mitigate business biases and improve multi‑objective ranking performance.
The article describes the end‑to‑end architecture, data collection, preprocessing, feature engineering, model selection, training, and hyper‑parameter tuning of 58’s car price estimation platform, which leverages Spark, XGBoost, LightGBM and custom business rules to predict vehicle resale values.
This comprehensive guide introduces data mining and machine learning concepts, walks through Python data preprocessing techniques, reviews common classification algorithms, demonstrates an Iris flower classification case, and offers practical tips for selecting the most suitable algorithm for a given problem.
This article explains how to use machine‑learning techniques—including data preprocessing, N‑gram, CBOW, and various RNN/LSTM models—to automatically recommend the next function in a test‑case step sequence, improving writing speed and efficiency for developers.
This article introduces logistic regression, explains its theoretical basis, details key scikit-learn parameters, and provides a complete Python example for breast cancer classification, covering data preprocessing, model training, prediction, and evaluation with classification reports.
This article explains that there is no universal solution for selecting machine learning algorithms and outlines practical factors—such as data characteristics, problem type, business constraints, and algorithm complexity—to help practitioners systematically narrow down and pick the most suitable models.
A participant from Harbin Institute of Technology shares practical data‑preprocessing tricks, model choices, useful feature ideas, and a resilient mindset gained while competing in the Tencent Advertising Algorithm Contest, offering tips that can help other data scientists handle large‑scale ad data.
This comprehensive tutorial walks you through Python data mining and machine learning fundamentals, covering data preprocessing techniques, common classification algorithms, an Iris flower classification case study, and practical tips for selecting the right algorithm, all illustrated with clear code examples and visualizations.
This article breaks down the Kaggle toxic comment classification competition, detailing thorough data cleaning, advanced word‑vector techniques, pseudo‑labeling, BPE tokenization, diverse neural models and ensemble strategies, and shares practical insights and pitfalls from the author's nine‑month competition journey.
This article walks through a complete machine‑learning workflow on the UCI Census Income dataset, covering data exploration, preprocessing (including log‑transformation and scaling), model training with Naïve Bayes, Decision Tree and SVM, performance evaluation, hyper‑parameter tuning via grid search, feature importance analysis, and feature selection, providing code snippets and visualizations.
The week‑3 champion team from Peking University shares their experience in the Tencent social advertising algorithm contest, detailing data preparation, feature engineering, and model training strategies that helped them secure the top spot.
This article provides practical guidance on data cleaning, covering the importance of data wrangling, using assertions, handling incomplete records, checkpointing, testing on subsets, logging, optional raw data storage, and validating the cleaned dataset to ensure reliable downstream analysis.
This article explains how feature engineering and principal component analysis (PCA) can be applied to a two‑feature binary classification problem in R, illustrating data exploration, model evaluation with ROC AUC, and the impact of dimensionality reduction on predictive performance.
This article presents nine fundamental laws of data mining—covering goals, knowledge, preparation, experimentation, patterns, insight, prediction, value, and change—explaining how business objectives and domain expertise drive each stage of the CRISP‑DM process and why technical metrics alone cannot guarantee success.
