Mastering Feature Engineering: From AutoML Dictionaries to Business‑Driven Insights

Feature engineering is often limited to textbook techniques such as missing‑value imputation, normalization, one‑hot encoding, and dimensionality reduction, which rarely boost the performance of powerful tree models like XGBoost or LightGBM. This article proposes a three‑pronged methodology to generate high‑impact features efficiently.

1. AutoML‑Style Brute‑Force Feature Dictionary

Generate an exhaustive set of combinatorial features from categorical variables, treating each possible interaction as a potential predictor. For two categorical fields A and B, examples include:

count:A_COUNT, B_COUNT, A_B_COUNT
nunique:A_unique_B
ratio:A_B_COUNT/A_COUNT
average:A_COUNT/A_unique_B
most:A_most_B
pivot:A_B1_count, A_B2_count
pivot2:A_B1_count-A_B2_count
stat1:A_stat_A_B_COUNT
stat2:A_stat_B_COUNT
serialization: LDA, NMF, SVD, Word2Vec, doc2vec, deepwalk, pPRoNE

Extending this to numeric, time, and target features yields virtually unlimited combinations, though it also creates many irrelevant features that can degrade model speed and accuracy.

2. Business‑Understanding‑Driven Feature Engineering

Derive features from domain knowledge, then validate them with data analysis. This approach yields highly interpretable and generalizable features. Examples:

Instacart Market Basket: weekend spikes in alcohol sales led to item‑time cross features.

TalkingData AdTracking Fraud Detection: low‑frequency IPs correlated with fraudulent clicks, prompting cross features between channel, ad, and IP frequency.

Strengths include strong interpretability and smaller models; weaknesses involve missing hidden strong features when business logic is incomplete.

3. Feature‑Importance‑Guided Engineering

Tree models provide feature‑importance scores that quickly highlight powerful predictors. By focusing on top‑ranked features, one can further enrich them through cross‑features, temporal aggregations, or embedding‑style transformations. Case studies include:

Two Sigma Rental‑Listing: manager ID identified as strong; enriched with categorical and numeric descriptors.

IJCAI 2018 competition: top numeric features were aggressively crossed to uncover additional value.

Limitations arise because this method depends on an accurate importance table; weak features are ignored.

4. Integrating the Three Approaches

Combine brute‑force dictionaries with business logic to filter out nonsensical combinations, and use feature importance to prune low‑value features. For example, replace generic categories A and B with user and item to generate:

count:user_COUNT, item_COUNT, user_item_COUNT
nunique:user_unique_item, item_unique_user
ratio:user_item_COUNT/user_COUNT
average:user_COUNT/user_unique_item
most:user_most_item
pivot:user_item1_count, user_item2_count
pivot2:user_item1_count-user_item2_count
stat1:user_stat_user_item_COUNT
stat2:user_stat_item_COUNT
serialization: LDA, NMF, SVD, Word2Vec, doc2vec, deepwalk, pPRoNE

Iteratively apply importance‑based pruning and business‑logic validation to avoid feature explosion while retaining predictive power.

5. Iterative Spiral Enhancement

Feature importance can also reveal unexpected strong features, prompting deeper business analysis and the creation of refined features (e.g., category_mean_price - price in the Avito Demand Prediction challenge). By repeatedly cycling through the three strategies—dictionary expansion, business insight, and importance‑driven refinement—practitioners achieve robust, high‑performing feature sets validated on hold‑out data.

Conclusion

The presented methodology offers a systematic, repeatable framework for feature engineering that balances exhaustive search, domain expertise, and data‑driven importance, acknowledging each approach’s strengths and weaknesses and emphasizing iterative validation.