10 Hidden Sklearn Features That Boost Your ML Pipelines

Scikit‑learn offers many powerful tools that go beyond the standard preprocessing transformers. The following sections demonstrate ten such features with concrete code snippets and practical guidance.

1. FunctionTransformer

Wrap any custom preprocessing function so it can be used inside a pipeline. The function must accept a feature array X and an optional target y, and return the transformed arrays.

from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import FunctionTransformer

def reduce_memory(X: pd.DataFrame, y=None):
    """Simple function to reduce memory usage by casting numeric columns to float32."""
    num_cols = X.select_dtypes(incluce=np.number).columns
    for col in num_cols:
        X[col] = X.astype("float32")
    return X, y

ReduceMemoryTransformer = FunctionTransformer(reduce_memory)
make_pipeline(SimpleImputer(), ReduceMemoryTransformer)

The transformer converts a plain Python function into a Scikit‑learn compatible step, preserving the pipeline’s atomic, single‑call semantics.

2. User‑Defined Transformers

When a preprocessing step cannot be expressed with existing transformers—e.g., applying a log transform to features that contain zeros—a custom transformer class inheriting from BaseEstimator and TransformerMixin can handle the edge case gracefully.

from sklearn.base import BaseEstimator, TransformerMixin
from sklearn.preprocessing import PowerTransformer

class CustomLogTransformer(BaseEstimator, TransformerMixin):
    def __init__(self):
        self._estimator = PowerTransformer()
    def fit(self, X, y=None):
        X_copy = np.copy(X) + 1  # add one to avoid log(0)
        self._estimator.fit(X_copy)
        return self
    def transform(self, X):
        X_copy = np.copy(X) + 1
        return self._estimator.transform(X_copy)
    def inverse_transform(self, X):
        X_reversed = self._estimator.inverse_transform(np.copy(X))
        return X_reversed

3. TransformedTargetRegressor

For regression problems where the target y needs preprocessing (e.g., log‑scaling), TransformedTargetRegressor wraps a regressor and applies a transformer to y before fitting.

from sklearn.compose import TransformedTargetRegressor
reg_lgbm = lgbm.LGBMRegressor()
final_estimator = TransformedTargetRegressor(
    regressor=reg_lgbm, transformer=CustomLogTransformer()
)
final_estimator.fit(X_train, y_train)

4. HTML Estimator Representation

Complex pipelines can become unreadable in the console. Setting display="diagram" via sklearn.set_config renders an interactive HTML diagram in Jupyter notebooks, making the pipeline structure clear.

from sklearn import set_config
set_config(display="diagram")
giant_pipeline

5. QuadraticDiscriminantAnalysis (QDA)

QDA achieved a 0.965 ROC‑AUC on the Kaggle Instant Gratification competition without hyper‑parameter tuning, outperforming many tree‑based models. It trains in seconds on a million‑row dataset but requires features that follow a normal distribution.

%%time
from sklearn.discriminant_analysis import QuadraticDiscriminantAnalysis
X, y = make_classification(n_samples=1000000, n_features=100)
qda = QuadraticDiscriminantAnalysis().fit(X, y)
# Wall time: 13.4 s

6. Voting Classifier / Regressor

Voting ensembles combine several models by majority vote (classification) or averaging (regression). Soft voting uses predicted probabilities, and optional weights can bias the contribution of each estimator.

from sklearn.ensemble import VotingClassifier
X, y = make_classification(n_samples=1000)
ensemble = VotingClassifier(
    estimators=[
        ("xgb", xgb.XGBClassifier(eval_metric="auc")),
        ("lgbm", lgbm.LGBMClassifier()),
        ("cb", cb.CatBoostClassifier(verbose=False)),
    ],
    voting="soft",
)
ensemble.fit(X, y)

7. Stacking Classifier / Regressor

Stacking trains multiple base models, collects their predictions as new features, and fits a final estimator on this meta‑dataset. Diversity among base learners (trees, linear models, neighbors, Bayesian, etc.) reduces bias and mitigates over‑fitting, a pattern common in winning Kaggle solutions.

from sklearn.ensemble import StackingClassifier
from sklearn.linear_model import LogisticRegression
X, y = make_classification(n_samples=1000)
ensemble = StackingClassifier(
    estimators=[
        ("xgb", xgb.XGBClassifier(eval_metric="auc")),
        ("lgbm", lgbm.LGBMClassifier()),
        ("cb", cb.CatBoostClassifier(verbose=False)),
    ],
    final_estimator=LogisticRegression(),
    cv=5,
    passthrough=False,
)
ensemble.fit(X, y)

8. LocalOutlierFactor + UMAP

Detecting outliers in high‑dimensional data is costly. Combining UMAP for dimensionality reduction with LocalOutlierFactor yields fast, accurate anomaly detection on datasets with dozens of features.

%%time
import umap
from sklearn.neighbors import LocalOutlierFactor
X, y = make_classification(n_samples=5000, n_classes=2, n_features=10)
X_reduced = umap.UMAP(n_components=2).fit_transform(X, y)
lof = LocalOutlierFactor()
labels = lof.fit_predict(X_reduced, y)
# Wall time: 17.8 s

9. QuantileTransformer

For heavily skewed or multimodal distributions, QuantileTransformer uses robust statistics (quartiles, median) to map features onto a near‑normal distribution, handling arbitrary numbers of peaks.

import pandas as pd
from sklearn.preprocessing import QuantileTransformer
qt = QuantileTransformer().fit(crazy_distributions)
crazy_feature_names = ["f18", "f31", "f61"]
crazy_distributions = pd.DataFrame(qt.transform(crazy_distributions), columns=crazy_feature_names)

10. PCA + tSNE / UMAP

When datasets are too large for direct tSNE, a two‑stage reduction—first PCA to 30‑50 dimensions, then tSNE or UMAP—balances speed and fidelity. On a synthetic 1 M‑row, 300‑feature dataset, PCA + tSNE took over 4 hours, while PCA + UMAP completed in ~15 minutes and preserved class separation.

from sklearn.decomposition import PCA
from sklearn.manifold import TSNE
manifold_pipe = make_pipeline(QuantileTransformer(), PCA(n_components=30), TSNE())
reduced_X = manifold_pipe.fit_transform(X, y)
# Wall time: 4.5 h (tSNE)

# Faster alternative
manifold_pipe = make_pipeline(QuantileTransformer(), PCA(n_components=30))
X_pca = manifold_pipe.fit_transform(X, y)
embedding = umap.UMAP(n_components=2).fit(X_pca, y)
# Wall time: 14 min 27 s

These hidden Sklearn utilities can streamline preprocessing, improve model performance, and reduce code duplication, making them valuable additions to any data‑science workflow.