An Introduction to Gradient Boosting Decision Trees (GBDT) and Its Applications in Consumer Finance

1. GBDT Algorithm Overview

GBDT (Gradient Boosting Decision Tree, Friedman 1999) is widely used across domains and has been successfully applied in many scenarios within the financial division. It can be viewed as an ensemble model or a gradient‑based boosting model, built on CART regression trees and gradient descent in function space. Compared with its successors XGBoost/LightGBM, GBDT only requires the loss function to be first‑order differentiable, allowing both convex and non‑convex losses.

2. Basic Principles of GBDT

The core ideas are additive boosting and gradient boosting. Additive boosting combines multiple weak learners into a strong learner, while gradient boosting selects each weak learner to move in the direction of steepest loss reduction.

Additive Boosting

Mathematically, the strong learner at iteration t is the sum of all previous weak learners:

(1)

where denotes the cumulative sum of weak learners up to time t , and the weak learner at iteration m . The strong learner at t is thus the previous strong learner plus the new weak learner (Equation 2).

(2)

Gradient Boosting

Assuming a dataset and loss function L , the goal is to minimize the loss by learning a set of weak learners. Using first‑order Taylor expansion, the optimal weak learner at iteration t is obtained by fitting the negative gradient of the loss (Equation 4).

(4)

Further derivations lead to the update rule (Equation 5) that ensures the loss decreases most rapidly.

(5)

3. GBDT Regression and Classification Algorithms

Regression typically uses Mean Squared Error as the loss. The target at iteration t becomes the residual between the true value and the current model prediction (Equation 6).

(6)

The learning rate (shrinkage) is usually set to a small constant such as 0.1 or 0.01.

Classification transforms the problem into a regression task using a multinomial logistic loss (cross‑entropy). For K classes and T boosting rounds, K·T trees are built, and the leaf outputs are converted to class probabilities via Softmax (Equations 7‑11).

(7)

(8)

(9)

(10)

(11)

4. Scikit‑learn Demo

The following Python code shows a minimal GBDT classification pipeline using sklearn.ensemble.GradientBoostingClassifier :

from sklearn.ensemble import GradientBoostingClassifier

# Hyper‑parameters

params = {

'n_estimators': 200,

'learning_rate': 0.1,

'max_depth': 5,

'subsample': 0.5,

'min_samples_leaf': 10,

'min_samples_split': 20

}

# Load data

X_train, y_train, X_test = load_data(data_path)

# Build classifier

clf = GradientBoostingClassifier(

n_estimators=params['n_estimators'],

learning_rate=params['learning_rate'],

max_depth=params['max_depth'],

subsample=params['subsample'],

min_samples_leaf=params['min_samples_leaf'],

min_samples_split=params['min_samples_split']

)

# Train

clf.fit(X_train, y_train)

# Predict probabilities

pred_labels = clf.predict_proba(X_test)[:, 1].reshape(-1)

5. Application in Consumer Finance

In consumer finance, fraud detection is a critical challenge due to its long‑tail distribution and rapid pattern changes. By combining large‑scale data pipelines with AI techniques, the team built a transaction‑level fraud detection model based on GBDT, achieving an AUC of 93% and KS of 63% in the “拿去花” product.

6. References

[1] Friedman, “Greedy Function Approximation: A Gradient Boosting Machine”, 1999.

[2] Friedman, “Additive Logistic Regression: A Statistical View of Boosting”, 2000.