How to Build a Personalized Fashion Recommendation System with Deep Learning

In this tutorial we demonstrate how to construct a personalized fashion recommendation system using four deep‑learning models to extract user and clothing features.

Product‑based recommendation

Other‑user behavior recommendation

User‑profile recommendation

Hybrid recommendation based on the above

We first build a user profile using attributes such as gender, age, height, and weight. AlphaPose evaluates the full‑body pose, detecting 19 key points (at least 17 required) to ensure a complete user portrait.

We fine‑tune the YOLO v3 image classifier on a subset of the DeepFashion dataset from MMLAB to recognize clothing items.

Another model, Dlib’s get_frontal_face_detector(), built on five HOG filters, quickly detects frontal and side faces. For age and gender estimation we follow standard data‑science practice using OpenCV and a convolutional neural network.

Body‑Mass‑Index (BMI) is estimated following the method described in “Learning and Transferring BMI from Facial Images” using Keras.

Model Integration

The following Python 3.5 code loads the models, performs pose assessment, and extracts face regions for precise age, gender, and BMI evaluation.

detector = dlib.get_frontal_face_detector()
# Load CNN models for age and gender
age_net, gender_net = load_caffe_models()
# Body Mass Index model
model_bmi = get_trained_model()
# Face detection and processing
img_h, img_w, _ = np.shape(image)
detected = detector(image, 1)
faces = np.empty((1, config.RESNET50_DEFAULT_IMG_WIDTH, 3))
if len(detected) > 0:
    for i, d in enumerate(detected):
        x1, y1, x2, y2, w, h = d.left(), d.top(), d.right()+1, d.bottom()+1, d.width(), d.height()
        xw1 = max(int(x1 - margin * w), 0)
        yw1 = max(int(y1 - margin * h), 0)
        xw2 = min(int(x2 + margin * w), img_w - 1)
        yw2 = min(int(y2 + margin * h), img_h - 1)
        cv2.rectangle(image, (xw1, yw1), (xw2, yw2), (255, 0, 0), 2)
        face_img = frame[yw1:yw2, xw1:xw2].copy()
        age, gender = get_age_and_gender(face_img, age_net, gender_net)
        faces[0, :, :, :] = cv2.resize(face_img, (config.RESNET50_DEFAULT_IMG_WIDTH, 3)) / 255.0
        bmi = round(float(model_bmi.predict(faces)[0][0]), 2)
        detection[i] = {'gender': gender, 'age': age, 'bmi': bmi}

Next, YOLO classifies the clothing items and suggests suitable attire.

def eval_cloth(img_test, categoria_test, size_test):
    filename = './ClothEmbedding/X_reduced2.sav'
    X_reduced, hasher, pca, df = joblib.load(filename)
    img = cv2.imread(img_test)
    img_c = cv2.resize(img, (80, 80), interpolation=cv2.INTER_CUBIC)
    img_data_test = img_c.reshape(-1).astype(np.float32)
    img_transformed = hasher.transform(img_data_test.reshape(1, -1))
    img_reduced = pca.transform(img_transformed)
    # Distance between sample and database
    dist = [np.linalg.norm(img_reduced - e) for e in X_reduced]
    df['distance'] = dist
    df_test = df.sort_values(['distance'])
    # Keep only the required category
    df_test = df_test[df_test['categoria2'] == categoria_test]
    # Keep only the required sizes
    cat_ns = ['tacones', 'chanclas', 'botas', 'bolsa', 'ropa_interior']
    if not (categoria_test in cat_ns):
        if len(size_test) == 2:
            true_table = [(size_test[0] in sizes_r or size_test[1] in sizes_r) for sizes_r in df_test['tallas']]
        else:
            true_table = [size_test[0] in sizes_r for sizes_r in df_test['tallas']]
        df_test = df_test[true_table]
    return df_test

The final component receives user information and full clothing data, compares garment features with the database, and delivers personalized outfit recommendations.