How EasyPhoto Turns Your Photos into AI Portraits with Stable Diffusion

Background

AI‑generated personal portrait applications have become popular, and open‑source projects such as FaceChain have emerged to help developers create customized portrait generators.

EasyPhoto Project Overview

EasyPhoto is an open‑source plugin for the SD‑WebUI ecosystem that enables users to upload several photos of the same person, train a LoRA model, and generate realistic portraits by combining user‑defined template images.

Project repository: https://github.com/aigc-apps/sd-webui-EasyPhoto

Principles and Generation Pipeline

EasyPhoto leverages Stable Diffusion, face‑related AI models, and ControlNet to create personalized portrait pipelines. The generation process consists of:

Face detection on the input template (crop & warp) and replacement with a digital avatar.

FaceID model selects the best ID photo and template for face fusion.

The fused image serves as the base; the replaced face is used as a ControlNet condition together with the LoRA model for localized image‑to‑image repainting.

Stable Diffusion plus super‑resolution produces a high‑definition result while preserving the identity.

Training Process

Training involves extensive face preprocessing and verification:

Cluster and score all uploaded images using FaceID and image‑quality metrics, filtering out non‑ID photos.

Detect faces and perform subject segmentation to isolate faces and remove backgrounds.

Apply a skin‑enhancement model to improve low‑quality faces.

Label the processed images with a single annotation and train the LoRA model.

During training, a FaceID‑based verification step saves checkpoints at intervals and merges models based on similarity.

The training step count follows a simple rule: when the photo count is low, steps = photo_num × max_steps_per_photo; otherwise, steps = max_train_steps.

Key Technologies

Stable Diffusion

Stable Diffusion is an open‑source text‑to‑image diffusion model (versions 1.5, 2.1, XL, etc.) trained on large image‑text datasets. It generates images by iteratively denoising latent representations guided by textual prompts.

ControlNet

ControlNet extends Stable Diffusion with additional conditioning inputs (e.g., edge maps, depth maps, pose) to achieve finer control over generated content.

LoRA

LoRA (Low‑Rank Adaptation) fine‑tunes large models with a small set of parameters, enabling efficient personalization of identity, style, or objects.

EasyPhoto & SDWebUI Integration

EasyPhoto is packaged as an SD‑WebUI plugin, integrating face preprocessing, LoRA training, and portrait generation into a single UI.

Installation

Two methods are supported:

In SD‑WebUI, go to Extensions → Install from URL and enter https://github.com/aigc-apps/sd-webui-EasyPhoto, then install and restart.

Clone the repository directly into the extensions folder of SD‑WebUI and restart to trigger dependency installation.

EasyPhoto also requires the ControlNet plugin Mikubill/sd-webui-controlnet and at least three ControlNet models (set “Multi ControlNet: Max models amount” in the settings).

Training UI

Upload 5‑20 portrait photos (prefer half‑body, no glasses), optionally adjust training parameters, set a user ID, and start training. The interface shows training logs and model checkpoints.

Inference

After training, switch to the Inference tab, refresh the model list, select a template (or upload a custom one), and generate portraits. Prediction parameters are displayed for fine‑tuning.

Conclusion

EasyPhoto combines community‑sourced models and techniques to explore Stable Diffusion in the AI‑generated portrait domain. All images are for demonstration only; contributions and feedback are welcomed.