Master ControlNet in ComfyUI: A Step‑by‑Step Guide to Advanced AI Image Generation

ComfyUI Introduction

ComfyUI is a powerful, highly customizable tool for Stable Diffusion image generation, offering workflow‑based automation and better reproducibility compared to the more beginner‑friendly Stable Diffusion WebUI.

ControlNet Overview

ControlNet adds dozens of control methods to AI image generation, allowing users to steer composition, pose, style, and more, which greatly improves the quality of AI‑generated artwork and opens applications such as artistic QR codes, line‑art coloring, old‑photo restoration, and style transfer.

Installation and Deployment

Local Deployment

Requires a suitable network environment, an Nvidia GPU with at least 8 GB VRAM, and basic computer‑hardware skills.

Cloud Deployment

ComfyUI can also run on cloud servers that provide GPU resources, simplifying setup for users without powerful local hardware.

Using ControlNet

Plugin Installation

Two recommended plugins enhance ControlNet capabilities:

ControlNet Preprocessor Plugin – https://github.com/Fannovel16/comfyui_controlnet_aux Advanced ControlNet – https://github.com/Kosinkadink/ComfyUI-Advanced-ControlNet Install them via the ComfyUI manager by selecting “Install via Git URL” and entering the links.

Core ControlNet Nodes

ControlNet consists of three main nodes:

ControlNet Loader : Loads ControlNet models. ComfyUI includes a standard loader and a DiffControlNet loader that also supports diffusers‑format models. Choose the appropriate model version (SD‑1.5 or SDXL) to match the base Stable Diffusion checkpoint.

Reference Image : Provides the visual reference (e.g., depth map, pose, color map) that the ControlNet model will follow.

ControlNet Application : Combines the model, reference image, and user parameters to produce conditioned prompts for the sampler.

Key parameters of the ControlNet Application are:

Strength : Influence of the ControlNet model on the final image (higher values make the reference more dominant).

Start Time : Step (0‑1) at which ControlNet begins influencing generation; early steps affect the main subject.

End Time : Step (0‑1) at which ControlNet stops influencing generation; lowering this allows more freedom in later details.

Inputs and Outputs

The loader requires a Stable Diffusion base model, connected to a “Checkpoint Loader” node. Prompt conditions are split into positive and negative encodings, fed into a CLIP Text Encoder, and then combined with ControlNet‑processed information. The ControlNet Application outputs conditioned prompts that are passed to the sampler for image generation.

Example with Preprocessors

An example workflow adds three extra nodes:

OpenPose Pose Preprocessor : Extracts human pose from the reference image, outputting a pose map.

Perfect Pixel : Calculates the optimal resolution for the pose map based on the target image size and selected stretch mode (e.g., “Crop and Stretch” or “Stretch and Fill”).

Preview Image : Shows the generated reference image for verification (optional).

Parameters such as width, height, and stretch mode determine how the pose map is resized to match the final image dimensions.

Multiple ControlNet Usage

Advanced workflows can chain several ControlNet nodes, each with its own condition, to combine multiple controls (e.g., pose + depth). Detailed examples are available in the author’s AI art column.

Workflow Configuration Files

Pre‑made workflow configuration files can be obtained via the author’s enterprise WeChat channel for one‑click import into ComfyUI.