FreeStyle: Mining Community LoRAs for Balanced Style‑Content Dual‑Reference Image Generation

The paper addresses the practical problem of generating images conditioned on both a content reference and a style reference while preventing the unwanted copying of specific objects or scenes from the style image.

Community LoRA Mining Pipeline

FreeStyle treats the abundant LoRAs available on platforms such as Civitai, TensorArt, and Liblib as a natural data source. It collects LoRAs, filters them based on base model compatibility, metadata, trigger words, demo images, and generation results, and classifies them into stable content LoRAs and style LoRAs.

Using the filtered LoRAs, the pipeline generates candidate images with a prompt pool and ComfyUI workflow, then employs a visual‑language model and feature‑similarity metrics to discard unstable or inconsistent samples. Finally, compatible content‑style LoRA pairs are selected, yielding two datasets:

SRef style‑reference data : ~619K sequences covering 622 styles, used for traditional style‑reference generation.

CRef + SRef dual‑reference data : ~480K sequences covering 1,704 styles, containing content reference, style reference, textual instruction, and target image.

Two‑Stage Curriculum Training

Stage 1 – Stable style‑reference learning : Trains on SRef data to teach the model how to transfer visual style to the content image. The authors observe that content leakage in SRef scenarios is linked to excessive attention on style‑reference tokens, especially in certain denoising steps and shallow transformer blocks.

To mitigate this, an attention‑level enrichment constraint is introduced, which reshapes the attention distribution so the model extracts sufficient style information without over‑relying on specific content from the style image.

Stage 2 – Content‑style dual‑reference learning : Incorporates CRef + SRef data, enabling the model to handle content reference, style reference, and textual prompts simultaneously. Here, leakage can also arise from RoPE positional encoding that creates local patch‑level correspondences between style and output images.

The authors propose a frequency‑aware RoPE modulation that suppresses high‑frequency positional signals (prone to copying local details) while enhancing low‑frequency components that convey global structure and overall style. This modulation is applied only to the style‑reference branch, preserving the content branch’s structural information.

Benchmark and Evaluation

FreeStyle builds a comprehensive benchmark evaluating both SRef and CRef + SRef tasks across five dimensions: style consistency, content consistency, instruction adherence, aesthetic quality, and leakage degree. A key metric is the Content Alignment Score (CAS), based on DINOv2 features with instance normalization to focus on structural similarity while ignoring style variations.

Experimental results show that FreeStyle does not chase a single metric but achieves a balanced improvement: strong style transfer, good content preservation, and reduced leakage. On the SRef benchmark, it attains high style consistency and verification scores; on the more challenging CRef + SRef benchmark, it maintains solid style transfer while limiting content copying.

Ablation Studies

Two core modules are validated:

Attention‑level enrichment constraint : Removing it increases content leakage; adding it significantly reduces leakage in SRef generation.

Frequency‑aware RoPE modulation : Without modulation, the model more readily forms local correspondences between style and output, leading to leakage; with modulation, leakage is mitigated while style transfer remains effective.

Additional comparisons demonstrate that models trained on the mined LoRA data outperform those trained on manually collected datasets, especially for complex and long‑tail styles.

Open‑Source Resources

The authors release code, datasets, benchmark data, and model checkpoints:

Project code: https://github.com/Blue2Giant/FreeStyle

Dataset: https://huggingface.co/datasets/Blue2Giant/FreeStyle_Dataset

Benchmark: https://huggingface.co/datasets/Blue2Giant/FreeStyle_Bench

Model weights: https://huggingface.co/Blue2Giant/FreeStyle_Checkpoint

The repository contains pipelines for LoRA data production, metadata, multi‑model inference and benchmark evaluation, and a minimal inference demo.

Conclusion

FreeStyle contributes three main advances: (1) a systematic community LoRA mining pipeline for large‑scale style‑content supervision data; (2) a two‑stage training strategy with attention‑level enrichment and frequency‑aware RoPE modulation to curb different forms of content leakage; and (3) a multi‑dimensional benchmark that assesses style, content, instruction compliance, aesthetics, and leakage jointly. The work demonstrates that community LoRA ecosystems can be leveraged to build diverse, high‑quality datasets for controllable image generation tasks.