Getting Started with Hugging Face TRL: Fine‑tune LLaVA using DPO

Hugging Face's TRL (Transformer Reinforcement Learning) library is an open‑source toolkit designed for reinforcement‑learning‑based fine‑tuning of large language models, supporting models such as Qwen and LLaMA and algorithms like PPO and DPO.

Because TRL builds on the transformers library, both packages must be installed. The required pip command is: pip install transformers, trl The article uses the multimodal LLaVA model as a concrete example and adopts the RLHF‑V preference dataset for training.

First, the llava_trainer.py file is edited: the original LLaVATrainer (a subclass of transformers.Trainer) is replaced with trl.DPOTrainer. The change is illustrated in the following image:

Next, the LazySupervisedDataset class is updated. Its __getitem__ method now reads image paths and both chosen and rejected answers from the RLHF‑V dataset, converting the answers into a dialogue‑template format. The modification is shown below:

Following the LLaVA SFT data‑processing pipeline, the chosen and rejected answers are encoded as IDs, labels are computed, and the final __getitem__ output format is displayed in the next figure:

The DataCollatorForSupervisedDataset class in train.py is also revised. Its __call__ method now builds a training batch from the IDs and labels of the chosen and rejected responses, with the resulting batch structure illustrated here:

Because the current TRL version does not support multimodal inference, the concatenated_forward method of DPOTrainer is patched to accept image inputs during training. The code change is depicted in the following image:

After these modifications, the DPO fine‑tuning pipeline for LLaVA is complete and can be executed using LLaVA’s existing training scripts.

Reference material:

https://huggingface.co/docs/trl/index

https://github.com/haotian-liu/LLaVA

https://github.com/YiyangZhou/CSR