If Life Were an RLHF, Who’s Shaping Your Rewards?

1 RLHF Overview

RLHF (Reinforcement Learning from Human Feedback) integrates reinforcement learning with human‑generated feedback to align large language model (LLM) outputs with human preferences. The process consists of three stages: pretraining a base model, supervised fine‑tuning (SFT), and finally applying RLHF to polish the model’s behavior.

Pretrained model : a massive, unfettered model trained on raw internet data, which inevitably contains unsafe content such as clickbait, misinformation, and hateful speech.

SFT (Supervised Fine‑Tuning) : the model is refined on higher‑quality data—e.g., StackOverflow, Quora, and manually annotated examples—so it conforms to societal standards.

RLHF : a polishing step that adds a “smile” to the fine‑tuned model, making it more suitable for target audiences.

All three steps can be skipped, but using them together yields the best results. Pretraining consumes about 98 % of total compute and data resources, while SFT and RLHF unlock capabilities already present in the pretrained model.

OpenAI’s 2017 paper highlighted the importance of aligning models with human preferences.

2 Stage 1: Pretraining LLM

Pretraining produces foundation models such as GPT‑x (OpenAI), Gopher (DeepMind), LLaMA (Meta), and StableLM (Stability AI). Language models encode statistical information about token likelihoods in context; their performance depends on data quality ("Garbage in, garbage out").

Data Scale Bottleneck

Since May 2023, training data size has only grown: GPT‑3 used 0.5 trillion tokens, LLaMA used 1.4 trillion tokens—roughly the text of 15 million books. Public data (the open internet) is used first; when insufficient, proprietary data (copyrighted books, transcripts, contracts, medical records, genomic sequences, user data, etc.) is added, giving companies with such data a competitive edge.

3 Stage 2: Supervised Fine‑Tuning (SFT) for Dialogue

Why SFT

Pretraining optimizes a model’s ability to complete text, enabling three kinds of behavior: adding context, asking follow‑up questions, and providing answers. SFT’s goal is to improve the third behavior—answering questions.

SFT trains the model on demonstration data formatted as [prompt, response], showing the model how to react. An InstructGPT model with only 1.3 B parameters outperforms the 175 B GPT‑3 model after SFT.

Demonstration Data

Human annotators (over 90 % with at least a university degree, one‑third with a master’s) created 13 000 high‑quality [prompt, response] pairs for OpenAI; DeepMind used a heuristic algorithm to filter pretraining data. Databricks’ Dolly used ~15 k pairs generated by its own staff.

4 Stage 3: RLHF

SFT + RLHF markedly improves performance. Demonstration data tells the model what responses are reasonable, but not how good they are. RLHF introduces a scoring function that evaluates responses, allowing the model to learn to generate higher‑scoring answers.

RLHF consists of two parts:

Training a Reward Model (RM) that assigns a score to a [prompt, answer] pair.

Using reinforcement learning to optimize the supervised policy so that it produces answers with higher RM scores.

In practice, a prompt is sampled, the SFT policy generates multiple answers (e.g., A‑D), and annotators rank them. The RM learns from these rankings (learning‑to‑rank) rather than absolute scores, reducing noise from individual annotator bias.

The trained RM then replaces human annotators: a prompt is fed to the policy, the answer is scored by the RM, and the score is fed back to the policy for further learning.

OpenAI’s instruction‑following article (https://openai.com/index/instruction-following/) illustrates this pipeline.

5 If Life Were an RLHF

Most people’s lives are shaped by a continuous training‑and‑feedback loop rather than free choice.

You = pretrained or fine‑tuned model

Your life experiences = training data

Your values, goals, happiness = reward function (ultimate objective)

Your actions and achievements = model outputs/actions

Ideal life state = optimized model output

RLHF Stages Mapped to Life Stages

Pretraining (0‑18 years) : Your foundational identity is formed by what you saw, how you were treated, and the feedback you received—large‑scale, generic data from family, society, media, and education.

Fine‑tuning (18‑28 years) : Starting in university you selectively absorb data, develop a personal style, then enter the workforce, creating value and sustaining life while still relying on pretraining biases.

Reward‑model stage (28‑35 years) : Personal goals and definitions of success, happiness, and fulfillment emerge, forming an internal reward system that guides future reinforcement.

Reinforcement‑learning stage (35 years onward) : Behaviors and feedback form a closed loop; you fine‑tune long‑term strategies, habits, and value realization rather than constantly rebuilding from new data.

True “life optimization” depends on a precise reward function, not more data. Without conscious intervention in the early stages, the later reinforcement loop will cement a familiar but less free version of yourself.

How Individuals Can Control Their Life‑Training Process

Periodically review your pretraining corpus : Ask, “Whose voice is the first one I hear when I make a decision?” Identify early‑internalized language and emotional rules.

Select high‑quality fine‑tuning data : Ask, “Are the people around me giving genuine feedback or just rewarding illusion?” Seek out those who provide honest, growth‑oriented input.

Actively construct your own reward function : Write down personal principles, evaluate daily actions against them, and iterate the “version” of your reward model.

Stop being reinforced by outdated feedback: Recognize when you are over‑fitting to past patterns and deliberately reset your parameters.