Unlocking LLM Reasoning: A Deep Dive into Post‑Training Techniques

1. Fine‑Tuning: Directed Model Evolution

Fine‑tuning updates all model parameters on downstream data (full fine‑tuning) but becomes inefficient as model size grows. Parameter‑efficient fine‑tuning (PEFT) freezes most pretrained weights and trains a small set of additional parameters.

1.1 Full Parameter Fine‑Tuning

Updates every weight of the pretrained model to adapt it to a specific task.

1.2 PEFT Techniques

LoRA : Introduces low‑rank trainable matrices \(A\) and \(B\) while keeping the original weight \(W\) fixed. Only \(A\) and \(B\) are optimized, drastically reducing trainable parameters.

AdaLoRA : Dynamically adjusts the rank of each layer based on importance metrics such as gradient norm.

QLoRA : Combines 4‑bit quantization of the pretrained weights with LoRA, cutting GPU memory usage by up to 70%.

Delta‑LoRA : Adds a momentum mechanism to LoRA updates for more stable fine‑tuning.

1.2.1 LoRA Pseudocode

import torch
import torch.nn as nn

class LoRALayer(nn.Module):
    def __init__(self, in_dim, out_dim, rank):
        super().__init__()
        self.A = nn.Parameter(torch.randn(in_dim, rank))
        self.B = nn.Parameter(torch.zeros(rank, out_dim))
        self.rank = rank  # rank size
    def forward(self, x):
        return x @ (self.A @ self.B)  # low‑rank matrix product

1.2.2 Prompt‑Tuning Techniques

Prefix‑Tuning : Inserts trainable vectors at the beginning of each layer.

P‑Tuning v2 : Learns hierarchical prompt positions across layers.

Prompt‑Tuning : Trains only input‑layer prompts, offering low parameter cost.

1.3 Domain‑Adaptive Fine‑Tuning

Adapts a pretrained model to specific domains (e.g., medical QA) by mixing generic instruction data with domain‑specific literature and fine‑tuning only selected layers to avoid catastrophic forgetting.

2. Reinforcement Learning: From Alignment to Reasoning

2.1 Reward Modeling

Trains a reward model to predict human preference scores for model outputs, using large amounts of preference data.

2.2 Bradley‑Terry Model

Models the probability that a human prefers output \(i\) over \(j\) as \(P(i \succ j) = \frac{e^{r_i}}{e^{r_i}+e^{r_j}}\), where \(r_i\) and \(r_j\) are reward scores.

2.3 Process vs. Outcome Rewards

Process Reward : Provides dense feedback at each generation step (syntax, coherence, factuality).

Outcome Reward : Evaluates only the final result (e.g., correct answer, passing unit tests).

2.4 Process Reward Example

def calculate_step_reward(response):
    # 1. Syntax check
    syntax = check_syntax(response)
    # 2. Coherence evaluation
    coherence = model.predict_coherence(response)
    # 3. Fact consistency
    fact_check = retrieve_evidence(response)
    return 0.3*syntax + 0.5*coherence + 0.2*fact_check

2.5 Reinforcement‑Learning Reasoning Enhancements

Tree‑of‑Thought (ToT) framework generates candidate thoughts, evaluates them with a value function, expands top‑k candidates, and back‑propagates accumulated rewards.

3. Test‑Time Extensions: Reasoning as Search

3.1 Main Inference‑Enhancement Techniques

Includes dynamic compute allocation, verifier‑augmented reasoning, and hybrid search strategies.

3.2 Dynamic Compute Allocation

Allocates resources based on estimated problem difficulty.

def dynamic_compute_allocation(query):
    difficulty = estimate_difficulty(query)
    if difficulty < 0.3:
        return greedy_decode()
    elif 0.3 <= difficulty < 0.7:
        return beam_search(width=3)
    else:
        return monte_carlo_tree_search(depth=5)

3.3 Verifier‑Enhanced Reasoning

Combines multiple validators (syntax, logical, factual, safety) and aggregates their scores multiplicatively to obtain a final confidence score.

4. Challenges and Future Directions

4.1 Current Bottlenecks

Reward Hacking : Models over‑optimize proxy metrics, producing misleading or harmful outputs.

Long‑Range Reasoning : Maintaining coherence over >128k tokens is computationally expensive and prone to forgetting.

Personalized Safety : Balancing user‑specific preferences with universal safety constraints.

4.2 Emerging Research

Meta‑Cognition : Enables models to assess their own uncertainty and allocate more compute when needed.

Physical Reasoning Fusion : Integrates symbolic physics rules with neural networks for accurate physical inference.

Swarm Intelligence : Coordinates multiple specialized agents (task decomposition, result integration) to solve complex problems.

5. Practical Guide: Choosing a Post‑Training Strategy

A decision flowchart (omitted here) helps practitioners select between full fine‑tuning, PEFT variants, prompt‑tuning, domain adaptation, or RL‑based alignment based on data availability, compute budget, and target application.