What Is Human‑AI Alignment? A New Framework from NeurIPS 2025

1. Why Alignment Again?

Large models improve each year, but alignment problems appear like whack‑a‑mole: suppressing harmful output leads to over‑rejection of normal requests, and a new RLHF SOTA quickly gets “broken” by users.

When harmful output is blocked, the model starts “over‑rejecting” benign queries.

After a RLHF breakthrough, the model is often “played‑out” within a week of deployment.

The root cause is treating alignment as a static, one‑way “fixing AI” problem. In reality, AI behavior → human feedback → AI iteration forms a dynamic, two‑way loop.

Alignment objectives must be pluralistic . Humans must retain “voice” over data, training, evaluation, and deployment pipelines. Alignment outcomes need to be quantified and overseen within a socio‑technical system.

2. Introduction: One‑Slide Overview of the HAA Framework

The tutorial visualizes the Human‑AI Alignment (HAA) framework, highlighting why humans must dominate the alignment process.

3. Foundations: Pluralistic Values, Ethics, and Norms

The framework breaks human values into multidimensional vectors such as morality, norms, culture, and law, and surveys classification systems, representative value theories, datasets, and validation methods needed for pluralistic alignment.

Foundations : Decompose “human values” into moral, normative, cultural, legal vectors.

Methods : Humans can intervene during data annotation, prompt design, RLHF, and inference.

Practice : After deployment, continuously monitor model impact on group behavior, social networks, and policy.

Challenges : Dynamic evolution, safety‑performance trade‑offs, deceptive alignment, multi‑agent games, etc.

4. Methods: Human Technical Specs and Alignment Techniques

Specific stages where humans can intervene are illustrated with representative papers and techniques.

Data Specification : Interactive Constitution Generator (ConstitutionMaker) – converts user feedback into a “constitution”.

Training : Jury Learning – a “jury” debates internally before voting for labels.

Inference : Meta‑prompting – the model asks itself “what does the user really want?”.

Evaluation : Chatbot Arena – real‑time Elo scores from blind 1‑v‑1 human tests.

5. Practice: Socio‑Technical Assessment and Oversight

The tutorial explores the cascading social effects of AI alignment, emphasizing safety‑focused alignment, interpretability, controllability, and supervision mechanisms. It also discusses large‑model simulation of societal impact and the need for customized value plugins for different user groups (teachers, doctors, game modders).

6. Challenges: Emerging Issues and Future Directions

Key open problems include dynamic‑evolutionary alignment, deceptive or “masked” alignment, and alignment in multi‑agent systems.

Dynamic evolutionary alignment.

Deceptive alignment and “pretend” alignment.

Alignment of AI agents within multi‑agent ecosystems.

https://hai-alignment-course.github.io/tutorial/