Beyond Average Accuracy: Representational Equality in Cross-Country LLM Value Simulation

Introduction

Assigning demographic profiles such as country, age, and gender to large language models (LLMs) and prompting them to answer social survey questionnaires has become a key method for synthetic respondents in computational social science. As model capabilities grow, researchers increasingly question the reliability, applicability boundaries, and group-level disparities of LLM-based social simulations. A critical issue arises: when LLMs represent different populations, do all groups receive equally reliable representations?

Prior work shows LLMs can reproduce overall attitude distributions and experimental effects, but this ability varies across groups, questions, and analysis levels. Even when a model matches human responses on average, it may systematically deviate for specific subpopulations. Therefore, average accuracy alone cannot fully judge a simulation system's capacity across diverse groups. This problem is especially acute in cross-country simulation because training corpora exhibit highly imbalanced language coverage, web data visibility, and cultural knowledge distribution. Models tend to reflect WEIRD (Western, Educated, Industrialized, Rich, Democratic) societies' viewpoints and value patterns.

Study Design and Representational Equality Metrics

Researchers from Tsinghua University, Fudan University, Nankai University, and Shanghai Jiao Tong University conducted a systematic evaluation of 9 open-source LLMs using World Values Survey (WVS) Wave 7 data. The evaluation covers 59 countries, 160 comparable value questions, 11 value dimensions, and 2,420 comparable sub-group units . The paper, titled "Representational Equality in Cross-country Value Simulation: A Systematic Analysis of Large Language Models," was accepted by Computational Linguistics in July 2026 (DOI: 10.1162/COLI.a.648).

Simulation Accuracy

For each question, the Jensen–Shannon divergence (JSD) measures the difference between the model's predicted answer distribution and the real survey distribution. Simulation accuracy is defined as 1 − JSD. Accuracy is first averaged within each of the 11 value dimensions, then dimensions are equally weighted to avoid dominance by dimensions with more questions, yielding a per-country simulation accuracy score.

Representational Equality

After obtaining per-country accuracy scores, the study uses the coefficient of variation (CV) of these accuracies as the primary representational equality indicator. Lower CV indicates smaller dispersion across countries, i.e., higher cross-country representational equality. Robustness checks with range, max-min ratio, and Gini coefficient yield highly consistent patterns.

Accuracy–Equality (AE) Composite Metric

Because a model could have high equality simply by performing poorly everywhere, the authors construct an AE metric that geometrically averages the mean simulation error (1 − mean accuracy) and the CV. Lower AE indicates both lower overall error and higher cross-country equality.

Baseline Results: Divergent Model Rankings on Accuracy vs. Equality

Across the 9 models, average simulation accuracy and cross-country representational equality produce different rankings:

GLM-4-9B achieves the highest average accuracy but exhibits relatively low representational equality.

ChatGLM3-6B does not lead in average accuracy yet attains the highest representational equality among the 9 models and obtains the best AE composite score.

Within some model families, increasing parameter size raises average accuracy but does not consistently improve cross-country representational equality.

This demonstrates that average performance and cross-country representational equality must be examined separately and jointly; overall leadership does not guarantee equally stable simulation capability across nations.

Cross-country differences show a clear regional pattern: Australia, North America, and European countries are generally simulated more accurately, while Middle Eastern countries such as Egypt, Jordan, and Iraq consistently exhibit larger simulation biases.

Which Countries Are Easier to Simulate Accurately?

To understand the cross-country disparities, the study correlates per-country simulation accuracy with economic, technological, governance, and cultural indicators. Across most models:

GDP per capita, internet penetration rate, and Global Innovation Index (GII) show positive correlations with simulation accuracy.

World Bank governance indicators (six dimensions) are also positively correlated; higher governance quality associates with higher accuracy.

Cultural dimensions reveal systematic associations: countries with lower power distance, higher individualism, and higher indulgence are simulated more accurately; other cultural dimensions show weaker or model-dependent links.

These results indicate that representational inequality in cross-country simulation has systematic structural correlates, forming a stable cross-national pattern tied to economic development, digital infrastructure, governance quality, and cultural characteristics.

Case Study: Asymmetric Convergence Between the US and Mexico

On the question "How worried are you about unemployment or not finding a job?" (Q142), real Mexican respondents concentrate on "very worried," while US respondents choose this option far less. However, Llama-3-8B predicts similar distributions for both countries, both closer to the real US distribution. Extending to all 160 questions, about two-thirds of the time the Mexican prediction is closer to the real US distribution than to the real Mexican distribution, even when the true distributions differ substantially.

This phenomenon aligns with the group heterogeneity compression problem: models fail to preserve original inter-group value differences, causing simulations to homogenize. The US–Mexico example suggests this homogenization may be asymmetric convergence : weakly represented groups' predictions over-converge toward the better-represented group's value patterns. Thus, representational inequality manifests not only as accuracy gaps but also as insufficient capture of unique value patterns of underrepresented groups.

Intervention Strategies: Do Accuracy Gains Translate to Equality Gains?

After observing significant cross-country representational disparities, the study evaluates two major improvement pathways within a unified framework: context adaptation (native language prompting, relevant group information supplementation) and model training (continued post-training, preference alignment). Each experiment assesses average accuracy, cross-country representational equality (CV), and AE composite metric.

Native Language Prompting

Replacing English prompts with Chinese, Arabic, or Spanish improves simulation accuracy for many language–model combinations, but gains vary markedly across models and language groups. For some models, benefits are balanced across language groups, raising or maintaining equality; for others, gains concentrate in specific language groups, sometimes replacing the previously best-simulated group without substantially reducing inequality. Hence, native language prompting improves target language groups but does not consistently enhance cross-country representational equality.

Relevant Group Information Supplementation

Augmenting minimal demographic profiles with additional context and value information relevant to the target group improves both average accuracy and cross-country representational equality in most model and retrieval settings. However, effectiveness depends on retrieval method: direct retrieval sometimes corrects existing biases more clearly, while model-rewritten query expansion may reintroduce the model's own priors, weakening equality gains. Moreover, this strategy relies on available external information; groups with scarce digital traces may have limited supplementary knowledge.

Continued Post-Training

Continued post-training on Arabic, Japanese, Russian, Serbian, Thai, and Turkish data raises average simulation accuracy for the corresponding language groups, but country- and question-level changes vary widely. For Arabic countries, relative accuracy changes range from −1.2% to +3.4%. This shows that adding target-language data improves overall simulation capability for that language group but does not guarantee simultaneous improvement in cross-country representational equality.

Preference Alignment

Using DPO and GRPO with both human-annotated and GPT-annotated preference data yields no consistent directional changes in accuracy or representational equality relative to base models. Human-annotated preferences tend to preserve simulation accuracy better, but this advantage does not stably extend to representational equality.

Conclusion: From Average Accuracy to Representational Equality

As LLMs are increasingly deployed to simulate public opinions, values, and social behaviors, evaluating social simulation capacity solely by average accuracy is insufficient. High overall similarity to human responses does not imply equal coverage across countries and groups; likewise, an intervention that boosts overall accuracy may concentrate benefits in specific populations, leaving cross-country representational equality unchanged or even reduced.

The study introduces representational equality to emphasize that alongside "how accurately the model simulates," we must examine how this simulation capability distributes across populations. In cross-country value simulation, this means assessing both the overall fit to each country's true response distribution and the comparability of simulation accuracy across countries.

As synthetic respondents enter pilot surveys, experimental design, and cross-cultural social simulation, building more reliable LLM-based social simulation systems requires simultaneously improving overall simulation capability and reducing systematic disparities in that capability across populations. Incorporating representational equality alongside accuracy into the core evaluation framework enables a more complete assessment of models and their improvement methods, providing a new perspective for constructing globally representative LLM social simulation systems.