Why OpenAI Is Building a New Indian Language Benchmark (IndQA) and What It Means for AI

English speakers account for over 41% of the global population, yet many large‑scale language model AI benchmarks fail to meet this demand.

For example, MMMLU (Multilingual Massive Multitask Language Understanding) has become saturated, with top‑ranking models clustering near high scores, which OpenAI says prevents them from reflecting genuine progress.

OpenAI notes that current multilingual benchmarks focus on translation and multiple‑choice tasks and do not accurately measure a model’s grasp of regional context, culture, and history.

To address these gaps, OpenAI is developing new benchmarks for languages and regions worldwide, starting with its second‑largest market—India.

The new benchmark, IndQA, will "evaluate AI models' understanding and reasoning on important questions in Indian languages across a broad range of cultural domains."

India has 22 official languages, with seven languages spoken by at least 50 million people. IndQA contains 2,278 questions covering 12 languages and 10 cultural domains, created with the help of 261 Indian experts—including journalists, linguists, scholars, artists, and industry professionals.

The covered languages are Bengali, English, Hindi, Hinglish (a mix of English and Hindi), Kannada, Marathi, Oriya, Telugu, Gujarati, Malayalam, Punjabi, and Tamil. Including Hinglish addresses code‑switching phenomena in conversation.

The cultural domains span architecture & design, arts & culture, daily life, food & cooking, history, law & ethics, literature & linguistics, media & entertainment, religion & spirituality, and sports & leisure.

According to OpenAI, each data point includes a culturally grounded prompt in an Indian language, an English translation for review, a scoring rubric, and the expected answer from domain experts.

OpenAI plans to use IndQA as inspiration to create similar benchmarks for other regions worldwide.

As OpenAI states, "IndQA‑style questions are especially valuable for language or cultural areas that existing AI benchmarks inadequately cover. Creating similar benchmarks can help AI research labs better understand the languages and domains models currently struggle with, providing new directions for future improvements."