Unlocking CBLUE: A Deep Dive into China’s Biomedical Language Benchmark

Introduction

With the rapid development of artificial intelligence (AI) technologies, researchers are increasingly focusing on AI applications in medical health, and a key step is establishing standard datasets and scientific evaluation frameworks. The Chinese Biomedical Language Understanding Evaluation Benchmark (CBLUE) was launched in April, covering eight classic medical natural language understanding tasks and quickly attracted over 100 participating teams.

Task Overview

CBLUE comprises four major categories of medical NLP tasks: medical information extraction, medical term normalization, medical text classification, and medical retrieval & QA. It provides real‑world data and a unified evaluation protocol to encourage research on model generalization.

Below are brief introductions of each sub‑task:

(1) Medical Information Extraction:

CMeEE (Chinese Medical Entity Extraction): entity recognition for terms such as diseases, drugs, examinations, focusing on pediatric diseases.

CMeIE (Chinese Medical Information Extraction): relation extraction between entities, e.g., linking “rheumatoid arthritis” with “joint tenderness” as a disease‑examination relation.

(2) Medical Term Normalization:

CHIP‑CDN (Clinical Diagnosis Normalization): maps diverse clinical expressions to standard codes (e.g., ICD), supporting insurance settlement and DRG systems.

(3) Medical Text Classification:

CHIP‑CTC (Clinical Trial Criterion Classification): classifies clinical trial eligibility criteria to automate participant screening.

KUAKE‑QIC (Query Intention Classification): identifies user intent in medical search queries to improve result relevance.

(4) Medical Retrieval and QA:

CHIP‑STS (Semantic Textual Similarity): determines semantic similarity between medical question pairs.

KUAKE‑QTR (Query/Title Relevance): matches user queries with page titles in search scenarios.

KUAKE‑QQR (Query/Query Relevance): assesses relevance between two search queries.

Task Characteristics

Key features of the eight CBLUE tasks include:

Data anonymization and privacy protection through thorough manual checks.

Diverse data sources ranging from textbooks and expert guidelines to real clinical trial registries and online medical Q&A.

Real‑world distribution with noise and long‑tail patterns, challenging model robustness and generalization.

Methodology

Eleven popular Chinese pretrained language models were evaluated as baselines, including BERT‑base, BERT‑wwm‑ext‑base, RoBERTa‑large, RoBERTa‑wwm‑ext, ALBERT‑tiny/xxlarge, ZEN, Mac‑BERT, and the medical‑specific PCL‑MedBERT.

BERT‑base: 12 layers, 768‑dim hidden size, 110M parameters.

BERT‑wwm‑ext‑base: whole‑word masking Chinese BERT.

RoBERTa‑large: removes next‑sentence prediction and uses dynamic masking.

RoBERTa‑wwm‑ext‑base/large: combines RoBERTa and whole‑word masking.

ALBERT‑tiny/xxlarge: shares parameters across layers, trained on MLM and SOP.

ZEN: n‑gram enhanced Chinese encoder.

Mac‑BERT: improved BERT with corrected MLM pre‑training.

PCL‑MedBERT: medical‑domain pretrained model from Pengcheng Lab.

Performance Evaluation & Analysis

The baseline results of the 11 models on CBLUE are shown below:

Larger models generally achieve better performance, yet tasks such as CTC, QIC, QTR, and QQR do not always benefit from whole‑word masking, indicating the challenges of CBLUE. Notably, ALBERT‑tiny matches baseline models on several tasks, showing that compact models can be effective. The medical‑specific PCL‑MedBERT underperforms expectations, further confirming the benchmark’s difficulty.

Conclusion

CBLUE aims to provide researchers with open, real‑world data and multi‑task settings to promote model generalization in medical AI. The publicly released baseline code (https://github.com/CBLUEbenchmark/CBLUE) is intended to accelerate progress in the Chinese medical NLP community.