Quality Control for Online Medical Consultations Using NLP and Knowledge Graphs at JD Health

Preface

Accelerated by the COVID‑19 pandemic, the internet medical industry has grown rapidly, but concerns remain about the standardization, process compliance, result relevance, and quality assurance of online consultations. As algorithm engineers in the field, the authors explore how to alleviate public doubts.

Background

China now has 900 internet hospitals and regulatory platforms in 30 provinces. Quality and safety are critical, and national policies such as the 2020 Yinchuan Internet Diagnosis Service Specification and recent government work reports emphasize the need for a pre‑, mid‑, and post‑process quality assurance system for online medical services.

Industry Pain Points

Process and content quality issues in online diagnosis, including incomplete or unreasonable diagnoses and medication risks.

Low fault tolerance; minor physician errors can trigger disputes and damage reputation.

Difficulty evaluating doctor performance due to fragmented dialogue, diagnosis, and patient satisfaction data.

High review cost because medical queries are free‑form, oral, and require complex pathological reasoning.

Solution

The JD Health technology product team uses natural language processing and knowledge graphs to identify and analyze doctor‑patient dialogue, classifying risks into process quality control and content quality control. Severe issues trigger an alarm module for immediate review, while minor or no‑issue cases go to manual sampling. Manual review results become training data for continuous model improvement.

Figure: Consultation Quality Control Process Diagram

The information extraction module consists of five functions:

Question identification: Detect whether a sentence is a question or statement.

Question matching: Link dialogue context to determine the status of doctor inquiry slots.

NER: Tag entities such as diseases, symptoms, and duration.

Relation extraction: Use a pipeline to infer relationships between identified entities.

Topic segmentation: Apply unsupervised LDA, text classification, and rule engines to extract key terms for each process.

Process Quality Control

Figure: Process Quality Control

The model checks whether the dialogue includes the necessary consultation steps, clearly informs the patient of a tentative diagnosis, records allergy, pregnancy, or other special conditions before prescribing, and provides proper medication usage instructions.

Content (In‑Depth) Quality Control

Using a knowledge graph and business rules, the system evaluates the rationality of a doctor’s diagnosis and prescription. After extracting patient symptoms, disease course, and personal information, candidate diseases are recalled from the knowledge graph; if the doctor’s diagnosis is not among them, a risk warning is issued.

Figure: Content Quality Control

Practical Implementation

The quality‑control service now covers all text‑based consultations in JD’s internet hospital. It achieves a recall rate above 99% and a precision of 90% for risky cases. Previously, a medical expert reviewed about 1,000 cases per week; with the human‑machine collaborative mode (automatic QC + expert review), efficiency has increased tenfold, reducing the number of cases needing manual review to one‑tenth of the original volume. Ongoing data accumulation will further improve system metrics and lower marginal costs of user growth.

Figure: Evaluation Scores by Primary Department

Future Directions

Future work will add report interpretation, image recognition, and other functions to parse every detail of “text‑image” consultations. As algorithms and the knowledge graph evolve, in‑consultation assistance and medication recommendation will be introduced, complementing existing intelligent triage and pre‑consultation modules to form an end‑to‑end, lifecycle‑wide online medical decision‑support system.