Self-Improving Agent RAG: Multi-Agent Clinical Trial Design with Evolutionary Optimization
This article details building a self-improving Agent RAG system for clinical trial design, featuring a collaborative multi-agent workflow, a five-dimensional evaluation suite, and an evolutionary outer loop that uses Pareto frontier analysis to optimize standard operating procedures for better recruitment feasibility while maintaining scientific rigor.
Medical AI Knowledge Infrastructure
The article begins by establishing a production-grade knowledge base for a clinical trial design Agent RAG system. It emphasizes that data source quality directly determines output quality. The stack uses a local-first, open-source approach with LangChain, LangGraph, Ollama, FAISS, DuckDB, and specialized libraries (biopython for PubMed, pypdf for FDA PDFs).
Installing the Open-Source Stack
A single pip command installs all required packages: langchain, langgraph, ollama, duckdb, faiss-cpu, sentence-transformers, biopython, pypdf, beautifulsoup4, etc. Each library's role is explained (e.g., langgraph for agent workflows, duckdb for high-performance analytics on MIMIC data).
Environment Configuration & Observability
API keys are managed via a .env file and python-dotenv. LangSmith is configured early for deep observability of complex agent interactions. The code shows loading environment variables, validating required keys (LANGCHAIN_API_KEY, ENTREZ_EMAIL), and setting the LangSmith project name.
Configuring Local LLMs: A Tiered Model Strategy
Instead of a one-size-fits-all model, a dictionary llm_config assigns specific open-source models to distinct roles via Ollama:
Fast/Flexible (7B-8B): planner (Llama 3.1 8B Instruct, JSON mode), drafter (Qwen2 7B), sql_coder (Qwen2 7B) for frequent, well-defined tasks.
Deep/Strategic (70B): director (Llama 3 70B, JSON mode) for high-stakes diagnosis and SOP evolution.
Embeddings: nomic-embed-text for vector stores.
This tiered approach optimizes cost-performance: small models handle high-frequency tasks; the large model handles low-frequency, high-reasoning tasks.
Preparing Knowledge Stores
Four real-world data sources are ingested and indexed:
PubMed Literature: download_pubmed_articles uses Biopython's Entrez to search (e.g., "(SGLT2 inhibitor) AND (type 2 diabetes) AND (renal impairment)") and save 20 abstracts as text files.
FDA Guidelines: download_and_extract_text_from_pdf downloads a PDF (FDA diabetes guidance), saves the raw PDF, and extracts text via pypdf for RAG.
Ethics Principles: A curated summary of the Belmont Report principles (Respect for Persons, Beneficence, Justice) is saved as a text file.
MIMIC-III Clinical Data: load_real_mimic_data uses DuckDB to load gzipped CSVs (PATIENTS, DIAGNOSES_ICD, LABEVENTS) into a persistent database. A two-stage process handles the massive LABEVENTS table: load as text, filter for relevant ITEMIDs (creatinine 50912, HbA1c 50852) with regex validation, then cast to numeric types.
All unstructured text sources are indexed into separate FAISS vector stores via create_vector_store (load → split with RecursiveCharacterTextSplitter, chunk_size=1000, overlap=100 → embed → index). create_retrievers returns a dictionary of retrievers (pubmed_retriever k=3, fda_retriever k=3, ethics_retriever k=2) plus the MIMIC DuckDB path.
Building the Inner Trial Design Network
The core is a collaborative multi-agent workflow orchestrated by LangGraph, governed by a dynamic GuildSOP (Standard Operating Procedure) — the "genome" of the pipeline.
Defining the Team SOP (GuildSOP)
A Pydantic GuildSOP model defines tunable parameters: planner_prompt, researcher_retriever_k, synthesizer_prompt, synthesizer_model (Literal["qwen2:7b", "llama3.1:8b-instruct"]), and feature flags use_sql_analyst, use_ethics_specialist. A baseline SOP v1.0 is instantiated with detailed prompts and defaults.
Defining Expert Agents
GuildState(TypedDict) serves as the shared workspace: initial_request, plan, agent_outputs (list of AgentOutput), final_criteria, and the sop itself.
Planner Agent: Reads sop.planner_prompt, uses structured output to emit a JSON plan with tasks for Regulatory Specialist, Medical Researcher, Ethics Specialist, Patient Cohort Analyst.
Retrieval Agent (Generic): A reusable function retrieval_agent takes a task, state, retriever name, and agent name. It dynamically adjusts k for the Medical Researcher based on sop.researcher_retriever_k. Returns formatted findings with source metadata.
Patient Cohort Analyst: A text-to-SQL agent. It fetches the DuckDB schema, uses a detailed prompt with key mappings (T2DM → ICD9 '25000', renal impairment → creatinine 1.5-3.0, uncontrolled T2D → HbA1c >8.0), generates SQL via sql_coder (Qwen2 7B), executes it, and returns patient count. Includes try/except for robustness.
Criteria Synthesizer: Aggregates all agent_outputs into a briefing packet, dynamically selects the drafter model per sop.synthesizer_model, and produces the final Inclusion/Exclusion Criteria document.
Orchestrating with LangGraph
A StateGraph with three nodes: planner → execute_specialists (loops over plan tasks, routes to appropriate agent functions, respects SOP feature flags) → synthesizer → END. Compiled into guild_graph.
Full Test Run
Using baseline SOP v1.0 and a Phase II trial concept for Sotagliflozin, the graph runs end-to-end. Logs show planner generating a 4-task plan, each specialist executing (Regulatory: 3 docs, Medical: 3 docs, Ethics: 2 docs, Cohort Analyst: SQL query returning 59 patients), synthesizer (qwen2:7b) producing a structured criteria document.
Multi-Dimensional Evaluation System
Five evaluators produce a 5D performance vector:
Scientific Rigor (LLM Judge): director (Llama 3 70B) scores alignment with PubMed context (0-1).
Regulatory Compliance (LLM Judge): Same model scores adherence to FDA context.
Ethical Soundness (LLM Judge): Scores against Belmont principles.
Recruitment Feasibility (Programmatic): Parses cohort analyst's patient count, normalizes against ideal 150 (score = min(1.0, count/150)).
Operational Simplicity (Programmatic): Penalizes expensive test keywords (MRI, biopsy, etc.) — score = max(0, 1 - 0.5*count).
All evaluators return a GradedScore (score + reasoning). run_full_evaluation orchestrates them, returning an EvaluationResult.
Baseline evaluation reveals: Rigor 0.90, Compliance 0.95, Ethics 1.00, Feasibility 0.39, Simplicity 1.00. The critical weakness is feasibility (only 59 patients vs. 150 target).
Evolution Engine: The Outer Loop
An SOPGenePool class stores SOP versions with evaluations and parent lineage.
Director-Level Agents
Performance Diagnostician: Uses director to analyze the 5D vector, output a Diagnosis (primary_weakness from Literal, root_cause_analysis, recommendation).
SOP Architect: Uses director with GuildSOP.schema_json() in prompt to generate 2-3 mutated GuildSOP objects targeting the diagnosed weakness.
Evolution Cycle
run_evolution_cycletakes the latest SOP, runs diagnostician, generates candidates, tests each via full inner loop + evaluation, adds results to gene pool. One cycle on baseline SOP v1 produced:
SOP v2: Feasibility jumps to 0.81 (patient count 121), Rigor drops slightly to 0.85 (acceptable trade-off).
SOP v3: Tried increasing researcher k; feasibility unchanged at 0.39.
Pareto Frontier Analysis
identify_pareto_frontimplements Pareto dominance (numpy vector comparison). From the gene pool, SOP v1 and v2 are non-dominated; v3 is dominated by v1. The frontier presents two optimal strategies:
SOP v1: "Maximum Rigor" — prioritizes scientific purity, sacrifices feasibility.
SOP v2: "High Feasibility" — small rigor sacrifice for large feasibility gain.
Visualizations: parallel coordinates plot (5D trade-offs) and scatter plot (Rigor vs. Feasibility) make the trade-off explicit for human decision-makers.
Understanding the Cognitive Workflow
Workflow Timeline (Gantt Chart)
invoke_with_timinguses graph.stream() to capture per-node durations. For SOP v2: planner 5.2s, execute_specialists 15.5s (bottleneck, contains parallel sub-tasks), synthesizer 4.8s. Visualized as a Gantt chart.
Performance Radar Chart
Radar chart compares SOP v1 and v2 across all five dimensions, showing overlapping strengths (Compliance, Ethics, Simplicity) and the Rigor-Feasibility trade-off.
Future Directions
Run evolution for hundreds of generations to enrich Pareto frontier.
Distill director reasoning into a smaller, faster policy model.
Allow director to dynamically add/remove specialist agents (e.g., Biostatistician).
Replace static MIMIC-III with real-time EHR API access.
Enhance SOP Architect with crossover operators (combine best parts of two SOPs).
Integrate human expert feedback as a reward signal.
Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.
