Cornell’s EMSeek Generates Insights from EM Images in 2–5 Minutes, 50× Faster Than Experts

Dataset covering breadth and difficulty

The authors constructed a benchmark dataset that reflects the current breadth and difficulty of electron‑microscopy analysis, containing thousands of pixel‑level annotated images from 20 material systems (perovskites, high‑entropy alloys, van‑der‑Waals heterostructures, single‑atom catalysts, etc.) and five typical tasks: atomic‑column localization, point‑defect annotation, nanoparticle contour extraction, irradiation‑induced defect counting, and single‑atom identification.

The dataset follows three selection criteria: (i) relevance to catalysis, energy storage, and semiconductor reliability; (ii) coverage of structural complexity from highly symmetric crystals to heavily defected or low‑contrast lattices; (iii) imaging diversity in accelerating voltage, dose, and detector mode, providing a strict test of model robustness.

EMSeek Framework: LLM‑driven task scheduling

Unlike pipelines that rely on a single deep‑learning model, EMSeek assigns subtasks to a hierarchy of specialized agents and uses a large language model (LLM) as a unified scheduler to plan, invoke, and execute each step, minimizing human intervention.

SegMentor

This core unit performs reference‑guided universal segmentation, producing atomic‑level and particle‑level masks across diverse materials and imaging conditions. Its backbone is Ref‑UNet, a lightweight U‑Net where the encoder blocks are replaced by a visual backbone; skip connections also carry learned embeddings of the user‑selected reference patch.

During forward propagation, the reference patch is tokenized, position‑encoded, and injected via cross‑attention layers that re‑weight channel responses based on patch similarity. The resulting context vector travels along the up‑sampling path, guiding pixel‑wise predictions toward features matching the reference while suppressing distractors.

CrystalForge (EM2CIF)

Operating under mask constraints, this module conducts reciprocal‑space search, combines database retrieval with candidate generation, and reconstructs crystal structures suitable for density‑functional‑theory (DFT) calculations, even for previously unseen chemical systems.

MatProphet

MatProphet employs a gated‑expert mixture‑of‑experts (MoE) model that fuses outputs from multiple atomic‑scale models. With only ~2 % labeled data for calibration, it predicts formation energies, defect energies, and associated uncertainties.

ScholarSeeker

This agent retrieves and synthesizes evidence from the literature, producing answers anchored by citations. It operates in three stages: literature retrieval via dense similarity search, evidence extraction with ranking and DOI/offset recording by a Guardian Agent, and reasoning where a Scribe Agent structures the argument and injects citations into the final report.

EMSeek accelerates materials research

On 20 material systems and five task categories, EMSeek achieves roughly twice the inference speed of the Segment Anything Model (SAM) while delivering higher accuracy. On the STEM2Mat dataset it reaches >90 % structural similarity, surpassing AtomAI and AutoMat across all difficulty levels. With only ~2 % annotated data, it matches or exceeds strong single‑expert models on three out‑of‑distribution property‑prediction benchmarks.

In a recent in‑situ TEM study, three experts required nearly 20 weeks to fully annotate 1,200 frames of irradiation defects. EMSeek processes the same data on four A100 GPUs in 146 ± 18 seconds— a 50‑fold speedup that compresses weeks of expert work into minutes, enabling near‑real‑time hypothesis testing and process optimization.

Future competition focus

The authors argue that AI is reshaping the "characterization‑understanding‑design" chain in materials science, reducing reliance on expert labor and enabling low‑data, low‑threshold intelligent characterization systems. They cite the ATOMIC framework from Duke and MIT as a complementary end‑to‑end approach, emphasizing that the next competitive edge will be the ability to efficiently schedule intelligent agents and knowledge resources.

References: "Bridging electron microscopy and materials analysis with an autonomous agentic platform" (Science Advances, 2023) – https://www.science.org/doi/10.1126/sciadv.aed0583.