Embodied AI Resources: Datasets, Modeling, Papers (Nvidia, ByteDance, Xiaomi)

Embodied AI Overview

Embodied AI extends AI from perception‑generation to agents that sense, decide, and act in the physical world, forming a closed perception‑decision‑action loop. Major technology companies and research labs have highlighted the need for large datasets, simulation environments, benchmark tasks, and systematic methods to advance this field.

Dataset Recommendations

BC‑Z Robot Learning Dataset – Size: 32.28 GB. Download: https://go.hyper.ai/vkRel. Developed by Google, Everyday Robots, UC Berkeley, and Stanford. Contains 25,877 task scenes covering 100 manipulation tasks, collected from 12 robots and 7 operators (125 h total). Supports training a 7‑DoF multi‑task policy conditioned on language descriptions or demonstration videos.

DexGraspVLA Robot Grasping Dataset – Size: 7.29 GB. Download: https://go.hyper.ai/G37zQ. Created by the Psi‑Robot team. Includes 51 human‑demonstration samples for high‑success‑rate grasping in cluttered, unseen environments, using a pre‑trained vision‑language model for high‑level planning and a diffusion‑based policy for low‑level control.

EgoThink First‑Person Visual Question‑Answering Benchmark – Size: 865.29 MB. Download: https://go.hyper.ai/5PsDP. Built from 700 images sampled from Ego4D, evaluates six core abilities across 12 dimensions to assess VLM performance on first‑person tasks.

EQA (Embodied Question Answering) – Size: 839.6 KB. Download: https://go.hyper.ai/8Uv1o. Based on House3D; agents must navigate a simulated environment, gather visual evidence, and answer questions (e.g., “What color is the car?”), testing integrated perception, navigation, and reasoning.

OmniRetarget Full‑Body Robot Motion Retargeting Dataset – Size: 349.61 MB. Download: https://go.hyper.ai/IloBI. Collected by Amazon, MIT, and UC Berkeley. Provides ~4 h of motion trajectories for robot‑object, robot‑terrain, and robot‑object‑terrain interactions, with URDF/SDF/OBJ models for visualization.

Benchmark and Paper Summaries

RBench Video Generation Benchmark – Paper: “Rethinking Video Generation Model for the Embodied World” (Peking University & ByteDance Seed). Paper: https://go.hyper.ai/k1oMT. Covers five task domains and four robot morphologies; evaluates correctness, visual fidelity, structural consistency, physical realism, and motion completeness. Shows current video‑generation models struggle with physically plausible robot behavior; Spearman correlation with human evaluation is 0.96.

Being‑H0.5 – Paper: “Being‑H0.5: Scaling Human‑Centric Robot Learning for Cross‑Embodiment Generalization” (BeingBeyond). Paper: https://go.hyper.ai/pW24B. Introduces a Vision‑Language‑Action model that treats human interaction trajectories as a universal “mother tongue”. Releases UniHand‑2.0, a pre‑training dataset covering 30 robot forms and >35,000 h of multimodal data, with a unified action space that maps heterogeneous robot controls to semantically aligned action slots.

Fast‑ThinkAct – Paper: “Fast‑ThinkAct: Efficient Vision‑Language‑Action Reasoning via Verbalizable Latent Planning” (NVIDIA). Paper: https://go.hyper.ai/q1h7j. Distills latent chain‑of‑thought reasoning from a teacher model, achieving up to 89.3 % reduction in inference latency while preserving long‑horizon planning, few‑shot adaptation, and failure‑recovery capabilities.

JudgeRLVR – Paper: “Judge First, Generate Second for Efficient Reasoning” (Peking University & Xiaomi). Paper: https://go.hyper.ai/2yCxp. Two‑stage paradigm: first judges answer correctness, then fine‑tunes a generation model. Improves accuracy by ~3.7 pp (in‑domain) and ~4.5 pp (out‑of‑domain) and reduces generated response length by 42 %.

ACoT‑VLA – Paper: “ACoT‑VLA: Action Chain‑of‑Thought for Vision‑Language‑Action Models” (Beihang University & AgiBot). Paper: https://go.hyper.ai/2jMmY. Introduces explicit and implicit action reasoners (EAR and IAR) to construct structured action chains. Achieves 98.5 % on LIBERO, 84.1 % on LIBEROPlus, and 47.4 % on VLABench, demonstrating strong real‑world transfer.

Tutorials and Open‑Source Systems

HY‑World 1.5 (WorldPlay) – Open‑source real‑time interactive world‑modeling system released by Tencent’s Hunyuan team. Uses streaming video diffusion to maintain long‑term geometric consistency while balancing speed and memory. Online demo: https://go.hyper.ai/qsJVe.

vLLM + Open WebUI with Nemotron‑3 Nano – NVIDIA’s 30B‑parameter LLM designed for inference and reasoning tasks, suitable for building AI agents, chatbots, and retrieval‑augmented generation systems. Online demo: https://go.hyper.ai/6SK6n.