Build a Full End‑to‑End Embodied AI Workflow with Isaac Lab Arena

Overview

The notebook demonstrates a full end‑to‑end workflow for embodied AI using Isaac Lab Arena , covering environment setup, data generation, policy fine‑tuning with GR00T‑N1.5 , and closed‑loop evaluation on a G1 robot box‑pick‑and‑place task.

1. Environment Setup and Resource Preparation

Start a PAI DSW instance and configure the following resources:

Docker image (private):

dsw-registry-vpc.${regionId}.cr.aliyuncs.com/pai-training-algorithm/isaac-sim:isaaclab-arena-gr00t-vnc-v3-20260307

Docker image (public):

dsw-registry.${regionId}.cr.aliyuncs.com/pai-training-algorithm/isaac-sim:isaaclab-arena-gr00t-vnc-v3-20260307

Instance type: ecs.gn8is.2xlarge (48 GB GPU, 8 CPU cores, 128 GB RAM)

Configure VPC endpoints for OSS access:

Internal endpoint: oss-${regionId}-internal.aliyuncs.com External endpoint:

oss-${regionId}.aliyuncs.com

Dataset and Model Resources

Small test dataset:

oss://pai-vision-data-${oss-region}/aigc-data/isaac/nb13/datasets/isaaclab_arena/locomanipulation_tutorial/arena_g1_loco_manipulation_dataset_generated_small.hdf5

Annotated human demonstration data: ...arena_g1_loco_manipulation_dataset_annotated.hdf5 Mimic‑augmented dataset (~21 GB): ...arena_g1_loco_manipulation_dataset_generated.hdf5 Converted LeRobot data: ...arena_g1_loco_manipulation_dataset_generated.zip GR00T‑N1.5 fine‑tuned checkpoint:

oss://pai-vision-data-${oss-region}/aigc-data/isaac/nb13/models/isaaclab_arena/locomanipulation_tutorial/checkpoint-20000.zip

2. Environment Validation

Run the provided verification cell to ensure that Isaac Sim , Isaac Lab Arena , Mimic , and GR00T are correctly installed. Set the following environment variables:

DATASET_DIR=/datasets/isaaclab_arena/locomanipulation_tutorial
MODELS_DIR=/models/isaaclab_arena/locomanipulation_tutorial

3. OSS Download Helper

A Python helper selects the appropriate internal OSS endpoint based on the DSW region:

def download_from_oss(url, filename, save_dir):
    url_prefix = {
        "cn-shanghai": "http://pai-vision-data-sh.oss-cn-shanghai-internal.aliyuncs.com",
        "cn-hangzhou": "http://pai-vision-data-hz2.oss-cn-hangzhou-internal.aliyuncs.com",
        "cn-shenzhen": "http://pai-vision-data-sz.oss-cn-shenzhen-internal.aliyuncs.com",
        "cn-beijing": "http://pai-vision-data-bj.oss-cn-beijing-internal.aliyuncs.com",
        "ap-southeast-1": "http://pai-vision-data-ap-southeast.oss-ap-southeast-1-internal.aliyuncs.com",
        "cn-wulanchabu": "http://pai-vision-data-wlcb.oss-cn-wulanchabu-internal.aliyuncs.com",
    }
    dsw_region = os.environ.get("dsw_region")
    prefix = url_prefix.get(dsw_region, "http://pai-vision-data-sh.oss-cn-shanghai.aliyuncs.com")
    full_url = os.path.join(prefix, url, quote(filename))
    # download logic omitted for brevity

4. Data Generation Workflow

4.1 Download Small Test Dataset

download_from_oss(
    "aigc-data/isaac/nb13/datasets/isaaclab_arena/locomanipulation_tutorial",
    "arena_g1_loco_manipulation_dataset_generated_small.hdf5",
    DATASET_DIR)

4.2 Download Annotated Human Demonstrations

download_from_oss(
    "aigc-data/isaac/nb13/datasets/isaaclab_arena/locomanipulation_tutorial",
    "arena_g1_loco_manipulation_dataset_annotated.hdf5",
    DATASET_DIR)

4.3 Mimic Data Augmentation

Generate 100 augmented trajectories (≈1 hour) using the Mimic module:

# Generate 100 demonstration trajectories
!/isaac-sim/python.sh isaaclab_arena/scripts/generate_dataset.py \
  --headless \
  --enable_cameras \
  --mimic \
  --input_file $DATASET_DIR/arena_g1_loco_manipulation_dataset_annotated.hdf5 \
  --output_file $DATASET_DIR/arena_g1_loco_manipulation_dataset_generated.hdf5 \
  --generation_num_trials 100 \
  --device cpu \
  galileo_g1_locomanip_pick_and_place \
  --object brown_box \
  --embodiment g1_wbc_pink

Key flags: --mimic: enable Mimic augmentation --input_file: path to human demonstrations --output_file: path for augmented data --generation_num_trials 100: number of synthetic trajectories

4.4 (Optional) Replay Augmented Data

!/isaac-sim/python.sh isaaclab_arena/scripts/replay_demos.py \
  --headless \
  --device cpu \
  --enable_cameras \
  --dataset_file $DATASET_DIR/arena_g1_loco_manipulation_dataset_generated.hdf5 \
  galileo_g1_locomanip_pick_and_place \
  --object brown_box \
  --embodiment g1_wbc_pink

5. GR00T‑N1.5 Fine‑Tuning

Optionally download a pre‑generated checkpoint to skip training:

download_from_oss(
    "aigc-data/isaac/nb13/models/isaaclab_arena/locomanipulation_tutorial",
    "checkpoint-20000.zip",
    MODELS_DIR)

Run the fine‑tuning script (quick‑validation parameters shown):

!/isaac-sim/python.sh isaaclab_arena/examples/policy_runner.py \
  --headless \
  --policy_type gr00t_closedloop \
  --policy_config_yaml_path isaaclab_arena_gr00t/g1_locomanip_gr00t_closedloop_config.yaml \
  --num_steps 1200 \
  --enable_cameras \
  galileo_g1_locomanip_pick_and_place \
  --object brown_box \
  --embodiment g1_wbc_joint

Important arguments: --policy_type gr00t_closedloop: use the GR00T closed‑loop policy --num_steps 1200: number of simulation steps --enable_cameras: render camera images

Remove --headless when you want to view the GUI via VNC (default password 123456 ).

6. Closed‑Loop Evaluation

6.1 Single‑Environment Evaluation (GUI)

!/isaac-sim/python.sh isaaclab_arena/examples/policy_runner.py \
  --headless \
  --policy_type gr00t_closedloop \
  --policy_config_yaml_path isaaclab_arena_gr00t/g1_locomanip_gr00t_closedloop_config.yaml \
  --num_steps 1200 \
  --enable_cameras \
  galileo_g1_locomanip_pick_and_place \
  --object brown_box \
  --embodiment g1_wbc_joint

Set --headless to False (or omit it) to watch the robot move through VNC.

6.2 Parallel‑Environment Evaluation (Optional)

!/isaac-sim/python.sh isaaclab_arena/examples/policy_runner.py \
  --headless \
  --policy_type gr00t_closedloop \
  --policy_config_yaml_path isaaclab_arena_gr00t/g1_locomanip_gr00t_closedloop_config.yaml \
  --num_steps 1200 \
  --num_envs 5 \
  --enable_cameras \
  --device cpu \
  --policy_device cuda \
  galileo_g1_locomanip_pick_and_place \
  --object brown_box \
  --embodiment g1_wbc_joint

This runs five simulations in parallel, improving statistical significance and throughput.

7. Training Process Analysis

Use TensorBoard to monitor loss curves and success rates. Typical observations:

Loss decreases smoothly over the first 1 000 iterations.

Evaluation success rate stabilizes, confirming the effectiveness of Mimic augmentation and GR00T fine‑tuning.

8. Summary

PAI fully supports the NVIDIA Isaac toolchain. This notebook showcases a complete pipeline—from scene construction, data augmentation, and model fine‑tuning to closed‑loop evaluation—executed entirely within a PAI‑DSW instance without external environment switches.