Train a $399 Microduck Robot with Reinforcement Learning on Cloud GPUs

Community-Trained Motions for Microduck

Since Pollen Robotics opened pre-orders for the 25 cm, 15-servo Microduck at $399, community developers have uploaded numerous motion policies trained via reinforcement learning. These include running (~1.65 m/s by HannesVonEssen), jumping (tested on real hardware by joanfox), single-leg standing, ground object pickup, left/right kicking, forward rolls, bowing, moonwalk, and more. Policies are shared as ONNX files on Hugging Face and can be previewed in MuJoCo simulation without physical hardware.

Training Requirements: Simulation, Reward Functions, and RL Algorithms

Microduck motion training relies on three pillars: a MuJoCo simulation environment, carefully designed reward functions, and a reinforcement learning algorithm. For walking, the reward encourages balance and target velocity; for jumping, weights shift to foot clearance, body posture, and landing stability. The framework supports thousands of parallel simulation environments, necessitating GPU acceleration. GPU cloud instances are used in this tutorial, with recommended configurations:

Single basic motion: V100S 32GB — runs thousands of parallel environments, suitable for entry-level training.

Multi-parameter parallel tests: RTX 30/40 series or equivalent — faster single-card compute, good for frequent iteration.

Multi-task or large-scale simulation: A800, H800 or equivalent 80GB GPUs — suited for multi-card training and larger experiments.

Suggested instance spec: 1×32GB GPU, ~10 CPU cores, 32GB RAM, Ubuntu 20.04+, 20GB free disk, NVIDIA driver and CUDA pre-installed.

Step-by-Step Training on GPU Cloud Host

1. Install the Training Framework

# Install uv
curl -LsSf https://astral.sh/uv/install.sh | sh
source ~/.bashrc

# Install Python 3.12
uv python install 3.12

# Clone training framework
git clone https://github.com/pollen-robotics/microduck_rl
cd microduck_rl

# Install dependencies
export UV_HTTP_TIMEOUT=600
uv sync

First install downloads CUDA dependencies; allow time and disk space. Optionally log in to Weights & Biases and Hugging Face for experiment tracking and model publishing:

uv run wandb login
uv run hf auth login

Provide API keys via environment variables or interactive login; never hard-code in source or public logs.

2. Train a Walking Policy

The framework includes a basic walking task. Launch offline training (no W&B sync):

WANDB_MODE=offline uv run train \
  Mjlab-Velocity-Flat-MicroDuck \
  --env.scene.num-envs 4096

If logged into W&B, omit WANDB_MODE=offline. Terminal outputs iteration count, reward values, and GPU usage; rising then stabilizing rewards indicate convergence. If OOM occurs, reduce parallel environments (e.g., 2048). Basic walking typically converges in a few hours depending on GPU, parameters, and convergence criteria.

3. Export ONNX Policy

With W&B logging, export using the run path:

uv run scripts/export.py \
  Mjlab-Velocity-Flat-MicroDuck \
  --wandb-run-path <username>/<project>/<run_id>

For offline training, use a local checkpoint:

uv run scripts/export.py \
  Mjlab-Velocity-Flat-MicroDuck \
  --checkpoint-file <checkpoint_path>

This produces an ONNX policy file.

4. Validate in Simulation

Linux:

uv run scripts/infer_policy.py \
  --walking output.onnx \
  --new-cmd-obs

macOS (requires mjpython):

.venv/bin/mjpython scripts/infer_policy.py \
  --walking output.onnx \
  --new-cmd-obs

Add --record recording.pkl to save observation data. Validation focuses on stability, target motion adherence, and robustness to initial pose changes or external disturbances.

5. Publish to Hugging Face Hub

uv run publish \
  --onnx output.onnx \
  --repo <HF_username>/microduck-walk \
  --kind perpetual \
  --slot walk \
  --description "Microduck walking policy"

Others can download via:

uv run hf download \
  <HF_username>/microduck-walk \
  policy.onnx \
  --local-dir policies/my-walk

On physical Microduck, load with the provided control tool:

sudo robotctl policy load \
  walk <HF_username>/microduck-walk
robotctl robot do walk

Before real-robot deployment, verify policy compatibility with current hardware, firmware, and control interfaces; reserve safe space for low-speed testing.

Extending from Walking to Running

Training new motions requires more than raising target speed; reward functions and robustness parameters must be redesigned. For running, typical adjustments include:

Increase target speed and speed cap

Raise forward progress reward weight

Constrain action magnitude changes

Add random external forces

Randomize trunk or head center-of-mass offsets

Set varied initial tilt angles

These perturbations reduce overfitting to a single simulation condition and improve sim-to-real transfer. The community extension microduck-playground provides a running task example:

git clone https://github.com/Vottivott/microduck-playground.git
cd microduck-playground
export UV_HTTP_TIMEOUT=600
uv sync

MICRODUCK_RUNNING_TARGET_MAX_SPEED=2.2 \
MICRODUCK_RUNNING_SPEED_CAP=2.4 \
MICRODUCK_RUNNING_FORWARD_PROGRESS_WEIGHT=5.0 \
MICRODUCK_RUNNING_ROBUST_PUSH_MPS=0.10 \
MICRODUCK_RUNNING_ROBUST_TRUNK_COM_M=0.008 \
MICRODUCK_RUNNING_ROBUST_HEAD_COM_M=0.006 \
MICRODUCK_RUNNING_ROBUST_INITIAL_TILT_DEG=2.0 \
WANDB_MODE=offline \
uv run train \
  Mjlab-Running-Flat-MicroDuck \
  --env.scene.num-envs 4096

Running converges slower than walking — often 10+ hours. Parameters are not "larger is better"; monitor speed, stability, motion smoothness, and energy consumption simultaneously.

Cost Estimation and Best Practices

Cloud training cost comprises GPU instance runtime, system disk, and public network traffic. At referenced prices, basic walking training costs tens of RMB; more complex motions (running, jumping) cost proportionally more due to longer training. To avoid extra charges after training:

Save checkpoints and exported ONNX files.

Upload needed artifacts to object storage or download locally.

Stop or release the GPU instance.

Use ordinary CPU environments for subsequent simulation replay and data analysis.

Microduck's open-source framework lowers the barrier to robot motion development, but sim-to-real deployment still demands parameter tuning, robustness training, and hardware validation. New developers should first reproduce the official walking policy, then incrementally modify speed, reward functions, and perturbation parameters.