EgoHumanoid-V2

🤖 Human-to-humanoid skill transfer for coordinated whole-body loco‑manipulation.

Human demonstrations capture rich whole-body motion across diverse scenes without robot teleoperation. EgoHumanoid-V2 transfers these coordinated skills to a humanoid through coarse-to-fine action alignment: kinematic reference correction improves motion targets, and dynamics-aware refinement improves their execution through a frozen whole-body controller.

Visual alignment replaces human arms with rendered robot arms, while training-time image augmentation improves viewpoint robustness. A π₀.₅ vision-language-action policy learns from the aligned demonstrations and executes through Unitree G1 + BrainCo Revo2 + the SONIC whole-body controller. The paper evaluates zero-shot transfer on four real-world tasks without target-task robot demonstrations.

---

Workflow & Release Checklist

1. Hardware Setup
   • Robot Hardware
   • Human Hardware
2. Environment Setup
   • Robot Control
   • Human Data Collection
   • Vision Alignment
   • Policy Training & Serving
3. Data Collection
   • Human Collection
4. Data Alignment
   • Action Alignment ⭐
   • Vision Alignment
5. Model Training
   • Training pipeline
6. Policy Deployment
   • Deployment pipeline

---

🛠️ Hardware Setup

Human Hardware 🔴 PICO 4 Ultra Tracking: PICO 4 Ultra + trackers for whole-body and hand tracking Software: XRobo​Toolkit Setup guide · Python SDK Connec​tion: same LAN as the PC enable Head, Hand and Send 🔵 5× PICO Trackers Placement: five trackers both wrists, waist and both ankles Calibration: PICO Motion Tracker pair and calibrate in the app (guide) Mode: Full Body in XRobo​Toolkit enable Hand tracking (setup guide) 🟠 GoPro Mount: neck-mounted bracket Example: JD listing Alignment: match height, pitch and framing to the robot camera Connec​tion: Bluetooth pairing and WiFi setup; WiFi / COHN recording and download (API)

Robot Hardware 🟢 Unitree G1 Platform: Unitree G1, 29 body DOF excluding hands (joint layout) Control: SONIC whole-body controller runs on an external Linux PC Connec​tion: wired or WiFi to the PC via the ZMQ bridge 🔵 BrainCo Revo2 Hand: 6 active DOF per hand 11 including passive joints (specs) Software: official SDK hand control and state feedback Mount: custom G1–Revo2 adapters STL: part 1 · part 2 🟠 GoPro Mount: head-mounted bracket STL: part 1 · part 2 Alignment: match height, pitch and framing to the human camera Connec​tion: USB control and video to robot; wired / WiFi forwarding to host (API)

💻 Environment Setup

Install the environments needed for your workflow below. Action alignment uses the Robot Control environment; vision alignment has its own installer.

🤖 Robot Control

Host

Ubuntu 22.04 · NVIDIA driver 580+.

1. Install dependencies.

sudo apt-get update
sudo apt-get install -y --no-install-recommends ca-certificates curl build-essential cmake git git-lfs libgl1 libegl1 libegl-mesa0 libglib2.0-0 ffmpeg libyaml-cpp-dev libeigen3-dev libmsgpack-dev libzmq3-dev nlohmann-json3-dev libssl-dev zlib1g-dev
curl -LsSf https://astral.sh/uv/0.11.14/install.sh | sh
source "$HOME/.local/bin/env"
curl -fL https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-keyring_1.1-1_all.deb -o /tmp/cuda-keyring.deb
sudo dpkg -i /tmp/cuda-keyring.deb
sudo apt-get update
sudo apt-get install -y --no-install-recommends cuda-minimal-build-12-4

2. Install TensorRT and ONNX Runtime. Download TensorRT-10.13.3.9.Linux.x86_64-gnu.cuda-13.0.tar.gz from NVIDIA to the repository root, then run:

sudo tar -xzf TensorRT-10.13.3.9.Linux.x86_64-gnu.cuda-13.0.tar.gz -C /opt
curl -fL https://github.com/microsoft/onnxruntime/releases/download/v1.16.3/onnxruntime-linux-x64-1.16.3.tgz -o /tmp/onnxruntime.tgz
sudo tar -xzf /tmp/onnxruntime.tgz -C /opt

3. Install environment. Repeat the exports when opening a new terminal.

export TensorRT_ROOT=/opt/TensorRT-10.13.3.9
export onnxruntime_ROOT=/opt/onnxruntime-linux-x64-1.16.3
export CUDAToolkit_ROOT=/usr/local/cuda-12.4
export LD_LIBRARY_PATH="$TensorRT_ROOT/lib:$onnxruntime_ROOT/lib:$CUDAToolkit_ROOT/lib64${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}"
bash setup/robot/install_host.sh

4. Activate environment.

source sonic/.venv/bin/activate

Robot

ARM64 · Ubuntu 20.04 · ROS 2 Foxy · Python 3.8.

1. Install environment.

sudo apt-get update
sudo apt-get install -y python3-dev python3-venv build-essential cmake git python3-colcon-common-extensions ros-foxy-rmw-cyclonedds-cpp ros-foxy-rosidl-generator-dds-idl libyaml-cpp-dev libssl-dev libgl1 libglib2.0-0
bash setup/robot/install_robot.sh

2. Activate environment.

source /opt/ros/foxy/setup.bash
source .venv-robot/ros_ws/install/setup.bash
source .venv-robot/bin/activate

👤 Human Data Collection

Ubuntu 22.04 · Python 3.10.

1. Install environment.

sudo apt-get update
sudo apt-get install -y ca-certificates python3.10-dev python3.10-venv build-essential cmake git ffmpeg bluez network-manager
bash setup/human/install.sh

2. Activate environment.

source .venv-human/bin/activate

🎨 Vision Alignment

Tested on Linux with a 48 GB NVIDIA GPU. Allow about 35 GB of disk space. Requires uv, FFmpeg, and an NVIDIA driver compatible with CUDA 13.0; reuse these if already installed.

1. Obtain model access. Request access to SAM 3.1, then authenticate:

uvx --from huggingface_hub hf auth login

2. Install the environments and models.

bash setup/vision/install.sh

The installer creates two isolated environments under runtime/vision: Python 3.12 for segmentation/rendering and Python 3.10 for inpainting. It downloads pinned models and verifies their checksums. The alignment command calls both environments automatically; no manual switching is needed.

Installation scripts, dependency pins, model manifests, and the SAM compatibility patch are in setup/vision/. Processing code, tests, and example motion files are in data_alignment/vision_alignment/.

🧠 Policy Training & Serving

Ubuntu 22.04 · NVIDIA GPU and driver.

1. Install dependencies.

sudo apt-get update
sudo apt-get install -y --no-install-recommends ca-certificates curl git build-essential libgl1 libglib2.0-0
curl -LsSf https://astral.sh/uv/0.11.14/install.sh | sh
source "$HOME/.local/bin/env"

2. Install OpenPI.

bash setup/policy/install.sh

3. Activate environment.

source openpi/.venv/bin/activate

🎥 Data Collection

👤 Human Data Collection

1. Prepare devices.

Connect the host and PICO headset to the same network. Start the XRoboToolkit PC service on the host and the XRoboToolkit app on the headset, then enable body and hand tracking. Turn on the GoPro, enable Bluetooth pairing and select Video mode; the collector connects it to the host's WiFi network in the next step.

2. Start recording.

eh2-collect-human --task-name "{task_name}" --task "{task_instruction}"

Open http://127.0.0.1:8080. Enter the host's WiFi credentials if needed, then click Prepare collection → Start recording → Save episode / Discard episode. Wait for saving before recording again. Episodes are saved in LeRobot format to data/{task_name}/; new recordings append to the dataset.

3. Check and synchronize data.

eh2-sync-human --root "data/{task_name}"

Checks the recordings, synchronizes motion and video at 50 Hz, then checks the output. Results are saved to data/{task_name}_synced/, with synchronization and data quality reports in meta/.

⚡ Data Alignment

🦾 Action Alignment

We provide one example trajectory for each of the four tasks in examples. The simulation replays below compare these trajectories before (left) and after (right) alignment.

close_laptop.mp4	open_fridge.mp4
pull_curtain.mp4	close_drawer.mp4

1. Align motion.

Place the input recording at {task_name}/input.parquet. Set {endpoints} to hands for hand alignment or feet for foot alignment. Pretrained hand and foot response MLP weights are included in models/response.

eh2-align \
  --input "{task_name}/input.parquet" \
  --endpoints {endpoints} \
  --output "results/{task_name}"

Results are saved under results/{task_name}/: data.parquet contains the human SMPL motion and aligned tokens; run.json records per-round metrics and the selected round.

2. Replay motion.

SMPL replay: encode the human motion online to simulate before alignment.

eh2-replay \
  --input "{task_name}/data.parquet" \
  --mode smpl \
  --output "results/{task_name}/before"

Token replay: replay the aligned motion tokens to simulate after alignment.

eh2-replay \
  --input "{task_name}/data.parquet" \
  --mode token \
  --output "results/{task_name}/after"

Each command saves replay.mp4 in its output directory. Results may vary slightly between runs.

🎨 Vision Alignment

The pipeline replaces human arms with rendered G1 arms while preserving the original gloves and hand–object contact. We upgrade the paper's SAM 2 to SAM 3.1 for more robust arm and wrist-tracker segmentation. DiffuEraser reconstructs the background.

Synchronized joint trajectories drive the G1 model through an approximate virtual camera. Wrist locations and segmentation masks then refine the rendered arms' placement in the image without changing the joint angles.

The examples below demonstrate visual alignment for closing a laptop, opening a fridge, and pulling a curtain. Each row shows RGB → Mask → Inpainting → Robot-arm rendering. Download the video and motion files to reproduce the results.

Close Laptop · Video · Motion

Human RGB	SAM 3.1 Mask	Inpainting	Robot-arm Rendering

Open Fridge · Video · Motion

Human RGB	SAM 3.1 Mask	Inpainting	Robot-arm Rendering

Pull Curtain · Video · Motion

Human RGB	SAM 3.1 Mask	Inpainting	Robot-arm Rendering

Align a recording.

Save the video as {task_name}/rgb.mp4 and its synchronized motion as {task_name}/motion.npz. The NPZ contains body_q (29 G1 joint angles in radians per frame), joint_names, and fps matching the video.

runtime/vision/.venv-sam/bin/eh2-visual-align \
  --video "{task_name}/rgb.mp4" \
  --motion "{task_name}/motion.npz" \
  --runtime "runtime/vision" \
  --output "results/{task_name}"

Results are saved under results/{task_name}/: aligned.mp4 contains the aligned video; run.json records settings and completed stages.

🤖 Model Training

We fine-tune π₀.₅ to predict SONIC motion tokens and Revo2 hand commands from aligned human demonstrations.

1. Prepare data and weights.

Use a LeRobot dataset with task instructions in its metadata and the following synchronized columns. State and hand channels must follow the controller's ordering.

Column	Content
observation.images.ego_view or observation.images.gopro	Aligned RGB image
observation.state	43-dimensional G1 state
action.motion_token	64-dimensional SONIC token
teleop.left_hand_joints, teleop.right_hand_joints	6 Revo2 commands per hand

Download pi05_base and convert it to PyTorch.

2. Compute normalization statistics.

eh2-norm-stats \
  --dataset "data/{task_name}" \
  --assets "assets/policy"

3. Train. The defaults are 30,000 steps, global batch size 256, and a 100-frame action horizon. This command uses eight GPUs:

torchrun --standalone --nnodes=1 --nproc_per_node=8 \
  --module eh2_policy.train \
  --dataset "data/{task_name}" \
  --pretrained "checkpoints/pi05_base_pytorch" \
  --name "{run_name}" \
  --assets "assets/policy" \
  --output "checkpoints"

For one GPU, use eh2-train with the same arguments and a smaller --batch-size. Set --action-horizon consistently for statistics and training. Image augmentation is enabled by default; --no-image-augmentation disables it.

Checkpoints are saved under checkpoints/eh2/{run_name}/{step}/ with weights, normalization statistics, and serving settings. Use --resume to continue training.

🚀 Policy Deployment

1. Serve the policy. In the Policy Training & Serving environment:

eh2-serve \
  --checkpoint "checkpoints/eh2/{run_name}/{step}" \
  --host 0.0.0.0 \
  --port 8000

Serving settings are loaded from the checkpoint; random image augmentation is disabled.

2. Connect SONIC. Start the controller following the SONIC deployment procedure and enable the GoPro stream. In the Robot Control host environment, run from this repository's root:

source sonic/.venv/bin/activate
python sonic/gear_sonic/scripts/run_vla_inference.py \
  --host "{policy_host}" \
  --port 8000 \
  --camera-host "{camera_host}" \
  --camera-port 5559 \
  --prompt "{task_instruction}" \
  --enable-control

The client starts paused. Use the controller's keyboard workflow to initialize and begin execution.

📝 Citation

If you use this work, please cite:

@article{chen2026egohumanoidv2,
  title={{EgoHumanoid-V2}: Human-to-Humanoid Transfer of Coordinated Whole-Body Skills for Loco-Manipulation},
  author={Chen, Jin and Jiang, Yiming and Xu, Chongyang and Shi, Modi and Peng, Shijia and Chen, Li and Li, Tianyu and Xu, Mu and Chen, Yilun and Hoi, Steven and Li, Hongyang},
  journal={arXiv preprint arXiv:2609.37181},
  year={2026}
}

---

🙏 Acknowledgments

We sincerely thank the following projects and teams:

VLA models	Whole-body control	PICO tracking	GoPro interface

---

Contact

Questions, ideas, or interested in collaborating? We'd love to hear from you—feel free to connect with Jin Chen on WeChat!