Best viewed with sound on :)
Play hosted video here · 1080p
Live human motion in. Terrain-aware humanoid behavior out.
NEXUS takes its name from the connections it builds—between human and robot, expressive motion and perceptive control—with a vision of carrying human intent across terrains, not just across embodiments.
NEXUS IN THE REAL WORLD
Human intent, different terrain. Explore whole-body behaviors across indoor obstacles and outdoor environments.
Explore the experiments in the full project video ↗.
Select a clip to watch in a larger player. Clips retain the supplied footage’s playback speed.
Try NEXUS in your browser. Choose a motion and terrain, then watch the robot respond.
Explore diverse motions—from walking and dancing to squatting, kneeling, and jumping—across flat ground, stairs, slopes, rough terrain, waves, and random platforms. Customize the terrain, import your own motion file, and try new combinations.
Launch Interactive Demo
Opens in a new tab
Best experienced in a desktop browser · Initial loading may
take a few seconds.
Whole-body teleoperation requires a humanoid robot to reproduce a human operator's behavior even when their terrains differ. This demands that the robot perceive local terrain and adapt its posture and contacts accordingly, rather than copy the operator's motion frame by frame. However, paired motion data linking the same behaviors across flat ground and different terrains remain scarce, limiting supervision for learning terrain-adaptive control.
To enable whole-body teleoperation across mismatched terrains, we introduce NEXUS, a perceptive whole-body control framework that combines human motion commands with onboard sensory feedback. We first develop a scalable terrain-aware adaptation algorithm that efficiently generates high-quality motion pairs across motions and terrains without per-motion or per-terrain tuning. Using a paired motion corpus totaling nearly 1,000 hours, we train a perceptive whole-body controller through teacher–student learning to reproduce commanded behaviors across terrains.
Experiments demonstrate efficient, scalable generation of high-quality motion data and show that NEXUS combines broad behavioral coverage with terrain adaptability and tracking fidelity, outperforming existing whole-body controllers on the evaluated benchmarks. Zero-shot real-world deployment enables real-time whole-body teleoperation on diverse unseen terrains, further validating the generalization of our method.
From offline motion adaptation to perceptive policy learning and zero-shot deployment.
01 / Motion adaptation
Detect foot and hand contacts in a flat-ground motion, adjust pelvis height to the terrain, and solve inverse kinematics at contact keyframes with penetration correction. Interpolating the pose corrections reconstructs a terrain-adapted trajectory aligned frame by frame with the original command.
02 / Policy learning
A privileged teacher learns to track adapted references using reinforcement learning (PPO). DAgger-style imitation then trains a student with the paired flat-ground command, proprioceptive history, and a depth image. The paired data connect the operator’s intent to the behavior required on the robot’s terrain.
03 / Deployment
At deployment, streamed human motion is retargeted to the robot in real time to form the motion command. The student combines this command with onboard depth and proprioception to produce joint-position targets. Terrain-adapted references are used only during training; at inference, terrain adaptation relies on robot-side perception.
From a flat-ground human command to whole-body behavior on the robot’s terrain.
Nearly 1,000 h
A motion corpus including terrain-adapted variants connects human behaviors to the posture and contacts needed on different support surfaces.
Hands + feet
Contact-guided adaptation handles both foot and hand support, adjusting contacting limbs while preserving motion in the rest of the body.
Real-time
Streamed human motion, onboard depth, and proprioception drive terrain-adaptive behavior in real time.
One adaptation pipeline, diverse motions and terrains. The same configuration generates paired references across ascending and descending stairs and slopes, without per-motion or per-terrain tuning. The controller learns from these pairs so that the operator can stay on flat ground while the robot responds to its own surroundings.
Beyond locomotion. Real-world demonstrations span walking, crouching, squatting, kneeling, kicking, single-leg balance, jumping, and cartwheels. The human supplies the behavior; NEXUS uses robot-side perception to adapt its execution.
Real-time whole-body teleoperation under human–robot terrain mismatch across indoor and outdoor terrains, without fine-tuning.
Policy inference runs onboard a Jetson Orin NX at 50 Hz, using depth images from an Intel RealSense D435i. Human poses are retargeted on a separate PC and streamed to the robot, where the policy combines them with local depth and proprioception.
Although training terrains contain only 10 cm steps and slopes of approximately 16.7°, NEXUS transfers without fine-tuning to continuous ascent of 15 cm stairs, grassy slopes reaching approximately 30°, and stepping or jumping down from a 60 cm platform. It also performs cartwheels on indoor ramps and stairs and on outdoor grassy slopes. In our trials, no failures were observed during routine pose tracking; falls occurred during some drops from the 60 cm platform and cartwheel attempts.
Current scope. The motion adaptation algorithm does not support overhanging terrain. Extending its geometric coverage and connecting the controller to task-level planners are directions for future work.
@misc{miao2026nexus,
title = {{NEXUS}: Perceptive Whole-Body Control for Terrain-Adaptive Teleoperation},
author = {Xiangyu Miao and Junsong Wu and Jiyuan Shi and Weiji Xie and Jinrui Han and Xingyi Wang and Weinan Zhang and Chenjia Bai and Xuelong Li},
year = {2026},
eprint = {2609.39000},
archivePrefix = {arXiv},
primaryClass = {cs.RO},
doi = {10.48550/arXiv.2609.39000},
url = {https://arxiv.org/abs/2609.39000}
}