Cross-Embodiment Offline Reinforcement Learning for Heterogeneous Robot Datasets

We scale robot pre-training by combining cross-embodiment learning and offline RL, so we can use both expert and suboptimal trajectories across many robot morphologies.

1. Why Cross-Embodiment + Offline RL?

Robotics faces a persistent data bottleneck: collecting high-quality demonstrations is costly and difficult to scale.

Cross-embodiment learning broadens data diversity by aggregating trajectories from multiple robot types into a shared training stream, expanding behavior and dynamics coverage beyond a single embodiment.

Offline reinforcement learning broadens data usability by learning effectively from mixed-quality trajectories, including suboptimal logs. Together, they form a practical path to scalable robot pre-training.

2. Benchmark + Dataset

The benchmark covers 16 robots (9 quadrupeds, 6 bipeds, 1 hexapod). For each embodiment, we construct 1M-step datasets with the following quality regimes:

• Expert: 1M steps collected by rolling out a fully converged PPO policy (high-quality trajectories).
• Expert Replay: interaction data from training start to expert-level performance, then uniformly subsampled to 1M steps.
• 70% Suboptimal Replay: 70% sampled from the early suboptimal phase and 30% from the late expert-like phase.

Each regime is provided in Forward and Backward variants. Importantly, we use a family of suboptimal-ratio datasets to stress-test transfer, conflict, and robustness as data quality changes in cross-embodiment settings.

3. Transfer Findings and Gradient Conflict Analysis

Cross-embodiment offline RL does not yield uniformly positive transfer. As the suboptimal ratio increases, some robots improve while others degrade. To identify the mechanism, we analyze gradient conflicts directly.

Gradient Conflict Analysis

To explain this behavior, we measure pairwise actor-gradient cosine similarity between embodiments:

The same trend appears in both IQL and TD3+BC, indicating that naive scaling is limited by optimization conflict.

Correlation Between Embodiment Distance and Gradient Conflicts

We represent each robot as a graph (torso, joints, feet) and compute embodiment similarity using fused Gromov-Wasserstein (FGW), then compare similarity with gradient cosine statistics.

IQL: embodiment similarity matrix, gradient cosine matrix, and scatter correlation.

TD3+BC: embodiment similarity matrix, gradient cosine matrix, and scatter correlation.

4. Embodiment Grouping (EG)

EG reduces cross-robot interference by updating morphologically compatible robots together. The method computes embodiment distances once, clusters robots, and then performs group-wise actor updates with a shared critic update.

1. Robot-as-graph: represent each embodiment with a torso-joint-foot graph.
2. FGW distances: compute pairwise embodiment distances from graph structure and node features.
3. Hierarchical clustering: cluster robots into M fixed groups before training.
4. Group-wise actor updates: update the critic globally, then update the actor sequentially by group.

Why it works: conflicting updates are suppressed across dissimilar robots, while positive transfer is preserved within each group.

Why practical: distances are precomputed once, and EG can be added to standard offline RL pipelines with minimal implementation overhead.

5. Results

EG achieves the best overall mean performance among compared methods. The largest improvements appear in suboptimal-heavy datasets, where conflicts are most severe. On 70% Suboptimal Forward, IQL+EG improves from 36.62 to 51.19 (+39.8%).

BibTeX

@inproceedings{abe2026crossembodimentoffline,
  title     = {Cross-Embodiment Offline Reinforcement Learning for Heterogeneous Robot Datasets},
  author    = {Abe, Haruki and Osa, Takayuki and Mukuta, Yusuke and Harada, Tatsuya},
  booktitle = {International Conference on Learning Representations (ICLR)},
  year      = {2026}
}