Contents
  1. I. Installation and Environment Setup
  2. 1.1 Installing robomimic
  3. 1.2 Installing Simulators
  4. 1.3 Testing the Installation
  5. II. Reproducing the Experiments
  6. 2.1 Reproducing the Paper Results
  7. 2.2 Dataset Analysis
  8. III. Customization
  9. 3.1 Generating a Custom Dataset

robomimic is a framework for robot learning from demonstrations. The project provides a range of demonstration data for robot manipulation and offline learning algorithms, enabling standardized testing of tasks and algorithms.

The version tested in this article is robomimic v0.3.

I. Installation and Environment Setup

Before installation, make sure you are running a Linux system with conda installed.

1.1 Installing robomimic

(1) Create and activate a virtual environment

conda create -n robomimic python=3.8.0 -y
conda activate robomimic

(2) Install PyTorch

According to the official documentation, the PyTorch and torchvision versions can be changed, but I have not tested this yet.

conda install pytorch==2.0.0 torchvision==0.15.1 -c pytorch

(3) Install robomimic

cd <你想保存robomimic代码的位置>
git clone https://github.com/ARISE-Initiative/robomimic.git
cd robomimic
pip install -e .

1.2 Installing Simulators

(1) robosuite

Running most robomimic examples and released datasets requires the robosuite simulator. Install it as follows:

cd <你想保存robosuite代码的位置>
git clone https://github.com/ARISE-Initiative/robosuite.git
cd robosuite
git checkout v1.4.1
pip install -r requirements.txt
pip install -e .

(2) D4RL

D4RL also provides standardized training environments and datasets. Install it as follows:

git clone https://github.com/Farama-Foundation/d4rl.git
cd d4rl
pip install -e .

1.3 Testing the Installation

Basic test:

cd <robomimic代码安装位置>
python examples/train_bc_rnn.py --debug

Issue: AttributeError: module 'mujoco_py' has no attribute 'builder' My current solution is as follows:

  1. Install mujoco-210. See 3(3) Prepare the mujoco environment
  2. Install mujoco-py: pip install mujoco-py
  3. Install Cython: pip install "cython<3"

Issue: Error compiling Cython file The solution is as follows:

  1. pip install "cython<3"

Issue: EGL_NOT_INITIALIZED The solution is as follows:

  1. On line 155 of robosuite/robosuite/renderers/context/egl_context.py, replace self.free() with pass
  2. On line 199 of robosuite/robosuite/utils/binding_utils.py, comment out self.gl_ctx.free()

Comprehensive test:

Run a debugging environment to ensure that robomimic is installed correctly:

cd <robomimic代码安装位置>/test
bash test.sh

The process may take several minutes. If the command line displays passed many times, it means there is no problem. If an EGL-related issue occurs along the way, you can ignore it for now. EGL is used for rendering on server-edition machines without a display.

II. Reproducing the Experiments

2.1 Reproducing the Paper Results

(1) Download the dataset

Here, I downloaded the tool_hang scenario for testing. For other available scenarios, see the official documentation.

cd <robomimic代码安装位置>/robomimic/scripts
python download_datasets.py --tasks tool_hang

(2) Generate the configuration files

python generate_paper_configs.py --output_dir <想要保存试验结果的绝对路径>

The generated configuration files are located in robomimic/exps/paper by default. You can copy commands from the core.sh file for training and testing.

(3) Run training

Find the command you want to test in core.sh and run it. For example:

python /home/mahaofei/Programs/Imitation/RoboMimic/robomimic/robomimic/scripts/train.py --config /home/mahaofei/Programs/Imitation/RoboMimic/robomimic/robomimic/exps/paper/core/tool_hang/ph/low_dim/bc.json

The visualization video generated during training is located at the following path by default:

/tmp/experiment_results/core/bc/tool_hang/ph/low_dim/trained_models/core_bc_tool_hang_ph_low_dim/20231227161534/videos/ToolHang_epoch_100.mp4

2.2 Dataset Analysis

(1) Description of the official hdf5 contents

The official datasets for each task are available in two types, image and low_dim, but their data structures are essentially the same:

  • data (group)
    • total (attribute): the number of state-action samples in the dataset
    • env_args (attribute): a json string containing the environment metadata and related parameters used to collect the data, including three keys
      • env_name: the name of the environment or task to create
      • env_type: one of the environment types supported by robomimic
      • env_kwargs: a dictionary of keyword arguments to pass to env_name
    • demo_0 (group): the group for the first trajectory
      • num_samples (attribute): the number of state-action samples in this trajectory
      • model_file (attribute): the XML string for the corresponding MuJoCo model; applicable only to robosuite datasets
      • states (dataset): the flattened MuJoCo states, ordered by time, with shape(N,D), where N is the trajectory length and D is the dimension of the state vector. This should be empty for non-robosuite datasets.
      • actions (dataset): environment actions, ordered by time, with shape(N,A), where N is the trajectory length and A is the dimension of the action space.
      • rewards (datasets): the completion signal, which is 1 if the state corresponding to the current action indicates that the eposide should end
      • obs (group): the group storing the observation keys, where each key is a dataset
        • obs_key_1 (dataset): the first observation key; the dataset name and shape may differ
        • …
      • next_obs (group): the group for the next observation
        • obs_key_1 (dataset): the first observation key
        • …
    • demo_1 (group): the group for the second trajectory
    • …
  • mask (group): present in datasets containing filter keys
    • filter_key_1: the first filter key, such as the list [“demo_0”, “demo_19”, “demo_35”]

(2) Download the raw, low_dim, and image data formats

cd <你想保存数据集的位置>

# 下载原始数据和 low_dim 数据
wget http://downloads.cs.stanford.edu/downloads/rt_benchmark/tool_hang/ph/demo_v141.hdf5
wget http://downloads.cs.stanford.edu/downloads/rt_benchmark/tool_hang/ph/low_dim_v141.hdf5
# 或
cd robomimic/robomimic/scripts
python download_datasets.py --tasks <任务名称> --dataset_types all --hdf5_types all --download_dir <数据保存路径>

# 转换得到 image 数据
# 注意，如果训练image数据，需要修改对应json文件中crop_height和crop_width为76
# 注意，转换后数据文件可能在 10GB 左右，注意磁盘空间
python dataset_states_to_obs.py --dataset=<原始数据的路径> --output_name=image_v141.hdf5  --done_mode 2 --camera_names agentview robot0_eye_in_hand sideview --camera_height 84 --camera_width 84

When downloading data in image format, download_datasets.py reports that there is no URL. Converting raw data to image data causes an image format error during training. After searching, I found that the v2.0 documentation provides data in image format. Download it from there if needed.

For example, in robomimic’s tool_hang environment, the image dataset’s demo_0 content is as follows:

  • actions: shape (n, 7), where n is the trajectory length and 7 is the action dimension
  • dones: shape (n, ), where n is the trajectory length
  • obs
    • agentview_image: shape (n, 256, 256, 3), the image from the agent’s viewpoint, corresponding to (n, H, W, C), where n is the trajectory length, H is the image height, W is the width, and C is the number of image channels. All images must be of the np.uint8 type.
    • object: shape (n, 44), where n is the trajectory length. This contains the position and pose of the three objects base, frame, and tool, as well as their position and pose relative to the robotic arm’s end effector (pos3, quat4, to_eef_pos3, to_eef_quat4), for a total of 3x14 values, plus the two Boolean values frame_is_assembled and tool_on_frame.
    • robot0_eef_pos: shape (n, 3), the robot end-effector position
    • robot0_eef_quat: shape (n, 4), the robot end-effector pose quaternion
    • robot0_eef_vel_ang: shape (n, 3), the robot end-effector angular velocity
    • robot0_eef_vel_lin: shape (n, 3), the robot end-effector linear velocity
    • robot0_eye_in_hand_image: shape (n, 256, 256, 3), the robot end-effector camera image
    • robot0_gripper_qpos: shape (n, 2), the end-effector gripper action states for the left and right sides
    • robot0_gripper_qvel: shape (n, 2), the end-effector gripper action velocities for the left and right sides
    • robot0_joint_pos: shape (n, 7), the rotation angles of the robot’s joints
    • robot0_joint_pos_cos: shape (n, 7), the cos values of the robot joint angles
    • robot0_joint_pos_sin: shape (n, 7), the sin values of the robot joint angles
    • robot0_joint_vel: shape (n, 7), the robot joint velocities
    • sideview_image: shape (n, 256, 256, 3), the side-view camera image
  • next_obs
    • Same as above
  • rewards: shape (n, ), where n is the trajectory length
  • states: shape (n, 58), where n is the trajectory length and 58 is the dimension of the state vector. This should be empty for non-robosuite datasets.

The contents of the low_dim dataset are essentially the same as those of image, except that obs does not contain any images.

(3) Dataset visualization

Use the playback_dataset.py script to view dataset trajectories.

python playback_dataset.py --dataset <path/to/.hdf5> --video_path <path/to/.mp4> --n 5

III. Customization

3.1 Generating a Custom Dataset

The script for collecting human demonstration data in this project is robosuite/robosuite/scripts/collect_human_demonstrations.py. It lets you specify the environment, the robot to use, whether to control one arm or two arms, the cameras used to collect demonstration data, and so on.

However, these are all provided by robomimic. If I want to use my own robot model and record task data in a custom environment, extensive modifications would be required. Therefore, I considered generating the data directly in my own MuJoCo environment according to the HDF5 data format analyzed above.

References:

  1. Robotmimic Official Documentation (robomimic documentation)