[Paper Notes] Residual Skill Policies: Learning an Adaptable Skill-based Action Space for Reinforcement Learning for Robotics

I. Paper Notes

Title: Residual Skill Policies: Learning an Adaptable Skill-based Action Space for Reinforcement Learning for Robotics Author Team: Queensland University of Technology Venue: CoRL Year: 2022 Code: https://krishanrana.github.io/reskill

1.1 Target Problem

Skill-based learning has emerged as a way to accelerate robot learning. Skills are extracted from expert demonstrations and are short sequences of single-step operations (translation, grasping, lifting, and other actions). These skills are embedded in a latent space, forming the action space for the upper-level RL policy. However, this approach has some problems:

• Randomly sampling all skills for exploration is extremely inefficient, because only a small subset of them is relevant to the task currently being performed, and these relevant skills are usually not clustered in the same neighborhood of the skill space.
• This approach assumes that the skills are optimal and that the lower-level tasks come from the same distribution as the skill space. Its generality and adaptability to changes are therefore limited. For example, skills learned from moving blocks cannot handle situations involving obstacles, object changes, different levels of friction, and so on.

To solve the problems above, this paper proposes the following innovative methods, collectively called Residual Skill Policies (ReSkill):

• State-conditioned skill prior: sample relevant skills to guide exploration
• Lower-level residual policy: adapt to task changes through fine-grained skill adaptation

1.2 Method

Overall, this method decomposes demonstration trajectories generated by a classical controller into task-agnostic skills and embeds them into a continuous-to skill space. It uses the skill space to achieve truly general learning: the upper-level agent can access the skill space but does not act, reducing the requirements for dataset granularity.

• Extract skills from existing controllers
• Learn skill embeddings and a skill prior
• Train a hierarchical reinforcement learning policy that uses a lower-level residual adaptive policy in the skill space.

(1) Data Collection

The paper collects demonstration data through manual control (basic manipulation tasks such as pushing and grasping objects). Although the tasks are simple, the trajectories contain complex skills that can be recombined to solve complex tasks.

A trajectory consists of state-action pairs. The paper randomly slices segments of length $H$ from them for unsupervised skill extraction and uses the extracted action a and state s to learn the state-action described in the next subsection.

Here, state s includes joint angles, joint velocities, gripper position, and object position, while the action is a continuous 4D vector that includes the end-effector position and velocity.

(2) Learning a State-Conditioned Skill Space for Reinforcement Learning

• Embed the extracted skills into a latent space: use a variational autoencoder (VAE) to embed skill $a$ into a latent space. The VAE includes an encoder and a decoder. The encoder encodes the complete state-action sequence into $z$, and the decoder reconstructs the action from the current state $s_t$ and the skill encoding $z$.
• State-conditioned prior for skills sampled during exploration: learn a conditional probability density over the latent skill space. Traditional Gaussian densities cannot handle multimodal information, so the paper uses RealNVP, a real-valued non-volume-preserving transformation. It learns a mapping from $Z\times S->G$. This mapping can transform the simple distribution G into the skill space Z, so f is the skill prior.

Variational autoencoder: a type of deep generative model Traditional approach: A traditional autoencoder consists of an encoder and a decoder. Through repeated training, the input data is encoded into an encoding vector, with each dimension of the vector representing a learned feature of the data, while the decoder attempts to decode the original input from the encoding vector. Limitation: The traditional approach uses a single value to represent how the input performs on a particular latent feature. In practice, however, it is more reasonable to represent a latent feature as a range of possible values. Improvement: A variational autoencoder therefore uses a probability distribution over values instead of the original single-value feature representation. Advantage: Each latent feature is represented as a probability distribution. During decoding, a value is randomly sampled from the latent-state distribution to generate an encoding vector as input to the decoder. This produces a continuous and smooth latent-space representation (values adjacent to each other in the latent space reconstruct similar results). Reference for understanding: https://zhuanlan.zhihu.com/p/64485020

(3) Reinforcement Learning in the State-Conditioned Skill Space

Once training is complete, the decoder and skill-prior weights are frozen and incorporated into the RL framework. The high-level reinforcement learning policy $\pi$ is a neural network that maps the state to a vector g in the skill-prior variation, at converting into latent skill Z.

The decoder then reconstructs the action according to the current state sequence over the skill horizon H. A lower-level residual policy adjusts the decoded skill at the same time.

1.3 Summary

This method is a skill-based reinforcement learning method.

1. Data collection: use the most basic controllers to generate trajectories for some basic tasks (moving and grasping), then divide these trajectories into fixed-length sequences, with each short segment containing actions and their corresponding states.
2. Learn the skill space: use a variational autoencoder to encode skills into a latent space; use RealNVP to map the skill latent space + robot state space into a simple distribution space (a Gaussian distribution), so that skills can be sampled directly from the state. This is called the skill prior.
3. Reinforcement learning: use a high-level policy network to generate a vector from the current state, select a skill according to the skill prior (skills related to the current state), and use the skill decoder to decode it into robot actions.

II. Code Reproduction

2.1 Environment Setup

(1) Install mujoco:

Download mujoco

wget https://mujoco.org/download/mujoco210-linux-x86_64.tar.gz

Create a hidden folder and try not to change this path

mkdir ~/.mujoco

Extract the mujoco library into the folder above

tar -xvzf mujoco210-linux-x86_64.tar.gz -C ~/.mujoco

Edit the environment variables

gedit ~/.bashrc

Add the following statements to the end of the file, remembering to change the username to your own

# Mujoco environment
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/lib/nvidia
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/用户名/.mujoco/mujoco210/bin
export LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libGLEW.so

Refresh the environment variables by restarting the terminal or running the following command

source ~/.bashrc

Test MuJoCo

cd ~/.mujoco/mujoco210/bin
./simulate ../model/arm26.xml

If a MuJoCo interface starts and displays a two-degree-of-freedom robotic arm, the installation was successful. There are also many other example models under ../model/; take a look if you are interested.

(2) Build the Python Environment

git clone https://github.com/krishanrana/reskill.git
cd reskill
conda env create -f environment.yml
conda activate reskill_new
pip install -e .
cd reskill

2.2 Data Collection

Use the following script to collect data

python data/collect_demos.py --num_trajectories 40000 --subseq_len 10 --task block

Here, task can be set to block or hook.

2.3 Training

Train the skill modules:

python train_skill_modules.py --config_file block/config.yaml --dataset_name fetch_block_40000

Visualize the performance of the trained skill modules:

python utils/test_skill_modules.py --dataset_name fetch_block_40000 --task block --use_skill_prior True

Train the ReSkill agent:

python train_reskill_agent.py --config_file block/config.yaml --dataset_name fetch_block_40000

Visualize the trained ReSkill agent:

python utils/test_reskill_agent.py --dataset_name fetch_block_40000 --env_name FetchSlipperyPush-v0

2.4 Logging

Use W&B. When training for the first time, enter your own API.

III. Understanding the Code

3.1 Basic Definitions

(1) Robot Actions

In the algorithm, robot actions are defined using the following method:

def _set_action(self, action):
	'''设置动作，在模拟环境中执行动作'''
	# 处理输入动作
	assert action.shape == (4,) # 确保输入的动作形状是（4，）
	action = action.copy()      # ensure that we don't change the action outside of this scope
	pos_ctrl, gripper_ctrl = action[:3], action[3]  # 将动作差分成位置控制[:3]和夹爪控制[3]

	# 对输入动作值进行处理，末端位置进行缩放，旋转固定，夹爪根据条件是否设0
	pos_ctrl *= 0.05  # 限制位置变化的最大值
	rot_ctrl = [1., 0., 1., 0.]  # 固定末端执行器的旋转，使用四元数表示
	#rot_ctrl = [ 0.5, -0.5, 0.5, 0.5 ]  # 90 deg rotation of the original end effector, expressed as a quaternion
	gripper_ctrl = np.array([gripper_ctrl, gripper_ctrl])   # 夹爪复制成两个
	assert gripper_ctrl.shape == (2,)
	if self.block_gripper:  # 如果block_gripper，则将手指位置设置为0
		gripper_ctrl = np.zeros_like(gripper_ctrl)
	# 将经过修改后的位置控制、固定的末端执行器旋转和处理后的夹爪控制连接成一个新的动作数组 action
	action = np.concatenate([pos_ctrl, rot_ctrl, gripper_ctrl])

	# Apply action to simulation. 将动作应用到仿真环境中
	utils.ctrl_set_action(self.sim, action)
	utils.mocap_set_action(self.sim, action)

As shown, in this algorithm, the robot action is defined as an array of length 4. The four values respectively represent the control position of the robot’s end effector and the degree to which the gripper is open or closed. (Rotation is ignored here because the task is to grasp a block and move it to a specified position, so the algorithm directly sets the end effector to always point vertically downward.)

action = [x, y, z, gripper]

In fact, in Gym, the robot’s action is controlled using an array of length 9. These values respectively represent 3 variables for the end effector’s spatial position, a quaternion for its spatial orientation, and the actions of the gripper’s two parallel plates.

action = [x, y, z, quat1, quat2, quat3, quat4, gripper_l, gripper_r]

(2) Observation State

The observation state is obtained using the following code:

    def _get_obs(self):
        '''获得环境的观察'''
        # 位置
        grip_pos = self.sim.data.get_site_xpos('robot0:grip')   # 获取机器人手爪的位置
        dt = self.sim.nsubsteps * self.sim.model.opt.timestep
        grip_velp = self.sim.data.get_site_xvelp('robot0:grip') * dt    # 计算手爪的线速度
        robot_qpos, robot_qvel = utils.robot_get_obs(self.sim)  # 使用辅助函数获取机器人的位置和速度

        gripper_state = robot_qpos[-2:]     # 提取了夹爪的状态和速度
        gripper_vel = robot_qvel[-2:] * dt  # change to a scalar if the gripper is made symmetric

        # 将夹爪的位置、状态、夹爪线速度、速度连接起来，形成初始观察
        obs = np.concatenate([
            grip_pos,
            gripper_state,
            grip_velp,
            gripper_vel,
        ])

        # 存储已经到达的目标
        achieved_goal = []

        # 遍历所有的方块
        for i in range(self.num_blocks):
        # for i in range(1):

            # 获取方块们的位置、姿态、速度、相对位置、相对速度
            object_i_pos = self.sim.data.get_site_xpos(self.object_names[i])
            # rotations
            object_i_rot = rotations.mat2euler(self.sim.data.get_site_xmat(self.object_names[i]))
            # velocities
            object_i_velp = self.sim.data.get_site_xvelp(self.object_names[i]) * dt
            object_i_velr = self.sim.data.get_site_xvelr(self.object_names[i]) * dt
            # gripper state
            object_i_rel_pos = object_i_pos - grip_pos
            object_i_velp -= grip_velp

            # 连接到观察值中
            obs = np.concatenate([
                obs,
                object_i_pos.ravel(),
                object_i_rel_pos.ravel(),
                #object_i_rot.ravel(),
                object_i_velp.ravel(),
                #object_i_velr.ravel()
            ])

            # This is current location of the blocks
            # 方块们的当前位置
            achieved_goal = np.concatenate([
                achieved_goal, object_i_pos.copy()
            ])

        achieved_goal = np.concatenate([achieved_goal, grip_pos.copy()])

        achieved_goal = np.squeeze(achieved_goal)

        if self.use_force_sensor:
            self.sim.data.get_sensor('force_sensor') 
            force_reading = self.sim.data.sensordata # Read force sensor reading from tray
        else:
            force_reading = [0,0,0]

        return {
            'observation': obs.copy(),
            'achieved_goal': achieved_goal.copy(),
            'desired_goal': self.goal.copy(),
            'force_sensor': force_reading.copy()
        }

As shown, the observation state is a dictionary:

Key	Value	Meaning
observation	[grip_x, grip_y, grip_z,	grip_pos, end-effector position
	grip_q1, grip_q2, grip_q3, grip_q4,	grip_quat, end-effector orientation
	gripper_left, gripper_right,	gripper_state, states on both sides of the gripper
	grip_vx, grip_vy, grip_vz,	grip_vel, end-effector velocity
	grip_wx, grip_wy, grip_wz	grip_w, end-effector angular velocity
	gripper_vl, gripper_vr,	gripper_vel, velocities on both sides of the gripper
	obj1_x, obj1_y, obj1_z,	obj_i_pos, position of block i
	obj1_q1, obj1_q2, obj1_q3, obj1_q4	obj_i_quat, orientation of block i
	obj1_rx, obj1_ry, obj1_rz,	obj_i_rel_pos, position of block i relative to the end effector
	obj1_rq1, obj1_rq2, obj1_rq3, obj1_rq4	obj_i_rel_quat, orientation of block i relative to the end effector
	obj1_vx, obj1_vy, obj1_vz,	obj_i_velp, velocity of block i relative to the end effector
	obj1_wx, obj1_wy, obj1_wz,	obj_i_wp, angular velocity of block i relative to the end effector
	obj2…	position, relative position, and relative velocity of block 2
	obji…]	position, relative position, and relative velocity of block i
achieved_goal	[obj1_x, obj1_y, obj1_z, obj1_q1, obj1_q2, obj1_q3, obj1_q4, …	Each call to the _get_obs() method saves the positions and orientations of all blocks
	grip_x, grip_y, grip_z, grip_q1, grip_q2, grip_q3, grip_q4 …]	Appends the current end-effector position to the end of achieved_goal
desired_goal	[[x1, y1, z1], [x2, y2, z2], … ]	Multiple target positions
force_sensor	[f_x, f_y, f_z]	force_reading, end-effector force, default [0,0,0]

💰