[Paper Notes] Robot Motion Imitation with Reinforcement Learning

1 Reinforcement Learning with Videos: Combining Offline Observations with Interaction

Title: Reinforcement Learning with Videos: Combining Offline Observations with Interaction Authors: University of Pennsylvania Venue: CoRL Year: 2020 Code: https://github.com/kschmeckpeper/rl_with_videos

1.1 Problem Statement

Applying reinforcement learning to teach robots skills typically requires large amounts of online robot data, but collecting robot data is cumbersome and difficult, making it hard to obtain enough data.

Human videos are abundant and diverse, so the authors consider reinforcement learning from human experience. However, human videos lack action labels, and human videos and robot camera images differ greatly in appearance and viewpoint. The specific challenges are as follows:

• The robot must be able to update its policy through observation alone, without any actions or rewards;
• Human hands and end effectors differ substantially in appearance, and their degrees of freedom differ as well, so variations in action space, morphology, viewpoint, and environment must be accounted for;

To address these issues, this paper proposes the Reinforcement Learning with Videos (RLV) framework, which uses human demonstration data together with robot data to learn a policy and value function.

1.2 Method

(1) Problem formulation

The paper formulates the problem as a Markov decision process (MDP), defined as the tuple $(S_{int},A_{int},P,R)$, where $S_{int}$ is the state space, $A_{int}$ is the action space, $P$ is the environment dynamics, and $R$ is the reward function.

The robot is first provided with human observations ${(s_{obs},s_{int}')_{1:t}}$. These observations are modeled as another Markov decision chain with different state and action spaces, but the dynamics and reward function of the two are the same.

(2) Method overview

The method proposed in this paper is shown in the figure below. It contains two replay buffers: one holds action-free observation data $(s_{obs},s_{obs}')\in D_{obj}$, and the other holds action-conditioned interaction data $(s_{int},a_{int},s_{int}',r_{int})\in D_{int}$. The interaction data is updated during training, while the observation data is only the initial observation dataset.

Replay buffer (reply pool): A data structure that stores the agent’s past experiences (state, action, reward, next state). By sampling from this buffer for training, the agent can learn more patterns from past experience and improve its decision-making ability.

• Left: Sample data $(s_{int},a_{int},s_{int}',r_{int})$ from the action-conditioned replay buffer, encode the observation states into features $h_{int},h_{int}'$ respectively, and train an invertible model to predict the action $a_{int}$ from the features.
• Center: Apply this invertible model to observation-state features $h_{obs},h_{obs}'$ to predict the missing robot actions $\hat a_{int}$ in offline videos. Set a large reward for the final step of the trajectory and small rewards for all other steps.
• Right: Use adversarial domain confusion (ADS) to align features, then apply an off-policy reinforcement learning algorithm to train on data $((h_{int},h_{obs}),(a_{int},\hat a_{int}),(h_{int}',h_{obs}'),(r_{int},\hat r_{obs}))$.

Adversarial Domain Confusion (ADC): Achieves cross-domain transfer learning by minimizing the feature distribution distance between the source domain and the target domain.

(3) Action prediction

This paper trains an inverse model with parameters $\theta$ via supervised learning to compute robot actions from a pair of invariant feature encodings $(h,h')$. Because the robot and human video environments are the same, we should be able to predict actions for data from either Markov decision process.

The loss uses the mean squared error between the predicted action $\hat a_{int}=f_{inv}(h_{int},h_{int}';\theta)$ and the true action $a_{int}$:

$$L_a(a_{int},h_{int},h_{int}',\theta)=||a_{int}-f_{inv}(h_{int},h_{int}';\theta)||^2$$

This paper uses the inverse model to predict action data in human-hand videos and uses them to train the reinforcement learning algorithm.

(4) Reward generation

A barrier to using reinforcement learning with observation data is the lack of rewards. Although the inverse model trained above can predict rewards and actions, the results may not be very good in practice.

This paper uses an alternative: assign a large constant reward to the final timestep of each observation-data trajectory and a small constant reward to every earlier timestep.

The goal of this approach is to ensure that observation data reaches the goal state at the end of the trajectory. Inaccuracies can be mitigated through training on interaction data collected by the robot.

(5) Domain adaptation

To use observation data $(s_{obs},s_{obs})'$, it must be mapped to an invariant quantity $h$.

To this end, this paper trains a feature encoder $f_{enc}$ to learn from observation state $s$ an encoded representation $h=f_{enc}(s;\psi)$. This encoder should contain all relevant information and be invariant to the observation domain.

This paper also trains a discriminator to distinguish features $h_{obs}$ extracted from observation data from features $h_{int}$ extracted from robot interaction data.

The feature encoder and discriminator are trained with an adversarial learning approach: the encoder tries to minimize the discriminator’s ability to correctly classify the domain of encoded features, while the discriminator tries to maximize classification accuracy.

The resulting encoder is what we need: an encoder that maps observation data and robot interaction data to the invariant quantity $h$.

(6) Joint optimization

The domain adaptation loss and the inverse model loss are jointly optimized.

2 Learning Generalizable Robotic Reward Functions from “In-The-Wild” Human Videos

Title: Learning Generalizable Robotic Reward Functions from “In-The-Wild” Human Videos Authors: Stanford University Venue: Robotics: Science and Systems (RSS) Year: 2020 Code: https://sites.google.com/view/dvd-human-videos

2.1 Problem Statement

For general-purpose robots to complete diverse tasks, a key requirement is the ability to know task success and rewards. The reward function must also generalize across different environments, tasks, and objects.

Because collecting large-scale robot interaction data is very complex and difficult, human videos contain abundant task information across diverse environments.

This paper proposes a Domain-agnostic Video Discriminator (DVD), which learns a multi-task reward function by training a discriminator to classify whether two videos perform the same task. It generalizes from a small amount of robot training data to the broad human video dataset.

Problems to address:

• Human wild data and the robot observation space involve large domain shifts, whether in agent morphology or scene appearance.
• Human action space and robot action space differ, so action mapping may not work well
• Human videos are often low quality and noisy, with complex backgrounds or viewpoints

Solution approach:

• Train a classifier to predict whether two videos complete the same task, i.e., the Domain-agnostic Video Discriminator (DVD)
• After training, DVD can take a human video as a demonstration and robot behavior as another video, and output a score that measures task-success reward.

2.2 Method

(1) Domain-Agnostic Video Discriminators

• A pretrained video encoder encodes video $d_i$ into feature $h_i$
• A fully connected neural network predicts whether two videos complete the same task
  • Loss function settings are given in the original paper
  • The reward function is obtained by training the classifier

The key idea in this paper is to train a classifier to learn $R_\theta$, which takes two videos as input and determines whether they belong to the same task. Videos may come from human datasets or robot datasets.

First, videos are sampled. Let the two videos be $d_i$ and $d_j$. Sample a batch of videos $(d_i,d_i',d_j)$, where $d_i$ and $d_i'$ complete the same task and $d_j$ completes a different task. Minimize the average cross-entropy loss to train $R_\theta$. The loss function is given in the original paper. The resulting reward function is:

$$R_\theta(d_i,d_j)=f_{sin}(f_{enc}(d_i),f_{enc}(d_j);\theta)$$

where $h=f_{enc}$ is a pretrained video encoder and $f_{sin}(h_i,h_j;\theta)$ is a fully connected neural network with parameters $\theta$ that predicts whether encoded features $h_i,h_j$ from two videos complete the same task.

(2) Executing tasks with DVD

Implemented with Visual Model Predictive Control (VMPC).

1. Train an action-conditioned video prediction model $\rho$ with the SV2P model
2. Use cross-entropy and this action model $\rho$ to select actions most similar to the human demonstration
   1. For the input image, sample multiple action sequences from the action distribution and use action model $\rho$ to predict the corresponding future trajectories
   2. Feed each predicted trajectory and the human demonstration video into DVD to obtain a task similarity score
   3. Execute the action trajectory with the highest similarity to the demonstration image

3 PLAS: Latent Action Space for Offline Reinforcement Learning

Title: PLAS: Latent Action Space for Offline Reinforcement Learning Authors: Carnegie Mellon University Venue: CoRL Year: 2021 Code: https://github.com/sfujim/BCQ

3.1 Problem Statement

Offline reinforcement learning can learn a policy from a fixed dataset.

In robotics, data collection is cumbersome and can be dangerous. Existing methods that learn from offline datasets are very limited in performance.

This paper proposes Policy in the Latent Action Space (PLAS).

3.2 Method

(0) Background — offline RL

Given a fixed offline dataset $D={(s_t,a_t,r_t,s_{t+1})_i}$, the difficulty is that the dataset does not cover the entire state space and action space of the Markov decision process (MDP).

The goal of offline RL is to learn a policy that maximizes reward, while the policy is constrained by what we know about the MDP, which is inferred from a limited dataset.

However, if we consider the goal of offline RL to be maximizing cumulative return under the MDP with a limited dataset, that can serve as an approximate substitute. Approximation errors also arise when approximating the Q function.

Given a state, the latent policy outputs a latent action, which a decoder maps back to the action space. (A perturbation layer can be added to improve generalization.)

(1) Policy in Latent Action Space (PLAS)

Given an offline dataset, this paper models the policy with a Conditional Variational Autoencoder (CVAE). To keep the policy within the support of the dataset, it uses a deterministic policy that maps from state to latent action, then decodes to the actual action with a decoder.

(2) Generalization

The latent policy provides constraints within the dataset support, but during training this paper allows out-of-distribution behavior by adding a perturbation layer and using a hyperparameter to limit the residual output of the perturbation layer on actions.

Of course, if the dataset has very high coverage in state–action space, this perturbation layer is unnecessary.

4 Demonstration-Guided Reinforcement Learning with Efficient Exploration for Task Automation of Surgical Robot

Title: Demonstration-Guided Reinforcement Learning with Efficient Exploration for Task Automation of Surgical Robot Authors: The Chinese University of Hong Kong (Yun-Hui Liu’s group) Venue: ICRA Year: 2023 Code: https://github.com/med-air/DEX

4.1 Problem Statement

Although reinforcement learning-based methods offer a possible path to surgical automation, they usually require collecting large amounts of data before learning can proceed. This paper therefore aims to improve exploration efficiency when learning from demonstrations and to make effective use of expert demonstration data.

Specifically, the current problems are as follows:

• With reinforcement learning, learning a task through exploration alone without demonstration data requires collecting a large amount of data;
• Methods that use demonstration data, such as giving demonstration data higher priority than robot exploration data, remain inefficient. Methods that set additional reward functions not only target specific environments but also easily lead to local optima;
• Under the actor-critic framework, regularizing the actor loss to measure behavioral differences between the robot and the expert is inefficient (especially early on when the robot and demonstrations differ greatly), and without critic regularization it easily leads to overestimation.

Contributions of this paper:

• Propose an actor-critic framework that reduces critic overestimation and encourages exploration of expert-like actions during reinforcement learning.
• Use nonparametric guided propagation to explore unobserved states
• Validate on the SurRoL surgical robot with strong results, and deployment on dVRK also shows strong potential.

dVRK (da Vinci Research Kit, da Vinci surgical robot system)

4.2 Method

DEX (Demonstration-guided EXploration), demonstration-guided exploration.

(0) Problem formulation

Surgical robot action learning is treated as an off-policy agent interacting in an environment built from a Markov decision process.

off-policy means the agent does not use its current policy to choose actions; instead it uses a different policy to generate behavioral data and learns optimal decision-making from past experience.

At time $t$, the robot executes an action according to the current state $s_t$ and deterministic policy $\pi$. The environment rewards the agent with $r_t=r(s_t,a_t)$, then transitions to state $s_{t+1}$.

Repeating this process, the agent stores each experience $(s+t,a_t,r_t,s_{t+1})$ in replay buffer $D_A$.

A demonstration buffer $D_E$ is also set up to store experiences from expert policy $\pi$.

As shown in the figure, the method consists of two parts:

• An actor-critic policy learning module (bottom right) that guides exploration from demonstration data;
• A nonparametric module based on nearest-neighbor matching and locally weighted regression (top left) that propagates demonstrations that differ too much from the current state to the current state.

(1) Expert-guided actor-critic framework

Existing actor-critic methods learn an optimal policy by maximizing expected return, but inaccurate Q-value estimates can hinder exploration. This paper augments the environment reward by exploiting the action gap between the agent and the expert policy.

$$\max_{\pi}\mathbb{E}_{\pi}\left[\sum_{t=0}^{\infty}\gamma^{t}(r_{t}-\alpha d(a_{t},a_{t}^{e}))\right],a_{t}^{e}:=\pi^{e}(s_{t}),$$

where $\alpha$ is the exploration coefficient and $d()$ is a similarity distance metric between agent actions and expert actions.

Based on this reward, this paper designs a regularized Q function (critic) and minimizes the gap between action value and state value.

(2) Guided propagation under limited demonstrations

During initial learning, the agent easily explores regions not covered by demonstrations and cannot achieve supervised actor exploration.

Common approaches include behavior cloning, but when states differ substantially, the policy can still differ greatly from expert actions. This paper therefore uses a nonparametric regression model to propagate experience from limited demonstrations for more stable guidance.

First, sample a small batch of states and actions from the demonstration buffer. Given a current state, search within the small batch of states, use k-nearest neighbors to find the closest states, then approximate the expert policy with locally weighted regression using an exponential sum function.

$$\hat{\pi}^e(s)=\frac{\sum_{i=1}^k\exp\left(-\|s-s^{(i)}\|_2\right)\cdot a^{(i)}}{\sum_{i=1}^k\exp\left(-\|s-s^{(i)}\|_2)\right)}.$$

5 Residual Skill Policies: Learning an Adaptable Skill-based Action Space for Reinforcement Learning for Robotics

Title: Residual Skill Policies: Learning an Adaptable Skill-based Action Space for Reinforcement Learning for Robotics Authors: Queensland University of Technology Venue: CoRL Year: 2022 Code: https://krishanrana.github.io/reskill

5.1 Problem Statement

Skill-based learning has become a way to accelerate robot learning. Skills extracted from expert demonstrations are short sequences of single-step operations (translation, grasping, lifting, and similar actions). These skills are embedded in a latent space and form the action space of an upper-level RL policy. However, this approach has several problems:

• Randomly sampling all skills for exploration is extremely inefficient, because only a small subset of skills is relevant to the task currently being executed, and those relevant skills usually do not cluster in the same neighborhood of skill space.
• The method assumes skills are optimal and that lower-level tasks come from the same distribution as skill space, so learned generality and adaptability to variation are limited—for example, skills learned from moving blocks cannot handle obstacles, object changes, different friction, and similar conditions.

To address the above, this paper proposes the following novel method, called Residual Skill Policies (ReSkill):

• State-conditioned skill prior: sample relevant skills to guide exploration
• Low-level residual policy: achieve adaptation to task variation through fine-grained skill adaptation

5.2 Method

Overall, the method decomposes demonstration trajectories produced by classical controllers into task-agnostic skills and embeds them in a continuous skill space. Using skill space enables genuinely general learning; the upper-level agent can access from the skill space but not actions, reducing requirements on dataset detail.

• Extract skills from existing controllers
• Learn skill embeddings and skill priors
• Train a hierarchical reinforcement learning policy with a low-level residual adaptation policy in skill space.

(1) Data collection

This paper collects demonstration data through manual control (basic manipulation tasks such as pushing and grasping objects). Although the tasks are simple, the trajectories contain complex skills that can be recombined to solve complex tasks.

Trajectories consist of state-action pairs. This paper randomly slices segments of length $H$ for unsupervised skill extraction and uses the extracted actions a and states s to learn state-action in the next subsection.

State s includes joint angles, joint velocities, gripper position, and object position. Actions are continuous 4D vectors including end-effector position and velocity.

(2) Learning a state-conditioned skill space for reinforcement learning

• Embed extracted skills in latent space: use a variational autoencoder (VAE) to embed skill $a$ in latent space. The VAE includes an encoder and decoder. The encoder encodes a full state-action sequence as $z$. The decoder reconstructs actions from current state $s_t$ and skill encoding $z$.
• State-conditioned prior for skills sampled during exploration: learn a conditional probability density over the latent skill space. Traditional Gaussian densities cannot handle multimodal information. This paper uses real NVP, real non-volume-preserving transforms. Learn a mapping from $Z\times S->G$; this mapping can transform from simple distribution G to skill space Z, so f is the skill prior.

Variational autoencoder: a deep generative model Traditional: A traditional autoencoder has an encoder and decoder. After repeated training, input data is encoded into an encoding vector; each dimension of the encoding vector represents a learned feature of the data, and the decoder tries to decode the original input from the encoding vector. Limitation: Traditional methods use a single value to represent how the input manifests along some latent feature. In practice, representing latent features as possible value ranges is more reasonable. Improvement: A variational autoencoder therefore uses probability distributions over values instead of single-value feature representations. Advantage: Each latent feature is represented as a probability distribution. During decoding, sample randomly from the latent state distribution to generate an encoding vector as decoder input. This yields a continuous and smooth latent space representation (values adjacent in latent space reconstruct similar results) Reference:https://zhuanlan.zhihu.com/p/64485020

(3) Reinforcement learning in the state-conditioned skill space

Once training is complete, decoder and skill prior weights are frozen and merged into the RL framework. The high-level reinforcement learning policy $\pi$ is a neural network that maps state to a vector g in the skill prior variation, which is converted to latent skill Z.

The decoder then sequentially reconstructs actions from the current state over skill horizon H. A low-level residual policy also adjusts the decoded skills.

5.3 Summary

This method is a skill-based reinforcement learning approach.

1. Data collection: use the most basic controller to generate trajectories for basic tasks (moving, grasping), then split these trajectories into fixed-length segments; each short segment includes actions and corresponding states.
2. Learn skill space: use a variational autoencoder to encode skills in latent space; use realNVP to map skill latent space + robot state space to a simple distribution space (Gaussian distribution), so skills can be sampled directly from state—called the skill prior.
3. Reinforcement learning: use a high-level policy network to generate a vector from the current state, select a skill from the skill prior (skills related to the current state), and decode it into robot actions with the skill decoder.

6 Watch and Match: Supercharging Imitation with Regularized Optimal Transport

Title: Watch and Match: Supercharging Imitation with Regularized Optimal Transport Authors: New York University Venue: CoRL Year: 2022 Code: https://rot-robot.github.io/

6.1 Problem Statement

Imitation learning today often uses inverse reinforcement learning: given demonstrations, alternately infer a reward function and a policy. However, this approach requires substantial online interaction to solve complex control problems.

This paper proposes Regularized Optimal Transport (ROT), which can adaptively match trajectory rewards with behavior cloning even with only a small number of demonstrations, accelerating imitation.

Optimal Transport (OT)-based imitation learning: Imitation learning, given an expert policy or trajectory, learns a behavior behavior policy $\pi^b$. Inverse reinforcement learning infers from expert trajectory $T^e$ a reward function $r^e$, then optimizes a policy with that reward to obtain behavior policy $\pi^b$. To compute $r^e$, OT-based inverse learning is one approach. Closeness between expert and behavior trajectories can be measured by optimal transport between the two trajectories.

6.2 Method

(1) BC pretraining

Use BC to train a randomly initialized policy on expert demonstration data.

BC corresponds to solving the maximum likelihood problem in the formula. Here expert trajectory $T^e$ refers to expert demonstrations. After training, it enables $\pi^{BC}$ to imitate actions corresponding to those in the demonstrations, but inference fails easily on unseen states.

(2) Online IRL fine-tuning

Fine-tune the policy online on top of the BC-trained model. Because this paper’s tasks have no explicit task reward, rewards are obtained from OT-based trajectory matching. (This paper uses n-step DDPG for continuous control.)

1. Regularized fine-tuning: Because error accumulation during online deployment easily causes distribution shift, this paper regularizes training of $\pi^{ROT}$ by combining $\pi^{ROT}$ with BC loss based on guided RL and offline RL. An adaptive weight $\lambda(\pi)$ controls the contribution of the two loss terms.
2. Adaptive regularization with soft Q-filtering: The adaptive weight $\lambda(\pi)$ is adjusted by comparing performance on a batch of data sampled from a replay buffer between current policy $\pi^{ROT}$ and with-training policy $\pi^{BC}$.
3. Considerations for image observations: apply data augmentation to visual observations, feed images through a CNN encoder, and compute OT rewards to reduce non-stationarity during ROT imitation.

6.3 Experiments

The model in this paper includes three neural networks: encoder, actor, and critic, all trained with mean squared error.

n-step DDPG is used as the RL backbone. The actor is trained with deterministic policy gradients. The critic is trained with clipped double Q-learning, primarily to reduce overestimation, so two Q functions are used for critic learning.