【Motion Imitation】Robot Motion Planning Methods Learned from Human Demonstrations

1 MimicPlay: Long-Horizon Imitation Learning by Watching Human Play

Title: MimicPlay: Long-Horizon Imitation Learning by Watching Human Play Research Team: Stanford University Venue: CoRL Year: 2023 Code: https://mimic-play.github.io/(code is coming soon)

1.1 Target Problem

Humans can complete long-horizon tasks faster than teleoperated robots, which motivates learning robot planning policies from human demonstrations.

To reduce data requirements, videos of humans interacting with the environment are used as data. A hierarchical learning framework is developed to learn latent planning and control methods from human data.

1.2 Method

(1) Learning Latent Plans from Human Data

Given the inputs: visual observation $o_t$, a future goal image $g_t$, and the current hand position $l_t$ During training, $g_t$ is treated as a future frame after the action has been executed. The objective of the planning phase is to generate an action plan for the goal image from the video prompt V.

1. Collect human demonstration data.
2. Track the three-dimensional trajectory of the human hand: use stereo cameras to obtain the hand’s 3D trajectory, apply an off-the-shelf hand detector to locate the hand in 2D images, and then reconstruct the hand’s 3D trajectory from the stereo views.
3. Learn latent plans: use two convolutional networks to process the current image and goal image separately into low-dimensional features, concatenate them with the hand position, and process them with an MLP to obtain latent planning features. A 3D hand-motion trajectory is generated. To handle different ways of performing the same task, a Gaussian mixture model is used to model the trajectory distribution of the latent plans.

(2) Plan-Guided Multi-Task Imitation Learning

The robot’s low-level policy is trained with a behavior cloning algorithm using robot demonstration data collected through teleoperation.

1. Video-conditioned latent plan generation: videos of teleoperated robot tasks prompt the latent planner to generate corresponding plans during training.
2. Transformer-based, plan-guided imitation: observations from the camera on the robot hand and proprioceptive pose information are processed into low-dimensional vectors, concatenated with the latent plan, and passed through a Transformer architecture to compute the final robot control commands.
3. Multi-task learning.

1.3 Thoughts

This work comes from Fei-Fei Li’s team and learns human-hand motion trajectories from videos. The code is coming soon, so I will wait and study it in more detail later.

2 One-Shot Imitation from Observing Humans via Domain-Adaptive Meta-Learning

Title: One-Shot Imitation from Observing Humans via Domain-Adaptive Meta-Learning Research Team: University of California, Berkeley Venue: arXiv Year: 2018 Code: Official version: https://github.com/tianheyu927/mil PyTorch version: https://github.com/daiyk/daml_pytorch

2.1 Target Problem

The paper proposes a method for learning from human videos. By using human and robot demonstration data from a variety of prior tasks, the method enables a robot to perform the task shown in a human demonstration.

2.2 Method

(1) Problem Formulation

Prior knowledge and a small amount of evidence are combined in the form of a human demonstration. The robot’s policy parameters for completing the task are then inferred from it.

(2) Domain-Adaptive Meta-Learning

The method can learn from human video demonstrations. It learns a set of parameters so that, after gradient descent based on a human demonstration, the model can effectively perform a new task.

Because human and robot actions do not correspond directly, the method instead learns an objective that acts only on policy activations.

(3) Learning a Temporal Adaptation Objective

Learning from human videos requires capturing relevant information in the video, such as human intent and task-relevant objects. Determining which behavior is being demonstrated and which objects are relevant usually requires examining multiple frames together to identify human motion. The learned adaptation objective in this paper therefore couples multiple time steps and operates on the policy across multiple time steps.

A convolutional network represents the adaptation objective, while a recurrent neural network, LSTM, performs temporal convolution.

(4) Probabilistic Interpretation

The learned adaptation objective is incorporated into a probabilistic graphical-model framework to infer policy parameters for a specific task.

2.3 Thoughts

The idea looks very promising, but the mathematical derivation is complicated and difficult to understand.

3 Waypoint-Based Imitation Learning for Robotic Manipulation

Title: Waypoint-Based Imitation Learning for Robotic Manipulation Research Team: Stanford University Venue: arXiv Year: 2023 Code: https://github.com/lucys0/awe

Behavior cloning (BC) currently has many problems. Waypoints can address them by narrowing the scope of BC, but conventional waypoints require manually supervised annotation.

This paper proposes an automatic waypoint-extraction module for imitation learning based on linear motion approximation. It decomposes demonstrations into a set of waypoints and uses linear interpolation to approximate the demonstrated motions.

The method can also be combined with task BC algorithms to improve their success rates.

4 Building Robot Intelligence by Scaling Human Supervision

Title: Building Robot Intelligence by Scaling Human Supervision Research Team: Stanford University Venue: Thesis Year: 2021

4.1 Research Background

For decades, we have imagined a world where robots can serve as personal assistants, completing the many tasks and chores we perform every day, such as cooking, cleaning, doing laundry, and even assembling cabinets. Robotics researchers have been working to realize this dream. Unfortunately, today’s autonomous robots remain far from this level of manipulation capability. Research has made impressive progress in enabling robots to complete specific tasks fully autonomously, including picking up objects or stacking them together. Yet a large gap remains between the manipulation capabilities of robots and humans. Humans use objects intelligently and interact with them in rich ways in everyday life, such as using a knife to cut vegetables while cooking or a screwdriver to tighten screws while assembling a cabinet. Such purposeful interaction with objects is extremely difficult for robots.

As humans, we accumulate a range of prior experiences over our lifetimes and draw on them in daily life. Moreover, even when we do not know how to do something, we can learn quickly by watching videos of other people—for example, learning to assemble a cabinet by watching someone else do it on YouTube. This raises a question: can we similarly provide robots with rich and varied prior experience and enable them to learn manipulation skills from these datasets?

This motivates data-driven robotics, a useful paradigm in which robots learn manipulation from large datasets. This approach usually has two variants. In the first, robots collect data themselves. The data are initially random but gradually improve over time. Because the robots must learn on their own, this limits the complexity of what can be learned. In the second, humans control robots and guide them through tasks. However, this is generally not scalable, so only small amounts of data can be collected, which again limits task complexity.

By contrast, fields such as computer data and natural language processing have pioneered unprecedented achievements through large-scale, high-quality datasets, and we hope to see a similar breakthrough in robotics.

To replicate this success and address the limited task complexity of data-driven robotics, we need to solve two key challenges. First, collecting large-scale human data is difficult. In computer vision, humans can annotate data directly, making parallel, large-scale annotation easy. In robotics, by contrast, humans must interact with robots in real time and guide them through tasks. This makes it difficult to provide intuitive and scalable methods for collecting data from many people. Second, learning from large-scale datasets may not be straightforward. In other fields, we can train networks to predict annotations that correspond to ground-truth labels. In robotics, however, there is no single true way to perform a task. Different people may collect different trajectories and follow different policies, so we need to determine how to learn from these datasets.

4.2 Research Objectives

The first part discusses how to collect large-scale data through human supervision rich in interactions, capturing human-like robot manipulation capabilities. This includes a platform built to address the need for large-scale human datasets in robotic manipulation, as well as real-world data collection.

The second part discusses how to use rich datasets to learn robotic manipulation skills.

The third part discusses possible extensions and applications of the method.

4.3 Collecting Human Manipulation Data

For the data to capture human manipulation, they should first be diverse in the types of problem-solving strategies demonstrated. As humans, we clearly understand when we should try different ways to achieve a goal. Robots should learn from all of these strategies because a particular situation may require one of them. Second, the data should contain dexterous manipulation. We want our robots to understand how to manipulate objects through force-based methods to achieve the intended results. Finally, the data should be large-scale. Humans are very good at solving problems in countless situations, but robots cannot yet do so. The more data we show them, the more likely they are to acquire this capability.

In this part, we present the RoboTurk platform, a data-collection platform that allows humans to teleoperate robots in real time. Operators see a video stream of the robot’s workspace in their web browsers and use their smartphones to control the robotic arm. The phone’s motion is coupled with the robot’s motion, allowing the arm to be controlled naturally. This makes it easy for people to provide task demonstrations, and the connection process is fast and simple. Experiments show that these data support policy learning on multi-step manipulation tasks, and that using many demonstrations during policy learning benefits both learning consistency and final performance.

4.4 Learning Manipulation from Large-Scale Human Datasets

In this part, we discuss how robots can learn manipulation skills from large-scale human datasets. Such datasets may exhibit enormous diversity and consist of suboptimal solutions, making them challenging to learn from. We propose a new algorithm for learning from large-scale demonstration datasets, Implicit Reinforcement without Interaction at Scale (IRIS). IRIS decomposes the control problem into a goal-conditioned low-level controller and a high-level goal-selection mechanism. The former imitates short demonstration sequences, while the latter sets goals for the low-level controller and selectively combines portions of suboptimal solutions to complete tasks more successfully.

Although the recent lack of open-source human datasets and reproducible learning methods for imitation learning and reinforcement learning makes it difficult to assess the state of the field, we test six offline-learning robotic manipulation algorithms on multi-stage manipulation tasks in five simulated and three real-world environments of varying complexity. We obtain a set of empirical findings, including sensitivity to different algorithm-design choices, dependence on demonstration quality, and different stopping criteria caused by training toward different objectives. We also highlight the potential of learning from human datasets, such as the ability to learn proficient policies for challenging multi-stage tasks beyond the scope of current reinforcement learning methods, and the ability to scale easily to natural, real-world manipulation scenarios where only raw sensory signals are available. We have open-sourced our datasets and all algorithm implementations to promote future research and fair comparisons of learning from human demonstration data.

4.5 Building More Capable Robots with Human Datasets

This part explores several different applications that bring us closer to the tasks we hope robots will be able to handle in the future. It focuses mainly on multi-task domains, such as kitchens, high-precision manipulation, and multi-arm manipulation tasks that require collaboration.

A common limitation of imitation learning methods is that, due to limited data in the training set, generalizing beyond the demonstrated behaviors remains an open challenge. For example, in a kitchen setting, we may want a robot to achieve many possible configurations involving multiple objects to manipulate and interact with, such as food, cooktops, microwaves, and sinks. In this chapter, we introduce task generalization through imitation, a novel imitation learning framework that enables robots to learn complex real-world manipulation tasks effectively from a small number of human demonstrations. New behaviors are synthesized that were not included in the collected demonstrations. Multi-task domains often exhibit a latent structure in which different task trajectories intersect in common regions of the state space. GTI is a two-stage online imitation learning algorithm that exploits this intersection structure to train goal-directed policies. These policies generalize to unseen combinations of start and goal states. In the first stage of GTI, we train a stochastic policy that uses trajectory intersections to acquire the ability to combine behaviors from different demonstration trajectories. In the second stage of GTI, we collect a small set of inferences from the unconditional stochastic policy in the first stage and train a goal-directed agent to generalize to new start and goal configurations. We validate GTI in simulated domains and in challenging long-horizon robotic manipulation domains in the real world.

Imitation learning methods also often struggle with high-precision manipulation tasks because such tasks require a sequence of precise actions to make meaningful progress, such as inserting a pod into a coffee machine to make coffee. Trained policies may fail in these scenarios because small deviations in actions can cause the policy to enter regions not covered by demonstrations. Intervention-based policy learning offers an alternative for addressing this problem: it allows an operator to monitor a trained policy and take control when failures occur. We extend RoboTurk so that remote operators can monitor and intervene in trained policies. We develop a simple and effective algorithm that iteratively trains the policy on new data collected by the system. We show that agents trained on data collected with our intervention-based system and algorithm outperform agents trained on the same number of samples collected by non-intervention demonstrators. We further demonstrate our method on challenging robotic threading and coffee-making tasks.

Finally, although imitation learning (IL) from human demonstrations collected through teleoperation is a powerful paradigm for teaching robotic manipulation skills, it has mostly been limited to single-arm manipulation. Many real-world tasks, however, require multiple arms, such as lifting heavy objects or assembling a table. Unfortunately, applying IL to multi-arm manipulation tasks has remained challenging—requiring a person to control multiple robotic arms can impose an enormous cognitive load, and usually no more than two robotic arms can be controlled. To address these challenges, we introduce Multi-Arm RoboTurk (MART), a multi-user data-collection platform that allows multiple remote users to teleoperate a set of robotic arms simultaneously and collect demonstrations of multi-arm tasks. Using MART, we collect demonstrations of five new dual-arm and three-arm tasks from several geographically distributed users. We show that learning from these data consequently poses challenges for centralized agents that directly attempt to model all robot actions simultaneously, and we comprehensively evaluate different policy architectures on the data, with varying degrees of centralization for our tasks. Finally, we propose and evaluate a basic residual-policy framework that allows trained policies to adapt better to the mixed coordination settings common in multi-arm manipulation, and show that centralized policies augmented with decentralized residual models outperform all other models on our benchmark task set.

5 Understanding and Learning Robotic Manipulation Skills From Humans

Title: Understanding and Learning Robotic Manipulation Skills From Humans Research Team: Stanford University Venue: Thesis Year: 2022 Code:

5.1 Background and Motivation

The performance of manufacturing robots is measured by their precision, accuracy, and speed. This has led to the design of rigid and bulky robots that are unsafe to operate alongside humans. Their controllers execute preprogrammed trajectories without force sensing, making them highly sensitive to position errors. Uncertainty in the environment can be reduced through the use of fixtures and jigs, such as those on assembly lines.

Real-world environments require lightweight, human-safe, torque-controlled robots. If robots are to be truly useful in everyday environments with constant changes and limited perception, alternatives to controlling them through preprogrammed trajectories must be found. One promising approach is to divide complex tasks into robust and reusable actions or primitives. In this thesis, we lay the theoretical and practical foundations for programming robots at a higher level of abstraction using generalizable compliant primitives.

The first step of the method is to collect data from human demonstrations. We then segment the data into the sequence of actions—primitives—performed during the task. Next, we project the data into a low-dimensional, physically meaningful space that allows us to understand human strategies. Finally, we encode these behaviors into robot controllers capable of performing the task. In addition, our framework uses visual and tactile feedback to keep humans in the loop for failure recovery and continual learning.

5.2 Learning from Human Demonstrations

The work in this thesis falls within the field of Learning from Demonstration (LfD). Humans are highly capable at manipulation, so collecting data from human demonstrations has become a popular way to teach robots new behaviors. In fact, we can learn not only single-arm behaviors but also dual-arm behaviors, as demonstrated by our system and explored in other work. Most prior work has focused on learning from visual data. Our work instead emphasizes the importance of force and data when performing contact-rich tasks.

In recent years, interactive learning has become a growing research field that keeps humans in the learning loop. To operate in a similar way, our framework keeps humans in the loop through a haptic interface. The failure-recovery data collected by our system can be combined with work on learning from failure to produce more robust autonomous behavior.

5.3 Robot Primitives

In this research, a primitive is defined by a compliance frame and a set of desired task parameters. A compliance frame consists of an origin and three directions in space along which we control motion and compliance. The compliant frame is attached to the object being manipulated. The task parameters include the desired force, torque, position, and orientation. This robot-agnostic task specification provides a physically meaningful, low-dimensional representation.

Primitive library. A library consisting of n primitives is generated to encode basic manipulation skills. By combining these primitives, new complex tasks can be solved in such a way that the number of required primitives does not increase with the number of tasks. Broadly speaking, the literature on motion primitives has mainly addressed three major research questions: generating motion primitives, parameterizing primitives, and combining primitives to complete tasks successfully.

Primitive generation. Primitives can be generated either by manually encoding the desired policy or by automatically extracting the policy from data. Prior research has turned to humans for inspiration and attempted to extract their strategies.

Primitive parameterization. Primitive parameterization concerns how actions are defined. Primitives are usually parameterized with trajectory segments. Defining motion primitives with trajectories has been shown to be successful, but this approach assumes low environmental uncertainty.

Compliant primitives. We use the concept of a frame to parameterize our primitive controllers. Prior research has used the same concept for object-centered task controllers. Unlike the work in this thesis, however, those prior approaches extracted controller parameters only from visual data. We believe that considering sequential data is advantageous when handling complex tasks. Compliance also plays an important role in tasks involving positional uncertainty and in contact-rich tasks, such as capture or the tasks studied in this thesis. One example is the concept of primitives in which compliant primitives are used to achieve robust grasping of small objects.

Planning with primitives. Probabilistic methods for combining motion primitives exploit the inherent uncertainty of the decision-making process. These methods may be fully automated, or they may use a hybrid approach that combines an automatic decision algorithm with user-specified images. Other recent methods use semantic models to learn manipulation task plans based on perception, while some methods use contact rather than vision to guide the decision-making process.

5.4 Multi-Layer Control Architecture

The architecture consists of three layers of perception-action feedback loops. Each layer operates at a different level of abstraction and at a different frequency.

The lowest layer relies entirely on the controller and facilitates the integration of high-rate sensory feedback and control to achieve safe and predictable robot motion. This level sends commands to the robot motors, so the perception-action loop must be closed at a very high frequency. The next layer provides inputs to the whole-body controller and therefore operates at a slower rate. Finally, the highest level of abstraction, which performs computationally expensive perception and planning, operates at the lowest rate.

The whole-body control level uses task priorities. Priority-based control allows us to focus on objects and their geometric constraints when designing primitive actions. A complete robot behavior can be viewed as a set of different tasks with different priorities. For example, a high-priority task may be to avoid singular configurations, while another task may handle obstacle or friction constraints. Similarly, one task is specifically responsible for achieving the desired geometric relationships among manipulated objects. This task is encoded as a primitive. In other words, the primitive action concerns only the objects, because all other aspects of the task, including non-geometric constraints and robot kinematics, are handled by other controller components.

Priority-based whole-body control uses null-space projection to ensure that all different constraints are satisfied. In addition, the operational-space formulation—which uses the dynamically decoupled inverse of the Jacobian to compute recursive null-space projections—ensures that tasks with different priorities are dynamically decoupled. Prior work has also used this hierarchical framework in the context of manipulation primitives.

6 Scaling Deep Robotic Learning to Broad Real-World Data

Title: Scaling Deep Robotic Learning to Broad Real-World Data Research Team: Stanford University Venue: Thesis Year: 2023 Code:

6.1 Background

A longstanding dream in robotics is a general-purpose household robot that can be placed in a home environment—perhaps one it has never seen before—and perform a range of useful tasks, such as making coffee, cleaning, and cooking. Such a robot would undoubtedly have a tremendous impact, both economically and by improving people’s quality of life through its assistance. Of course, this dream remains a dream. Achieving it presents countless challenges, including better robot hardware, battery technology, and sensing. One of the central challenges, however, is generalization: a robot’s ability to succeed with new objects, environments, and tasks. Humans have precisely this ability, and it enables us to complete tasks such as making a cup of coffee even in a new kitchen with new objects. The relevant question therefore remains: how can we train our robots to generalize broadly?

One way to address this problem is to use human intuition, along with hand-designed systems and representations for robot planning and control. Under this approach, humans define the relevant object classes and their properties and relationships—for example, color, shape, pose, above versus below, and so on. State-estimation techniques can then measure these quantities from sensor observations, and classical search and motion-planning methods can execute the task. Crucially, however, this approach depends on human specifications of relevant objects and properties and, in some cases, even 3D models of the objects in every environment and task. This prevents the approach from being used easily in the presence of new objects and environments.

Some work has studied how robots can learn behaviors purely from data and their own trial and error when building general-purpose robots, without relying on human intuition or hand-designed systems. Specifically, deep reinforcement learning studies the problem of learning deep neural-network policies that take actions given sensor observations, enabling the policy to maximize a defined reward by learning from interaction. In principle, this approach could allow robots to learn skills entirely on their own with only a small number of success metrics. In practice, however, running deep reinforcement learning on robots introduces many challenges, such as the need for human supervision during resets and reward assignment. Most critically, deep reinforcement learning typically requires millions of online interactions with the target environment to learn, and once learning is complete, the learned policy is valid only for the environment and data on which it was trained. Thus, while standard deep reinforcement learning eliminates much of the need for human-designed systems, it still does not immediately bring us closer to robots that can operate in new environments and perform new tasks.

Taking a step back, we can look for inspiration in other areas of machine learning. In particular, natural language processing and computer vision have made tremendous progress over the past few years, largely through a simple recipe: (1) large and diverse offline datasets, (2) self-supervised or inexpensively supervised training objectives, and (3) expressive, end-to-end trained neural-network models. This foundation-model paradigm is especially exciting because these models demonstrate impressive generalization. For example, visual models from ImageNet can adapt to entirely new tasks such as cancer detection, while pretrained language models such as BERT have applications ranging from medical coding to visual question answering. This level of generalization is precisely what we hope to see in a general-purpose robot that can be placed in a new environment and quickly learn to solve new tasks.

Why, then, has this recipe not yet been realized in robotics? Real-world robotic manipulation presents many unique challenges that make it difficult to reproduce the recipe directly: we lack sufficiently large and diverse datasets of robot interactions, and it is unclear what types of learning algorithms or supervision can enable us to learn useful skills from these datasets at scale. The goal of this thesis is to address these challenges and reproduce the recipe of large-scale data and learning in the context of robotic manipulation. Specifically, my research focuses on answering three broad questions. First, how can we scalably collect large and diverse datasets of robots interacting with the physical world? Second, how can we design self-supervised reinforcement learning algorithms that can consume such broad offline data—which may come from non-experts and lack reward labels—and learn from them to reach unseen goals? Third, how can we unlock the broad sources of data available on the internet, such as human videos and natural language, to enable more effective learning in our robots?

7 Learning Perceptual Prediction: Learning From Humans and Reasoning About Objects

Title: Learning Perceptual Prediction: Learning From Humans and Reasoning About Objects Research Team: University of Pennsylvania Venue: Thesis Year: 2023 Code:

7.1 Target Problem

Humans use a wide variety of perceptual modalities to make predictions, but this work focuses primarily on learning from vision. Human vision appears to be highly optimized for predicting future visual observations.

Studying prediction with visual sensors also offers many practical advantages. First, high-quality cameras are readily available and have low size, weight, and power requirements, allowing them to be included on most robot platforms. Because cameras are widespread in non-robotic applications, they have become commoditized. Second, visual observations provide rich information about the environment, including pose, texture, and semantics, that other sensors cannot easily match. Obtaining large amounts of rich information about the world is important for enabling agents to interact with it.

Current methods for learning action-conditioned visual prediction models rely on access to large amounts of specific data. Collecting these data is expensive and time-consuming, which prevents visual prediction methods from being used in many applications. This is especially true in robotics, because collecting large amounts of robot data is expensive, time-consuming, and potentially unsafe. Existing work shows that visual prediction methods scale well with increasing amounts of data, so finding new data sources is essential to making these models broadly usable.

In this thesis, I propose three different methods for using non-robotic data to improve visual prediction and robot control. In the first two works, I use human data to improve robot performance, while in the third, I use an existing non-robotic dataset to create an object-centered prediction framework.

7.2 Learning from Humans

Large and diverse real-world datasets are essential for broad generalization and high performance. Large datasets can be collected through automated pipelines or human teleoperation. Automated data-collection processes can gather very large datasets, but they have difficulty reaching interesting parts of the environment and require extensive interaction with the environment. Learned exploration policies can improve an agent’s ability to reach interesting configurations, but these methods still require a great deal of exploration. The second approach is to collect teleoperated trajectories of human demonstrations. This allows datasets to reach interesting and task-relevant configurations easily. However, it is limited by its reliance on humans to operate robots, which requires trained operators and quickly becomes very expensive. Inspiration from how humans learn suggests an alternative that avoids the difficulties of both approaches.

Humans can learn skills not only through their own interactions with the world, but also by observing others. Consider an infant learning to use a tool. To use a tool successfully, the infant needs to learn how the tool interacts with other objects and how to move the tool to trigger that interaction. This intuitive understanding of physics can be learned by observing how adults use tools. More generally, observation is a powerful source of information about the world and about how actions lead to outcomes. Yet when embodiments differ—for example, between an adult’s body and an infant’s—using observations is challenging because there is no direct correspondence between the demonstrator’s and observer’s actions. Evidence from neuroscience suggests that humans can effectively infer these correspondences and use them to learn from observation.

Using observations of humans provides an opportunity to increase the scale and usefulness of available data dramatically. Unlike autonomously collected data, human data can focus exclusively on interesting parts of the configuration space and avoid dangerous or boring interactions. Unlike datasets collected through teleoperation, human datasets can be much larger. Publicly available human datasets such as Ego4D or SomethingSomething contain hundreds of thousands of videos and thousands of hours of footage across hundreds of tasks and dozens of locations. These datasets are comparable in size to autonomously collected datasets and are one to two orders of magnitude larger than datasets collected through teleoperation. More importantly, learning from actionless observations of humans opens the possibility of learning from publicly available videos on the internet, such as videos on YouTube, which offer several additional orders of magnitude of data. Current methods can use only limited subsets of these data for specific tasks, but our work takes a step toward broader use.

In this thesis, we consider the following question: can an agent learn to use both its own interactions and passive observations of other agents to solve tasks? We explore this question in two settings: the first involves learning action-conditioned visual prediction models, and the second involves end-to-end reinforcement learning policies.

In Chapter 3, we propose a method that uses actionless human data and the agent’s own exploration to perform reinforcement learning. We propose methods for overcoming the domain shift between in-the-wild human data and simulated robot data, adding actions to actionless human data, and estimating rewards for human data. By using human videos collected in the real world, we are able to accelerate the learning of simulated robot agents.

7.3 Object Reasoning

Although learning from human observations can provide access to large amounts of data, it does not answer the question of what should be learned. We focus on learning end-to-end models that map directly from sensor inputs to predicted future frames or desired actions, with little internal structure. By choosing the correct set of inductive biases for a task and using existing models pretrained on non-robotic data, we should be able to train our models with less data and achieve higher performance. Properly structuring the learning problem can also make models easier to use for downstream tasks. We focus on the simple inductive bias that assumes the world is made up of objects.

Most dynamic interactions in manipulation can be modeled by decomposing a scene into objects. Although some materials, such as liquids or granular media, cannot be represented easily as objects, most manipulation tasks involve manipulating discrete objects. Sorting boxes, rearranging rooms, and even drinking a cup of coffee are all tasks governed primarily by interactions with discrete objects.

Object-centered prediction models have demonstrated successful performance in prediction and difficult tasks. By decomposing a scene into discrete objects, these prediction models can preserve the coherence of each object over longer time horizons. Furthermore, by internally representing world state as a set of objects, object-centered prediction models can interface easily with planners and provide a highly interpretable interface that aids debugging and validation.

8 Affordances from Human Videos as a Versatile Representation for Robotics

Title: Affordances from Human Videos as a Versatile Representation for Robotics Research Team: CMU, Meta AI Venue: CoRL Year: 2023 Code: https://robo-affordances.github.io/

8.1 Target Problem

The goal is to learn actionable motion representations from human videos. Supervised by future frames, the model predicts contact points and trajectory waypoints.

The paper focuses primarily on three questions:

1. How should affordances be represented?
2. How can this representation be learned in a data-driven and scalable way?
3. How can visually inspired methods be deployed across different robots?

In response to these three questions, the paper presents the following three perspectives:

1. Contact points and post-contact trajectories are effective representations for robotic manipulation.
2. Egocentric datasets are used, focusing on all frames that contain humans, to predict contact points and post-contact trajectories. Off-the-shelf tools are used to estimate ego-motion, human pose, and hand-object interactions.
3. A method called Vision-Robotics Bridge(VRB) is implemented, enabling these functions to integrate seamlessly with different types of robots.

8.2 Method

(1) Affordance Representation

The most natural way to extract inspiration from humans is to observe how people interact with the world. The conventional idea is to model human motion accurately from video, but this produces a human-centered model that does not generalize easily because human and robot morphologies are completely different.

This paper therefore starts from first principles driven by robot requirements. Information about the robot’s embodiment is usually known, so using motion planning to reach a point in 3D space is easy. The key difficulties are determining where to interact with the environment and how to move after contact.

Inspired by this observation, the method uses the contact point $c$ and post-contact trajectory $\tau$ as a simple representation of manipulation inspired by vision that can be transferred easily to robots. Here, $\tau=f(I_t,h_t)$, $I_t$ is the image at time step $t$, and $h_t$ is the human-hand position in pixel space.

(2) Learning Manipulation from Egocentric Videos

The next question is how to process the human body or hand in the visual input and extract contact points $c$ and trajectories $\tau$ from human videos.

1. Extracting manipulation from human videos

For a given video $V$, such as one showing a person opening a door, an existing hand-object detection model generates a 2D bounding box for the hand and a discrete contact variable for each image frame $I_t$, denoted by $o_t$. Using this information, we can filter for frames in which $o_t$ indicates contact and thereby find the first time step at which contact occurs, $t_{contanct}$.

The hand’s position in pixel space forms the post-contact trajectory $\tau$. To extract the contact points, we use the hand bounding box and color segmentation to find points where the hand intersects the bounding boxes of other objects, then fit these contact points with a Gaussian mixture model.

We must also account for the fact that when a person opens a door, not only does the hand move, but the camera also moves. Camera motion must be compensated for. A matrix is used to address this issue, and a homography matrix is obtained by matching features between consecutive frames, producing a transformed trajectory.

A visual transfer must also be completed. The training videos contain human hands, whereas the views in robot tasks do not. All affordances are therefore mapped back to the first frame, before the human has entered the scene. If a person is always present in the video, the person is either cropped out or the video is discarded.

2. Training the model

Conditioned on an input image, the model is trained to predict contact points and post-contact trajectories. The learning task is multimodal: for example, a person picking up a cup from a table may intend to drink from it or pour its contents elsewhere. The method therefore models a spatial probability distribution and predicts multiple heatmaps to address this issue.

The input image is encoded with ResNet to produce a latent-space representation. Convolutional layers then project this latent representation into K probability segmentations, yielding estimates of the labels for the GMM means.

To estimate post-contact trajectories, the paper uses Transformer-based prediction. Given a scene, a human may interact with many objects that might not exist in the training data. We address this by sampling around the contact points, achieving better generalization.

(3) Robot Learning

The method is used to guide existing robot learning approaches, with four different robot modes considered.

1. Imitation learning with offline data collection

Given an image input, the model produces a contact point and trajectory, and we store this set of data in a dataset. After enough data have been collected, we use imitation learning to control the policy and perform a specific task.

2. Reward-free exploration

The goal is to discover as many different skills as possible. In practice, however, exploration from scratch is too inefficient because the robot may spend a great deal of time trying to explore yet still fail to learn meaningful skills that solve tasks humans care about.

We first collect data and then rank all trajectories. Subsequent data collection begins by using highly exploratory trajectories for guidance, enabling further exploration.

3. Goal-conditioned learning

Known knowledge, such as an image of an open door, is used to supervise exploratory learning.

4. Affordances as the action space

Robot manipulation in continuous space is parameterized spatially, with one primitive assigned to each position. A large number of predictions are obtained through learning, and a GMM is fitted to these trajectory points to obtain discrete contact points and trajectories. The robot then needs to search only within this space.