Contents
  1. Robot Foundation Models
  2. Google Brain-Related Work
  3. Past Work
  4. Recent Work

Video Link: https://www.youtube.com/watch?v=ct4tdyyNDY4

Over the past two years, developments in large language models, speech technology, and vision technology have brought about a 180-degree shift in robotics research.

Robot Foundation Models

When large-scale models are trained on massive amounts of data, they usually exhibit two relatively important properties:

  • Emergence: When something very simple works at a small scale, its performance improves as the scale increases—for example, with more data and larger models.
  • Homogenization: A model can combine many downstream tasks to achieve generalization.

Of course, we may wonder why there is currently no foundation model for robotics. Unlike fields such as audio, language, and images, which already have relatively general-purpose foundation models, no one has yet developed a foundation model for robotics.

So how could we build a foundation model for robotics?

(1) Design Reinforcement Learning Algorithms

  1. Use high-performance architectures such as self-attention
  2. Apply scaling laws: scale not only the model size, but also computation, the dataset corpus, and the number of tokens
  3. Dataset size is more important than data quality

(2) Internet-Scale Model Diffusion

  1. Generative models have already demonstrated emergent capabilities in language, programming, vision, audio, and other fields, repeatedly exceeding our expectations. This trend will continue for the foreseeable future.
  2. Emergence and acceleration mean that these models can become better “on their own”

(3) Shift from Online Robot Learning to Offline Learning

  1. Current large models are all trained on enormous offline datasets

Past Work

(1) 2016 - Robot Farm

  1. Reinforcement learning: Google built a robot farm containing 7 Kuka robotic arms. They performed grasping actions 7x24 hours for reinforcement learning training.
  2. Qt-Opt: A Q-learning algorithm that accepts visual input while performing continuous control.
  3. RL-CycleGan: Converts images from a simulated environment into real images, enabling robots to perform tasks better in the real world.

(2) 2020 - Kitchen Environment

  1. BC-Z: Multi-task imitation learning
  2. AW-Opt: Combines reinforcement learning with imitation learning guidance

(3) 2022

Some problems emerged. For example, success rates in some scenario-based tasks had stabilized at 50~70%. Some methods required highly specific data distributions, and if a policy had not been trained on the current data, the task was very likely to fail. To solve this problem, Google carried out the following work:

  1. Multi-task imitation learning
  2. Use of large-scale datasets

Recent Work

(1) RT-1

The main focus was on how to scale imitation learning.

  1. Over a year and a half, 100k demonstrations covering 700 tasks were collected from 13 robots
  2. BC-Z was used for training
  3. Because Transformer-based algorithms are not yet sufficient to support the high-frequency, real-time requirements of robot learning when processing images, the hope is that the dataset can understand language models.

At a high level, RT-1 is a Robotics Transformer. It accepts visual input from a robot camera as well as natural-language instructions, uses only transformer for decoding, separates target object categories, and uses a pretrained EfficientNet backbone to obtain discretized actions.

(2) SayCan

A robot’s skills can be limited, and large language models are also constrained: they do not know the robot’s state or the surrounding environment.

This work therefore focuses mainly on enabling language models to speak the language of robots.

(3) Inner-Monologue

It incorporates feedback from the dynamic environment into the closed loop and also uses a language API to communicate what is present in the environment.