Large language models can encode a wealth of semantic knowledge about the world. Such knowledge could in principle be extremely useful to robots aiming to act upon high-level, temporally extended instructions expressed in natural language. However, a significant weakness of language models is that they lack contextual grounding, which makes it difficult to leverage them for decision making within a given real-world context. For example, asking a language model to describe how to clean a spill might result in a reasonable narrative, but it may not be applicable to a particular agent, such as a robot, that needs to perform this task in a particular environment.
We propose to provide this grounding by means of pretrained behaviors, which are used to condition the model to propose natural language actions that are both feasible and contextually appropriate. The robot can act as the language model’s “hands and eyes,” while the language model supplies high-level semantic knowledge about the task.
We show how low-level tasks can be combined with large language models so that the language model provides high-level knowledge about the procedures for performing complex and temporally extended instructions, while value functions associated with these tasks provide the grounding necessary to connect this knowledge to a particular physical environment.
We evaluate our method on a number of real-world robotic tasks, where we show that this approach is capable of completing long-horizon, abstract, natural language instructions on a mobile manipulator.
Boy Makes Biomimetic Turtle Robot
't came out into plain view. Darkington glimpsed a slim body and six short legs of articulated dull metal.'
Origin F1 Humanoid Robot's Facial Skin
'I could look down at that face of carefully molded synthetic rubber, tinted the exact shade of the doctor's living flesh.' - Rog Philips, 1950.
Technovelgy (that's tech-novel-gee!)
is devoted to the creative science inventions and ideas of sf authors. Look for
the Invention Category that interests
you, the Glossary, the Invention
Timeline, or see what's New.
Reflect Orbital Sunlight On Demand
'I don't have to tell you about the seven two-mile-diameter orbital mirrors that circulate around the satellite, making it habitable.'
Boy Makes Biomimetic Turtle Robot
't came out into plain view. Darkington glimpsed a slim body and six short legs of articulated dull metal.'
Elon Musk Wants Data Centers In Space
'Internally it’s made up of millions of components, but the most important ones are the thinking and memory parts of the Mind proper.'
Origin F1 Humanoid Robot's Facial Skin
'I could look down at that face of carefully molded synthetic rubber, tinted the exact shade of the doctor's living flesh.'