MIT’s FLEX Helps Robots Infer More Than Human Commands Say

From a specific warning to a broader rule

FLEX was demonstrated in simulated scenarios involving a robot arm serving coffee, moving a candle and carrying a bowling ball. In one example, a person types, “Don’t spill on the laptop!” As the robot moves the coffee cup across a cluttered table, FLEX reasons that the concern extends beyond the laptop to other electronics, including a nearby phone.

The platform uses a large language model, or LLM, to identify patterns and connections related to the features a person has mentioned. The researchers used GPT-4o for this part of the system. According to MIT, the model can connect examples such as liquid damaging a phone or computer with a broader instruction to avoid electronics.

This allows the robot to adjust its path without requiring the person to list every individual object or rule. The approach is intended to help robots interpret an underlying preference from limited natural language commands.

How FLEX works

FLEX is a three-step process called “Add,” “Generalize” and “Mask.” In the first step, the robot adds an object or feature mentioned in a command to a list of things that matter for the task. For example, if a person says, “Don’t drop it on my mug,” while the robot is moving a bowling ball, the mug becomes something the robot should avoid.

In the second step, “Generalize,” GPT-4o considers the features identified by the person and searches for broader patterns. A mug, for example, can lead the system to generalize that fragile objects should be avoided, even if the person did not state that rule directly.

The final step, “Mask,” uses the robot’s standard computer vision system to determine which objects are no longer present. FLEX then removes those objects from the feature list. MIT says the full three-step process takes about five seconds.

Toward more practical household robots

The study was led by Andreea Bobu, an assistant professor of aeronautics and astronautics at MIT. Nathan Dennler, a postdoc in MIT’s Computer Science and Artificial Intelligence Laboratory, is a study author. Audrey Lee SM ’26 and Ilia Sucholutsky of New York University are also co-authors.

Bobu leads MIT’s Collaborative Learning and Autonomy Research Lab, which develops autonomous agents that learn to work for, with and around people. The researchers say current robot systems can use language to clarify a task, redirect a trajectory or update what they consider a safe path, but FLEX expands the information a robot can draw from those commands.

The work, titled “Flexible Language-Guided Representation Editing for Adaptive Robot Behavior,” is to be presented at the annual Conference on Robot Learning this month. The researchers say the system could support more intuitive household and assistive robots by letting people communicate preferences in ordinary language rather than learning how to provide detailed instructions.