Wayfarer lets AI agents discover their own skills and master Atari games

What Wayfarer is and who built it

The work is described in the paper Mastering Atari 2600 Games with Discovered Options. It was co-authored by Erik M. Lintunen, recruited under the Canada Impact+ Research Training Award, and his supervisor Marlos C. Machado, an Amii Fellow and Canada CIFAR AI Chair.

The pair present Wayfarer as an autonomous decision-making architecture that can master complicated behaviours using only the agent's data stream. According to Amii, the approach promises agents that are better at long-term decision-making and can handle more complex tasks efficiently without constant human intervention.

The problem options are meant to solve

Reinforcement learning agents learn by trial and error, but they can struggle with tasks that need many small steps. Amii uses the example of an agent navigating a large house, where every step is a decision about direction, speed and even whether to move at all. This can produce choices that make sense in the short term but are inefficient for the long-term goal, along with time-consuming dithering at every decision point.

Agents may also lack the purposeful behaviour needed to try something different, such as going to another floor. They can struggle to work out which of millions of decisions was responsible for a success, and what they learn is not necessarily easy to carry over to a new situation.

Options address this by grouping smaller steps into a single decision. Instead of making a hundred small choices to cross a room of obstacles, the agent makes one strategic choice, like "get across the room." Amii describes this as a skill. According to the source, options have long been promising, but until now they largely had to be crafted by humans, or refined by agents using human feedback.

How the three networks work together

Wayfarer is a domain-agnostic algorithm that combines three networks:

• Representation network: takes in the environment's observation and builds a representation of it, helping the agent distinguish what varies slowly versus quickly, what seems likely to happen or not, and what it can or cannot control.
• Skill network: builds a collection of options from that representation and from the agent's experience.
• Task-solving network: evaluates the challenge and decides which actions and options would provide the maximum rewards.

The researchers call the interaction a "virtuous loop". The representation defines internal goals used to build options, the agent gains experience from the options it uses, and it then learns from that experience to build a better representation and stronger options. Amii describes this as learning in a constructivist way, by actively building knowledge from experience.

Results on hard Atari games

The team tested Wayfarer on the Atari 2600 challenging set, ten difficult games used to benchmark reinforcement learning agents. Many of them, including Montezuma's Revenge and Private Eye, require long-term planning, exploration and strategy.

According to Amii, Wayfarer achieved a median performance boost of 2.5 times over learning without options. It also outperformed established baselines such as IQN and Rainbow and matched heavy model-based systems. This was done without human intervention or designed options, as the agent discovered its own.

What it could mean

Machado, who has spent a decade working on the potential of options in RL agents, says Wayfarer fulfills the promise of options, perhaps for the first time. Amii says the work could lead to better autonomous agents for complex tasks where human guidance is not possible or feasible, from industrial robotics to autonomous vehicles, using fewer resources and less computing power. The source reports results on Atari benchmarks only, so those applications remain prospects rather than demonstrated outcomes.