Apple trains small UI agents on live iOS apps with online RL
Apple researchers have described an online reinforcement learning framework that trains UI navigation agents directly on live iOS applications. According to the paper, their 8B agents tie with Gemini-3.1-Pro on real iOS apps and outperform Gemini-3.5-Flash.
- Apple researchers propose an online RL framework that trains UI navigation agents on live iOS apps.
- The agent is split into a Planning agent and a UI Grounding model, trained with PPO and dense plus terminal rewards from an LLM-as-a-Judge.
- GiGPPO is a neighbor-conditioned value estimation method meant to stabilize the critic.
- 8B agents show nearly a sevenfold gain over the SFT baseline in out-of-distribution generalization to Android.
- On real iOS apps the 8B agents beat Gemini-3.5-Flash and tie with Gemini-3.1-Pro, according to the paper.
The problem with cheap UI agents
The paper, titled "Online Dense Reinforcement Learning with Planning-Grounding Agent for UI Navigation on Live iOS Apps", starts from a trade-off. According to the authors, frontier vision-language models are strong at navigating user interfaces, but their high inference costs prohibit everyday use.
Smaller models are cheaper, yet the authors say those trained through supervised fine-tuning (SFT) or offline reinforcement learning often memorize static trajectories. The result, they write, is brittle execution and an inability to recover from errors. The new framework is meant to close this gap by training agents online, directly on live iOS applications.
The work is credited to Peitong Duan, Seyed Khashaiar Gatmiry, Jeff Nichols, Emmanuel Abbe and Samy Bengio. Duan and Gatmiry contributed equally.
How the framework works
The system splits the agent into two specialized components: a Planning agent and a UI Grounding model. Training relies on an LLM-as-a-Judge that supplies reward signals of two kinds:
- Dense, step-level feedback that evaluates intermediate progress.
- A terminal reward for overall task completion.
To make use of both signals, the team optimizes the agent with Proximal Policy Optimization (PPO) using step-level advantage estimation. The authors also introduce GiGPPO, which they describe as a neighbor-conditioned value estimation method. It improves critic stability by drawing on reward signals from visually similar states across concurrent trajectories.
Training online at this scale needs specific infrastructure. The paper says the team built a system that orchestrates hundreds of concurrent virtual iOS devices.
Benchmark results
The authors evaluated their agents on UINavBench, iOSWorld and AndroidWorld. According to the paper, the online RL agents substantially outperform both SFT and terminal-only RL baselines, and they also speed up training convergence.
Two results stand out in the abstract. First, the 8B RL agents achieve nearly a sevenfold improvement over the SFT baseline in out-of-distribution generalization to Android environments, even though the training took place on iOS. Second, on real iOS applications the 8B agents outperform Gemini-3.5-Flash and tie with Gemini-3.1-Pro.
These figures come from the authors' own evaluations as reported in the abstract. The source text does not give the underlying scores.
What it means
The results suggest that a compact model trained on live apps with step-level rewards can approach the performance of much larger systems on UI tasks, which speaks directly to the cost barrier the authors identify. The paper also points to better recovery from errors and stronger generalization than SFT-only training, two weaknesses it attributes to static-trajectory approaches.
The source is a research paper abstract, so it does not mention any product plans or availability.
