EAGLE-4 speeds up prompt enhancement for image and video models

Why prompt enhancement needs fast decoding

Large language models generate text one token at a time. At small batch sizes this leaves GPU compute underused, because the operations are limited by memory bandwidth. Speculative decoding tackles this by letting a lightweight model draft several tokens, which the target model then verifies together. According to the source, with the appropriate verification procedure the target model's output distribution is preserved.

fal argues that the method matters for image and video generation. Many such systems first run the user's prompt through a step called prompt enhancement or prompt expansion, which turns a brief request into a more detailed scene description and adapts it to the target model's expected format. As generative models get faster, this step can take a larger share of total latency. fal notes that some video models can generate video in real time, yet an unoptimized expansion stage would take longer than the generation itself.

To measure this use case, the authors are releasing Prompt Enhancement SpecBench. It holds 400 examples from 18 unique datasets across text-to-video, image-to-video, text-to-image and image-to-image, with 100 examples per category.

How EAGLE-4 works

EAGLE-4 builds on EAGLE-3. A lightweight drafter combines the target model's hidden states with token embeddings to predict possible continuations. The released model, built for Qwen 3.8 27B, makes two changes to the drafting path:

• Sliding-window attention bounds the context used by the drafter's attention layer.
• A low-rank candidate head first scores the vocabulary in a smaller representation, then computes full logits only for the selected candidates.

These target the cost of attention and next-token prediction respectively. Multiple draft layers, mixture-of-experts support and Latent ReLU are not yet in the released model, and fal says it expects future models to use them.

The verification side uses a tree instead of a single chain, so the target model can check alternative continuations in one forward pass. A custom attention mask keeps each branch's context separate. A harness manages the tradeoff between tree size and cost. After a one-time offline calibration on the target GPU type, it builds a cost table of verify latency across batch size, context length and number of candidates. It widens or deepens the tree while the estimated probability gain per marginal cost stays above a threshold. The result, according to fal, is a wide, deep tree at batch size 1 or long context, and a narrow tree or plain chain at large batch, from one drafter with no fixed depth or top-k settings.

Benchmark results

All experiments ran on NVIDIA B300 GPUs with tensor parallelism set to 4. The evaluation used fal's 400-prompt expansion dataset and a Fireworks curated dataset, which yielded 138 usable prompts. The chain-versus-tree comparisons use all 138 Fireworks prompts and the 200 text-only fal prompts.

Creative prompt expansion proved harder for drafters. Acceptance length was 2.66 in the highest-entropy quartile of prompts, against 2.97 in the lowest. The tree improved acceptance length across every task split, with an aggregate relative gain of 9.6% on the Fireworks dataset and 12.4% on the fal text-only subset.

The advantage grew with the verification budget. At 16 tokens, the largest tested, the tree beat the chain by 21.1% on Fireworks and 26.1% on the fal subset. The chain showed diminishing returns, particularly on fal's data. Per-position analysis showed the gain concentrated early: on the Fireworks dataset the first-token hit rate rose from 72.5% to 85.3%, an increase of 12.8 percentage points.

The source text also compares decoding throughput of DSpark, DFlash2, the EAGLE-4 static chain and the EAGLE-4 Harness, but the available excerpt ends before those results.

What it means

For teams building interactive image and video tools, the work points to prompt enhancement as a latency bottleneck that speculative decoding can address. The public benchmark gives others a shared way to test drafters on open-ended creative text, which the results show is harder than the general workloads in the Fireworks set.