Amazon and Booking.com shrink LLMs to run up to 20 times faster

Why travel platforms need faster models

Travel platforms serve millions of daily users who ask about destinations, seek hotel recommendations and plan trips. Many have turned to large language models (LLMs) for these tasks. According to the Amazon Science post, such models give accurate and contextually relevant advice thanks to their size, typically tens of billions of parameters.

That size comes at a price. The models are computationally expensive, and the expense grows with the number of customer queries. They can also be too slow for web and mobile applications, where, in the authors' words, a delay of even 100 milliseconds can cost a company customers and revenue.

The authors, Daniel Zagyva and Laurens van der Maas, both of AWS, worked with Booking.com on three features. The AI Trip Planner is a conversational assistant that must extract travel preferences from dialogue in less than a second. Smart Filters converts natural-language search queries into structured filters in real time. Accommodation Description Generation produces property descriptions across millions of listings, where cost matters most.

Two techniques, one pipeline

The team combined two methods. Knowledge distillation (KD) transfers knowledge from a large teacher model to a smaller student model. The teacher is fine-tuned on labeled data and then generates predictions on unlabeled, domain-relevant text. The student is trained on the combined human-labeled and machine-labeled data and learns to mimic the teacher.

Medusa targets inference time. An LLM normally generates one token at a time. Medusa adds prediction modules, called Medusa heads, to the base model, one for each token decoded in parallel. Because GPU hardware is optimized for parallelism, this adds little overhead. According to the authors, Medusa typically offers a 2-4-fold speedup without compromising generation quality.

In the unified pipeline, Medusa heads are attached to the student during training on the combined corpus. The result is a single compact model that inherits the teacher's knowledge and can predict multiple tokens at once.

What the experiments showed

The team used TinyLlama, with 1.1 billion parameters, as the student and Llama-3.1 with 70 billion parameters as the teacher. They ran four progressive experiments, each building on the last.

• Fine-tuning TinyLlama on task-specific data for each use case made it significantly more accurate than few-shot GPT-4o, but it still fell short of the teacher.
• Knowledge distillation closed that gap by more than 90%, bringing the small model near parity with the teacher, and made inference roughly 6 times as fast.
• Adding Medusa heads sped up inference by a further 2 to 3.6 times.

Two Medusa variants were tested. With Medusa-1, the base LLM is fine-tuned first, then its weights are frozen and the heads are fine-tuned separately. With Medusa-2, everything is trained in a single stage. Both achieved comparable speedups with no meaningful quality loss, but the authors say the simpler pipeline of Medusa-2 made it more practical for production teams managing multiple models.

Results for Booking.com and what it means

According to the authors, the final KD plus Medusa-2 model matched the quality of the 70-billion-parameter teacher while running 10 to 20 times as fast, depending on the use case. A dedicated model was fine-tuned for each feature.

• AI Trip Planner: a 2.9% increase in user engagement versus GPT-3.5.
• Smart Filters: responses 15 times as fast for 99 percent of queries.
• Accommodation descriptions: 1.4% more "helpful" ratings from users, at less than one-third the cost of GPT-4.

The authors say the combined approach maintains the quality of the best open-source LLMs at less than one-third of the cost while running up to 20 times as fast. They argue that travel platforms and other applications relying on LLMs can use the framework to deliver fast, cost-effective and accurate responses in real-world scenarios. The work was presented at the Association for Computational Linguistics (ACL) meeting, where it received the Industry Track best-paper award, according to the post.