Speeding up language models for travel applications
Captured source
source ↗Speeding up language models for travel applications - Amazon Science
Close
Close
Social
bluesky
threads
youtube
github
rss
Menu
Research
Research areas
Automated reasoning
Cloud and systems
Computer vision
Conversational AI
Economics
Information and knowledge management
Machine learning
Operations research and optimization
Quantum technologies
Robotics
Search and information retrieval
Security, privacy, and abuse prevention
Sustainability
Our scientific contributions
Publications
Research from our scientists and collaborators.
Conferences
Our experts present and discuss cutting-edge research at scientific meetings globally.
Research areas
Automated reasoning
Cloud and systems
Computer vision
Conversational AI
Economics
Information and knowledge management
Machine learning
Operations research and optimization
Quantum technologies
Robotics
Search and information retrieval
Security, privacy, and abuse prevention
Sustainability
Our scientific contributions
Publications
Research from our scientists and collaborators.
Conferences
Our experts present and discuss cutting-edge research at scientific meetings globally.
News & blog
The latest from Amazon researchers
Amazon Science Blog
Technical deep-dives and perspectives from our scientists.
News
Research milestones and recent achievements.
The latest from Amazon researchers
Amazon Science Blog
Technical deep-dives and perspectives from our scientists.
News
Research milestones and recent achievements.
Collaborations
Amazon Research Awards
Overview
Call for proposals
Latest news
Research stories
Recipients
Amazon Nova AI Challenge
Overview
Rules
FAQs
Teams
Research collaborations
Overview
Carnegie Mellon University
Columbia University
Hampton University
Howard University
IIT Bombay
Johns Hopkins University
Max Planck Society
MIT
Tennessee State University
University of California, Los Angeles
University of Illinois Urbana-Champaign
University of Southern California
University of Texas at Austin
Virginia Tech
University of Washington
Amazon Research Awards
Overview
Call for proposals
Latest news
Research stories
Recipients
Amazon Nova AI Challenge
Overview
Rules
FAQs
Teams
Research collaborations
Overview
Carnegie Mellon University
Columbia University
Hampton University
Howard University
IIT Bombay
Johns Hopkins University
Max Planck Society
MIT
Tennessee State University
University of California, Los Angeles
University of Illinois Urbana-Champaign
University of Southern California
University of Texas at Austin
Virginia Tech
University of Washington
Resources
Code and datasets
Amazon Nova
Try Amazon’s frontier foundation models.
Code and datasets
Amazon Nova
Try Amazon’s frontier foundation models.
Careers
Careers
Explore our open roles.
Amazon Scholars
Faculty research opportunities on industry-scale technical challenges.
Postdoctoral Science Program
Early-career research opportunities alongside experienced industry scientists.
Careers
Explore our open roles.
Amazon Scholars
Faculty research opportunities on industry-scale technical challenges.
Postdoctoral Science Program
Early-career research opportunities alongside experienced industry scientists.
Search
Submit Search
Conversational AI
Speeding up language models for travel applications
Two optimization techniques, when combined, can significantly improve a model’s speed and cost effectiveness without sacrificing quality.
By Daniel Zagyva , Laurens van der Maas
October 29, 2026
5 min read
Share
Share
Copy link
X
Line
QZone
Sina Weibo
分享到微信
x
Listen • 7:25 min
Key takeaways
Combining knowledge distillation (KD) with the Medusa-2 framework produced a final model (TinyLlama at 1.1 billion parameters) that matched the quality of the 70-billion-parameter Llama-3.1 teacher while running 10 to 20 times as fast. For Booking.com's Smart Filters, the KD-plus-Medusa approach delivered responses 15 times as fast for 99% of queries; knowledge distillation alone closed the student-teacher quality gap by more than 90% and made inference roughly six times as fast, after which Medusa heads added a further 2 to 3.6× speedup. On the two remaining use cases, the pipeline drove a 2.9% increase in user engagement for the AI Trip Planner (vs. GPT-3.5) and 1.4% more "helpful" ratings for AI-generated accommodation descriptions at lower costs than other models.
Was this answer helpful?
Travel platforms serve millions of daily users by answering questions about destinations, offering hotel recommendations, and helping users plan their trips. In recent years, many platforms have turned to large language models (LLMs) to improve these services. The LLMs give accurate, contextually relevant travel advice thanks to their size: they typically have tens of billions of parameters. Unfortunately, such large models are also computationally expensive, and that expense scales with the number of customer queries. What’s more, such large models can be too slow for web and mobile applications, where a delay of even 100 milliseconds can cost a company customers and revenue. To address these scaling and latency challenges, we explored two complementary optimization techniques. One is a compression technique known as knowledge distillation (KD), in which a slow but powerful model produces training targets for a faster, lightweight model. The other, called Medusa , is a framework that transforms sequential next-token predictions — a common bottleneck in LLMs — into parallel computations. Working with leading online travel agency Booking.com, we applied KD and Medusa techniques to the LLMs that power various aspects of its user experience, looking to improve either latency or cost effectiveness. We focused on three features in particular:
The AI Trip Planner: A conversational assistant that must extract travel preferences from dialogue in less than a second (latency); Smart Filters: A feature that converts natural-language search queries into structured information filters in real time (latency); Accommodation Description Generation: A feature that produces property descriptions on the scale of millions of listings (cost effectiveness).
We demonstrated that combining both KD and Medusa maintains the quality of the best open-source LLMs at less than one-third the cost while...
Excerpt shown — open the source for the full document.
Notability
notability 5.0/10Amazon post on optimizing LLMs for travel apps.