From prototype to production: High QPS for Databricks AI Search

From prototype to production: High QPS for Databricks AI Search | Databricks Blog Skip to main content

Summary

Real-time search bars, recommendation and personalization systems, and entity resolution can run at peak production traffic without managing replicas, sizing nodes, or building load balancers in front of the index

Scale endpoints to thousands of QPS with a single configuration parameter, generally available today

Built-in production observability: endpoint QPS, latency, and health visible in the AI Search UI for every endpoint

Search is everywhere:  product discovery on retail sites, voice queries on smart TVs, recommendations in every feed, and identity matching during account lookups. Each fires a request per page view, per keystroke, per user action. At consumer scale, that translates into thousands of queries per second hitting the search index, with peak traffic often several times higher. Search also sits on the critical path of revenue. In retail, shoppers who use search convert at two to three times the rate of those who simply browse. On streaming platforms, recommendations drive most of what users watch. Reaching production-level QPS used to mean building a bespoke retrieval stack per app. Load testing across index sizes, query types, and filters. Manually sizing capacity. Wiring up load balancers in front of the index. Every new search use case restarts the work. The pain is universal across vector databases, search engines, and DIY stacks. Announcing high-QPS scaling for Databricks AI Search Today we're announcing high-QPS scaling for Databricks AI Search which is generally available. Standard endpoints can now scale to thousands of QPS with a single, human-readable parameter. You tell us your target. We provision the infrastructure to meet it. Set  target_qps on the endpoint at creation, or update it on any existing endpoint at any time via the SDK, REST API, or the endpoint UI. Databricks provisions the infrastructure to meet the target. No replica counts to manage, no nodes to size, no load balancers to wire up.

Unity Catalog governance and Delta Sync stay in place. The same endpoint that powered your prototype now scales to production traffic without leaving the platform. When you need high-QPS scaling Three real-time production patterns require it. Search bars,  like product search on e-commerce, content discovery on streaming and media platforms, voice search on connected devices. A typeahead box can fire a search call per keystroke, so QPS scales with active typing volume. Latency directly affects conversion. For an end-to-end architecture, see  Building Real-Time Product Search on Databricks . Recommendation / personalization systems,  like "You may also like" panels on e-commerce, personalized feeds on media and streaming platforms. Every page view fires a recommendation query, so traffic spikes hit retrieval first. Recommendation latency is on the critical path of the request. Real-time entity resolution,  like   identity matching, deduplication, lookup against large catalogs at request time. Query rate here is an operational SLA, not a burst you can buffer. If you're seeing any of the following, you probably need this: 429 (Too Many Requests) errors showing up in production logs P95 latency spiking as traffic ramps, even at moderate average utilization Workarounds like duplicate endpoints or client-side load balancing

Deep dive into how it works Declare. You set a QPS target on the endpoint. Databricks calculates and provisions the compute capacity to deliver that target. No replica counts, no node sizing, no capacity planning. Works with existing endpoints. Update any Standard endpoint via the Python SDK, REST API, or the UI. New capacity takes effect the next time an index on the endpoint is created or synced.

Monitor scaling state. The  scaling_info field on the endpoint tracks progress as it transitions from  SCALING_CHANGE_IN_PROGRESS to SCALING_CHANGE_APPLIED .

Production observability. Operating a retrieval system in production needs visibility into requests per second, request latency, and endpoint health. The endpoint UI now surfaces all three for every endpoint. Use service principal auth to maximize performance.  Service principal traffic is routed through performance-optimized networks built for high-QPS production workloads. Personal access token (PAT) traffic is capped at a few tens of QPS, making it suitable for prototyping but not production.   See the  performance guide for the full walkthrough. Sizing. Use the endpoint observability UI and native Genie integration to understand your traffic patterns and set  target_qps  with sufficient headroom for spikes. Start building today The difference between prototype and production is now a configuration parameter. High-QPS scaling is generally available today, no opt-in required. Two paths to start: Spin up a new endpoint with  target_qps set to your initial target Update an existing endpoint with  target_qps to scale up the index already serving your application

What’s coming next We’re continuing to make search easier to operate at scale. Automatic scaling for traffic spikes—without manual capacity planning or sizing—and support for Storage-Optimized endpoints are planned for later this year. To go deeper: High QPS documentation Databricks AI Search overview Performance best practices Building Real-Time Product Search on Databricks

Get the latest posts in your inbox Subscribe to our blog and get the latest posts delivered to your inbox.

Sign up

View all blogs