Building An Ai Retail Assistant At The Edge With Small Language Models And Intel Xeon Cpus

Arcee AI | Building an AI Retail Assistant at the Edge with Small Language Models and Intel Xeon CPUs

Trinity Large Thinking: Available on OpenRouter.

Try now ↗

ENTERPRISE

Research

COMPANY

Get API

Blog / Building an AI Retail Assistant at the Edge with SLMs on Intel CPUs

Building an AI Retail Assistant at the Edge with SLMs on Intel CPUs Julien Simon ,

•

June 7, 2025

Thanks to a chatbot interface powered by open-source small language models and real-time data analytics, store associates can interact naturally through voice or text.

We ran a live stream on June 10 from the floor of Cisco Live. You can watch it on YouTube . The Power of AI at the Retail Edge The retail landscape is rapidly evolving, with AI-driven solutions reshaping how stores operate and engage with customers. Today's consumers expect personalized, efficient service, while staff need immediate access to accurate information on customer traffic, inventory, sales, and other key metrics. Meeting these demands requires powerful edge computing solutions that can process and deliver insights in real-time. Powered by Intel Xeon 6 CPUs running in a Cisco UCS server, the Edge IQ Retail Assistant exemplifies this potential. This technical demonstrator will be featured in the Intel Showcase (#3035) at Cisco Live 2025, taking place in San Diego, CA, from June 8 to 12, 2025. Attendees will get a firsthand experience of how generative AI can transform retail operations without relying on GPUs. Thanks to a chatbot interface powered by open-source small language models and real-time data analytics, store associates can interact naturally through voice or text, receiving immediate information about product availability from Chooch 's inventory system or crowd density from WaitTime 's analytics platform. The assistant seamlessly translates these inquiries into actionable insights, helping staff make informed decisions that enhance customer experience while optimizing store operations, all powered by CPU processing. ‍ The Edge Advantage: Why Small Language Models on CPUs Make Sense for Retail The Edge IQ Retail Assistant represents a fundamental shift in AI deployment strategy for retail environments. While cloud-based AI services dominated early generative AI implementations, this solution demonstrates why running optimized language models directly on CPU servers at the edge delivers superior results for retail operations. Low Latency : In the fast-paced retail environment, latency is a critical factor. Edge deployment eliminates network round-trips that typically add hundreds of milliseconds to each interaction, thereby enhancing overall performance. When a customer is waiting for information about product availability, this speed difference creates a noticeably more responsive experience. The assistant delivers consistent responses regardless of internet conditions—something cloud alternatives simply cannot match. Resilience : Continuous operation remains essential for retail technology. Edge-deployed models continue functioning during internet outages, ensuring the assistant remains available during critical business hours. Store operations cannot pause when connectivity issues arise, making local processing a necessity rather than a luxury. A cost-effective CPU-based deployment provides this reliability without requiring specialized hardware. ‍

Data Privacy : Data privacy concerns have grown increasingly important for retailers. By processing queries locally on CPU servers, sensitive business information, including inventory levels, sales data, and staffing details, never leaves the store's infrastructure. This dramatically reduces potential data exposure while maintaining full functionality—a compelling advantage over cloud alternatives that must transmit this information externally.

Bandwidth Efficiency : The bandwidth efficiency of local processing proves particularly valuable when working with high-resolution video streams from inventory and crowd monitoring systems. These data-intensive feeds remain within the local network, eliminating costly and bandwidth-intensive cloud transfers that would otherwise constrain system performance. ‍

Cost Predictability : Perhaps most compelling for retail operations, edge deployment on standard CPU servers provides consistent and predictable operational costs. Unlike cloud services with usage-based pricing that can spike during busy periods, the fixed infrastructure cost provides budget certainty, simplifying financial planning across store networks.

Real-Time Data Integration: Crowd Analytics and Inventory Management When a store associate inquires about current customer traffic, the assistant queries WaitTime 's API, interpreting the crowd analytics data coming from on-premise cameras to provide meaningful insights about congestion points, checkout wait times, and optimal staffing distribution. This real-time information enables managers to direct employees where they're most needed, thereby enhancing both operational efficiency and customer satisfaction. Similarly, inventory queries trigger connections to Chooch 's Vision AI platform, which continuously monitors store shelves through computer vision. The assistant translates Chooch's detailed inventory data into actionable insights, informing staff about current stock levels, identifying low-stock items, and helping prevent potential stockouts before they impact customers. By presenting this information conversationally, the assistant makes sophisticated inventory intelligence accessible to every employee, regardless of technical expertise, all processed locally on the CPU without requiring GPU acceleration.

Harnessing Open Source AI on CPU-Only Infrastructure The Edge IQ Retail Assistant demonstrates the impressive capabilities of modern CPU-based AI inference. At its core, the application runs three sophisticated small language models entirely on Intel Xeon processors —no GPUs are present in the system architecture. This CPU-only approach highlights the significant progress in AI optimization and the remarkable capabilities of modern server processors for AI workloads. Arcee AI SuperNova Lite : Serving as the central intelligence of the system, this 8-billion parameter open-source conversational model is a high-performance distilled version of the larger Llama-3.1-405B-Instruct model. We quantized the model to 4 bits using the Hugging Face Optimum Intel library,...