Llms Inside The Product A Practical Field Guide

LLMs Inside the Product: A Practical Field Guide | Groq is fast, low cost inference. Building with LLMs has taught me one clear lesson: the best AI feature is often invisible. When it works, the user doesn’t stop to think “that was AI.” They just click a button, get an answer quickly, and move on with their task. When it doesn’t work, you notice right away: the spinner takes too long, or the answer sounds confident but is not true. I’ve hit both of these walls many times. And each time, the fix was less about “smarter AI” and more about careful engineering choices. Use only the context you need. Ask for structured output. Keep randomness low when accuracy is important. Allow the system to say “I do not know.” This guide is not about big research ideas. It’s about practical steps any engineer can follow to bring open-source LLMs inside real products. Think of it as a field guide. Think of it as a field guide with simple patterns, copy-ready code, and habits that make AI features feel reliable, calm, and fast. How It Works — The Four‑Step Loop Every reliable AI feature follows the same loop. Keep it consistent. Boring is good. 1) Read What: Take the user input and only the smallest slice of app context you need. More context means higher cost, slower responses, and more room for the model to drift. Examples Support — “Where is my order?” → pass the user ID and the last order summary, not the entire order history. Extraction — “Pull names and dates from this email thread” → pass the thread text only, not unrelated attachments. Search — “Find refund policy” → pass top three snippets from your docs, not the whole knowledge base.

2) Constrain What: Set rules so the model stays within your desired constraints. Do this System prompt as a contract State what the assistant is and is not Require valid JSON that matches a schema If there is missing information, ask the user for a short follow‑up or answer “I don’t know.” Keep privacy rules explicit (do not log sensitive data) Version your prompts and test them

Match temperature to the task (there is no one setting that fits all) Low (≈0.0–0.2): Extraction, classification, validation, RAG answers with citations, reliable tool choice Medium: Templated drafts and light tone variation High: Brainstorming and creative copy where variety matters

Keep context tight in all cases. If your stack supports it, use a seed in tests for repeatability. 3) Act What: Aim to produce LLM-generated outputs that can be used as inputs in the next part of your workflow without further processing required. When to use what: Structured Outputs when the next step is programmatic, e.g., for updating UI, storing fields, and running validation. Why: Such outputs are ready to be used as inputs in the next step of a workflow or application as it's structured, parsable data that needs no further manual processing. Example: Extract {name, date, amount} from an invoice.

Code: Structured Outputs with Pydantic 1 from groq import Groq 2 3 from pydantic import BaseModel 4 5 from typing import Literal 6 7 import json 8 9 client = Groq() 10 11 class ProductReview ( BaseModel ): 12 13 product_name: str 14 15 rating: float 16 17 sentiment: Literal [ "positive" , "negative" , "neutral" ] 18 19 key_features: list [ str ] 20 21 response = client.chat.completions.create( 22 23 model= "moonshotai/kimi-k2-instruct" , 24 25 messages=[ 26 27 { "role" : "system" , "content" : "Extract product review information from the text." }, 28 29 { 30 31 "role" : "user" , 32 33 "content" : "I bought the UltraSound Headphones last week and I'm really impressed! The noise cancellation is amazing and the battery lasts all day. Sound quality is crisp and clear. I'd give it 4.5 out of 5 stars." , 34 35 }, 36 37 ], 38 39 response_format={ 40 41 "type" : "json_schema" , 42 43 "json_schema" : { 44 45 "name" : "product_review" , 46 47 "schema" : ProductReview.model_json_schema() 48 49 } 50 51 } 52 53 ) 54 55 review = ProductReview.model_validate(json.loads(response.choices[ 0 ].message.content)) 56 57 print (json.dumps(review.model_dump(), indent= 2 )) Copy

Learn more: Structured outputs docs Function calls (tools) when the model needs live data or to trigger an action that your code controls such as search, fetch, compute, notify, or connect to external systems. Why: Tools let the model use fresh information instead of relying only on what it learned during training. That means it can query a database, call an API, or look up the latest records without inventing answers. The model proposes the action, your code decides whether to run it, and you keep a clear audit trail. Example: The model calls search_docs() to find relevant text, then render_chart() to create a visualization, and finally explains the result back to the user.

Plain text when the result is narrative only, such as a summary or a short answer. Why: Simplest path when nothing else needs to consume the output.

Code: function calling (tools) 1 import json 2 from groq import Groq 3 import os 4 5 # Initialize Groq client 6 client = Groq() 7 model = "llama-3.3-70b-versatile" 8 9 # Define weather tools 10 def get_temperature ( location: str ): 11 # This is a mock tool/function. In a real scenario, you would call a weather API. 12 temperatures = { "New York" : "22°C" , "London" : "18°C" , "Tokyo" : "26°C" , "Sydney" : "20°C" } 13 return temperatures.get(location, "Temperature data not available" ) 14 15 def get_weather_condition ( location: str ): 16 # This is a mock tool/function. In a real scenario, you would call a weather API. 17 conditions = { "New York" : "Sunny" , "London" : "Rainy" , "Tokyo" : "Cloudy" , "Sydney" : "Clear" } 18 return conditions.get(location, "Weather condition data not available" ) 19 20 # Define system messages and tools 21 messages = [ 22 { "role" : "system" , "content" : "You are a helpful weather assistant." }, 23 { "role" : "user" , "content" : "What's the weather and temperature like in New York and London? Respond with one sentence for each city. Use tools to get the information." }, 24 ] 25 26 tools = [ 27 { 28 "type" : "function" , 29 "function" : { 30 "name" : "get_temperature" , 31 "description" : "Get the temperature for a given location" , 32 "parameters" : { 33 "type" : "object" , 34 "properties" : { 35 "location" : { 36 "type" : "string" , 37 "description" : "The name of the city" , 38 } 39 }, 40 "required" : [ "location" ], 41 }, 42 }, 43 }, 44 { 45 "type" : "function" , 46...