How Flo Health built safer, more useful open models with Databricks AI Runtime

How Flo Health built safer, more useful open models with Databricks AI Runtime | Databricks Blog Skip to main content

Summary

Flo Health built a specialized evaluation framework to measure AI responses across both medical safety and usefulness, calibrated against clinician review.

Using privacy-first synthetic data and AI judges, Flo created training data without human-annotated training pairs, then used supervised fine-tuning and Direct Preference Optimization (DPO) to improve an open model.

Flo ran distributed fine-tuning on 32 NVIDIA H100 GPUs with Databricks AI Runtime, improving medical safety from 34% to approximately 61%, usefulness from 42% to 76%, and the combined score from 16% to 47%.

For Flo Health, the bar for generative AI is unusually high. With 81 million monthly active users, the women’s health app operates at a scale where even small weaknesses in an AI system matter. Because Flo’s products deal with health and well-being, responses need to be medically safe and genuinely useful. As Flo moved from using only traditional ML towards personalized insights powered by generative AI, the team saw the potential for a more flexible, personalized product. But relying on a single general-purpose model introduced other constraints: cost at scale, dependence on a model provider’s lifecycle, and limited ability to encode Flo’s own domain expertise. Flo’s longer-term vision was a compound AI system that could use the right model for each task while maintaining a consistent safety standard across them. That led Flo toward fine-tuned open models and a new post-training pipeline built with Databricks. The team used AI judges and privacy-first synthetic data to turn Flo’s standards for medical safety and usefulness into a training signal, then used  Databricks AI Runtime to run distributed fine-tuning on serverless NVIDIA H100 GPUs. That approach also made sense economically at Flo’s scale. The team estimates that using tailored, fine-tuned models can reduce inference costs by an order of magnitude compared with relying on frontier model APIs, while giving Flo greater control over model behavior and lifecycle. But before Flo could train models against its own standards, it needed a reliable way to define and measure what “good” looked like. Start with evaluation, not training Flo defined two goals for its AI systems: medical safety as the foundation and usefulness as the goal. A model can avoid unsafe statements and still provide little value, or give a detailed, personalized answer while missing an important medical safeguard. Consider a question such as, “Why is my period late?” An answer that simply tells the user to see a medical professional may be safe but not particularly useful. An answer that lists only common lifestyle causes may be more helpful, but incomplete from a medical-safety perspective. Flo wanted responses that could balance both: useful information, appropriate context, and the right safety netting. To make that balance measurable, Flo’s medical, product, and engineering teams built an evaluation framework that decomposed medical risk into specific rubrics that could be tested consistently. For example, after identifying that responses to symptom questions could be overly reassuring, Flo created a “balanced causes” rule requiring appropriate coverage of both benign and potentially serious causes when relevant. Those rubrics were then turned into AI judges, each with a definition, grading instructions, and training cases. Flo validated the judges against synthetic cases and anonymized cases from real-user experiments. When an AI judge disagreed with an initial human assessment, blinded clinical reviewers adjudicated the disagreement to establish ground truth. Flo then used part of those cases to refine the judges and reserved the rest to measure performance. Turning expert judgment into a scalable training signal With the evaluation system in place, Flo could use those same judges as the reward signal for training. The final framework included 26 judges across three categories: 12 medical-safety judges acted as hard constraints. 6 “micro” judges evaluated more nuanced behaviors such as voice, grounding, framing, and fine-grained safety. 8 usefulness criteria measured factors, including completeness, relevance, coherence, and tone.

Building a conventional human-labeled fine-tuning dataset would have required significant clinical expertise for every example. Flo concentrated its medical experts’ time on designing and calibrating the evaluation system, which could then score training candidates at scale. The resulting pipeline generated more than 15,000 synthetic questions using privacy-first principles and created training pairs without human-annotated responses. Flo derived the inputs from model-generated or anonymously aggregated signals rather than user-written free text or identifiable information. Clinicians remained essential to defining and validating what “safe” meant, but they no longer needed to manually label every individual training pair. Fine-tuning with AI judges and preference optimization With calibrated judges and synthetic training data in place, Flo adopted a Reinforcement Fine-Tuning (RTF) approach that generates multiple candidate responses during training, uses the evaluation system to identify the strongest ones, and fine-tunes the model so that quality is encoded in its weights rather than relying on expensive best-of-N search on every production request. Flo’s workflow followed five stages: Synthesize privacy-preserving questions. Generate multiple candidate answers. Score them across medical safety, usefulness, and behavioral criteria. Select the strongest responses and build preference pairs. Train with supervised fine-tuning and DPO, then rerun the judges to verify the results.

The first version of this approach exposed an important limitation: usefulness improved while medical safety remained relatively flat because the preference pairs did not distinguish sufficiently between degrees of medical quality. Flo addressed that by adding a graded safety score. Responses that passed the binary safety gate were re-ranked according to how thoroughly they met the applicable medical criteria, creating preference pairs with a clearer medical-safety contrast. Scaling post-training with Databricks AI Runtime Running this generate-score-select-train loop at scale is compute-intensive. Flo’s training stack combined Databricks AI Runtime,...