Flo Health fine-tunes open model with Databricks, lifting medical safety to about 61%
Flo Health, the women's health app with 81 million monthly active users, has fine-tuned an open model with Databricks AI Runtime and raised its combined medical safety and usefulness score from 16% to 47%, according to Databricks. The company built its own evaluation framework first, then used it to generate training signals without human-annotated training pairs.
- Flo Health raised its model's combined safety and usefulness score from 16% to 47% after fine-tuning, according to Databricks.
- Medical safety rose from 34% to approximately 61%, and usefulness from 42% to 76%.
- Flo built 26 AI judges, calibrated against blinded clinical review, and more than 15,000 privacy-first synthetic questions.
- Training used SFT and DPO on 32 NVIDIA H100 GPUs with Databricks AI Runtime.
- Flo plans to reuse the pipeline on open-weight models such as Gemma, Qwen and Kimi.
Why Flo moved away from a single general-purpose model
As Flo moved from traditional machine learning toward personalized insights powered by generative AI, relying on one general-purpose model brought constraints. Databricks lists cost at scale, dependence on a model provider's lifecycle, and limited ability to encode Flo's own domain expertise.
Flo's longer-term vision is a compound AI system that uses the right model for each task while keeping a consistent safety standard across them. That led the company toward fine-tuned open models. The team estimates that tailored, fine-tuned models can reduce inference costs by an order of magnitude compared with frontier model APIs, and give Flo more control over model behavior and lifecycle.
Evaluation first, then training
Flo defined two goals: medical safety as the foundation and usefulness as the goal. A model can avoid unsafe statements yet offer little value, or give a detailed answer that misses an important safeguard. For a question like "Why is my period late?", an answer that only says to see a medical professional may be safe but not very useful, while one listing only lifestyle causes may be incomplete from a safety view.
Flo's medical, product and engineering teams broke medical risk into specific rubrics. After finding that answers to symptom questions could be overly reassuring, they created a "balanced causes" rule requiring coverage of both benign and potentially serious causes where relevant. The rubrics became AI judges, validated against synthetic cases and anonymized cases from real-user experiments. When a judge disagreed with an initial human assessment, blinded clinical reviewers decided the outcome.
The final framework had 26 judges:
- 12 medical-safety judges acting as hard constraints
- 6 "micro" judges covering voice, grounding, framing and fine-grained safety
- 8 usefulness criteria, including completeness, relevance, coherence and tone
Synthetic data and preference optimization
Flo's pipeline generated more than 15,000 synthetic questions using privacy-first principles, derived from model-generated or anonymously aggregated signals rather than user-written free text or identifiable information. Clinicians focused on designing and validating the evaluation system instead of labeling each training pair.
The workflow had five stages: synthesize questions, generate multiple candidate answers, score them, select the strongest and build preference pairs, then train with supervised fine-tuning and DPO and rerun the judges. The first version improved usefulness, but medical safety stayed relatively flat because the preference pairs did not separate degrees of medical quality well enough. Flo added a graded safety score, re-ranking responses that passed the binary safety gate by how thoroughly they met the applicable criteria.
Infrastructure and results
The training stack combined Databricks AI Runtime, LLM Foundry and Compose-RL, with MLflow for tracking and checkpoints. AI Runtime supplied the serverless GPU environment, LLM Foundry handled distributed training, and Compose-RL provided the preference-optimization layer. Flo used 32 NVIDIA H100 GPUs with PyTorch and Fully Sharded Data Parallel.
Vladislav Nedosekin, Director of Engineering, Data & AI Platform at Flo Health, said: "We can spend less time managing GPU infrastructure and more time improving the models themselves."
The first model Flo evaluated scored 34% on medical safety and 42% on usefulness, with only 16% of responses meeting both criteria. After fine-tuning, medical safety rose to approximately 61%, usefulness to 76%, and the combined score to 47%.
What it means going forward
Databricks frames the reusable evaluation and training pipeline as the more durable outcome than the 47% figure. Flo plans to apply it to newer open-weight models, including Gemma, Qwen and Kimi, while keeping the medical safety and usefulness criteria it has already validated. According to Databricks, this lets Flo iterate on new models without rebuilding the training environment.
