TinyFish Runs Its Web-Native Agent Model on Databricks AI Runtime

One platform from training to production

TinyFish offers Search, Fetch, Browser and Agent APIs for large-scale, authenticated web workflows. In these workflows an agent may have to navigate dynamic sites, retrieve information, interact with applications and finish a task over several steps. The company is developing its own specialized web-native model for this kind of work.

As the model work grew, the team also had to build the infrastructure around it: GPU compute for post-training, evaluation that measures the behavior of the full agent, production serving, and data to track how the system performs. Instead of spreading these pieces over separate systems, TinyFish began building its model development lifecycle on Databricks.

According to Databricks, AI Runtime provides managed GPU infrastructure for training and fine-tuning, MLflow supports evaluation, and Databricks Model Serving is the production serving layer. Zach Richardson, Head of AI at TinyFish, said the team can "train on GPUs, evaluate with MLflow, and serve our web-native model without building a separate stack for every stage."

Evaluating the whole agent, not single responses

TinyFish started post-training its web-native model with supervised fine-tuning and is moving toward Direct Preference Optimization, or DPO. The workloads run on Databricks-managed GPUs through AI Runtime, so researchers do not have to operate a separate training platform.

Because the model is one part of a larger web agent, TinyFish argues that judging single model responses tells only part of the story. It evaluates whether the full agent completes the requested task. A separate LLM judge reviews the original request, the agent's action history and the final result. A task passes only when the requested action is completed or the correct answer is returned with all requirements satisfied. Partial completion, incorrect answers and invalid results count as failures.

On the internal go/no-go benchmark, that process produced the 77.6% result across 2,178 tasks. The team uses such results to see how model changes affect the whole agent and to guide further post-training.

Serving and cost figures

TinyFish already serves production models on Databricks, including its search reranker and query rewriter, and it also hosts its web-native model there. Over a recent 15-day period, the company recorded 294,781 agent runs involving the model, covering 6.4 million tool steps.

The estimated model-serving cost in that period was $0.088 per run, or $0.0040 per tool step. Databricks stresses that these figures cover serving costs for the web-native model only, not the full agent stack. Evaluation results, usage data and cost reporting stay in Databricks next to the serving workload.

TinyFish is also consolidating access to external foundation models through Unity Gateway, for centralized cost tracking, key management and governance. General-purpose foundation models still have a role in its agent architecture, but the web-native model lets the team optimize for the tasks its product needs rather than relying only on external model APIs.

What it means for TinyFish and similar teams

The case shows how a growing AI company can keep post-training, evaluation and serving in one environment. Sudheesh Nair, Co-Founder and CEO of TinyFish, said the company does not want engineers "managing GPU infrastructure or moving models and data between disconnected systems."

Looking ahead, TinyFish plans to expand beyond supervised fine-tuning into DPO and to explore larger training workloads. According to Databricks, the workflow is meant to stay the same as those techniques change: evaluation shows where the model needs work, that feeds post-training, new versions are tested against complete tasks, and production-ready models are served from the same environment.

The benchmark and cost numbers come from TinyFish's own internal measurements as reported by Databricks, and they have not been independently verified.