Roboflow test shows how a 90-second GPU traffic spike causes failure

A demanding vehicle-counting workload

Roboflow tested the two deployment approaches with a workflow that counted vehicles in 1280×1280 aerial images of a parking lot. The workflow used SAHI to divide each frame into 16 overlapping 427×427 tiles, run the detector on each tile, merge overlapping detections and count the remaining boxes.

That process produced 102 detected vehicles on the test frame, compared with two when the full image was passed through the detector in one operation. Although the client made one API request, each request triggered 16 model inferences and the stitching process, creating a substantially heavier workload than a simple single-image benchmark.

The same traffic exposed a sharp difference

Roboflow compared its hosted inference service, using the address serverless.roboflow.com, with a single cloud VM running an NVIDIA L40S and Roboflow's open-source inference server, version 1.7.4 with default settings. Both used the same Locust load-test file and configuration. The only change between the tests was the host URL.

The test began with 20 simulated users, equivalent to five requests per second, for one minute. It then increased to 80 users, or 20 requests per second, for 90 seconds before returning to 20 users for 30 seconds. Because each request generated 16 model calls, the peak workload reached 320 model calls per second. Requests taking longer than 20 seconds were counted as failures.

At five requests per second, both systems had a median latency of about one second. During the burst, the self-hosted server completed roughly 13 to 15 requests per second while 20 requests per second continued arriving. Its median latency climbed to the 20-second timeout, and successful responses later fell to zero, including after traffic returned to five requests per second.

Roboflow Serverless also saw its median latency rise, reaching about three seconds early in the burst, but it later settled around 1.3 seconds while handling roughly 16 to 24 requests per second. After the burst, its median latency fell to about half a second.

Reported results and the limits of autoscaling

Across the three-minute run, Roboflow reported that Serverless returned 2,273 successful requests with a median latency of 1.4 seconds. The self-hosted server returned 1,095 requests, with a whole-run median latency of 20 seconds. Locust marked 50% of the self-hosted requests as failures, compared with 0%, rounded, for Serverless.

Roboflow says operators have two main ways to address the self-hosted result: provision enough GPUs for the largest expected burst, or autoscale. The company measured about five minutes to boot the GPU VM, pull the inference container and start the server in its demonstration. Since the traffic burst lasted 90 seconds, a new GPU would have arrived after the burst had ended.

The company says its serverless system uses queues for each model and GPU workers that prefer models already loaded in VRAM. Roboflow states that a warm model answers in about 5 ms, while a cold load takes around 30 seconds. The system scales based on backlog across models and maintains a warm floor of spare capacity, while Roboflow's own training jobs step aside when inference requires more GPUs.

What the comparison means for deployment choices

The test supports Roboflow's distinction between steady and bursty workloads. Self-hosting can be a reasonable choice when traffic is predictable, but a workload can spike because more cameras come online, a production line starts, footage is backfilled, or a workflow adds more image tiles without increasing the request rate.

For users running the tested workflow, the practical tradeoff is between planning for peak GPU demand in advance and using a service designed to add capacity during a spike. Roboflow says the same workflow ran on Serverless with no change other than the URL. The results are the company's benchmark of this specific setup, rather than a general performance claim for every inference workload.