Google's LiteRT Tensor API lets developers build on-device models op by op in C++

Why Google built a native authoring path

Google says the most common way to deploy models in production is to convert a trained model, for example a PyTorch model with LiteRT Torch, and run it with the LiteRT runtime. The company calls this the quickest and most reliable route and still recommends it for most projects.

Some developers want more control, for example to support new architectures or to try new lowering logic without waiting for upstream framework updates. They often write computations by hand in native C++ or shader languages. Google points to Apple's MLX and llama.cpp as examples of how far this code-first approach can go.

The cost, according to the post, is that such implementations are tightly coupled to specific backends such as Metal, OpenCL or Vulkan and must be rewritten for each new hardware target. Managing zero-copy buffer interoperability and synchronization across CPU, GPU and NPU, while joining pre-processing, inference and post-processing, also takes substantial manual engineering.

What the Tensor API offers

Developers build the computation graph in C++ and choose exactly which ops run and how they connect. The same graph can run through LiteRT on CPU, GPU (powered by the open-source ML Drift) and NPU, or directly on CPU through XNNPACK without a model file or runtime. Google lists four key features:

• Full control over graph structure: developers decide every op and connection, from elementwise math to accelerator-optimized LLM composite ops such as scaled dot-product attention, RMSNorm and KV-cache updates.
• Direct weight ingestion: weights load straight from checkpoints such as Hugging Face .safetensors files, with no conversion scripts in between.
• Tensor-centric buffer interop: pre-processing, model partitions and post-processing pass tensors through shared accelerator buffers, so intermediate data stays on the accelerator with zero copies.
• Building blocks for AI agents: because models are plain C++ built from well-defined ops, coding agents can construct a model, check its results against a reference and benchmark variants on each device.

In the code examples, each layer is a C++ function over Tensor values. The backend is a template argument: TfLiteMixinTag exports a .tflite file that LiteRT runs on CPU, ML Drift (GPU) or NPU, while XnnpackMixinTag runs the same code directly on CPU through XNNPACK. Tensors can be tagged with both to keep both options open.

Google suggests a workflow that uses both: iterate on the direct XNNPACK path, loading weights from the .safetensors checkpoint with no model file to rebuild, then export a .tflite when ready to ship with LiteRT on mobile or on the web through LiteRT.js.

Segment Anything demo and performance claims

To show the API in practice, Google built an interactive Segment Anything app that runs entirely on-device in the browser on WebGPU. It performs real-time multi-object segmentation and tracking with Meta's Segment Anything Model 2 (SAM 2), triggered by clicks or by natural language prompts handled by Gemma 4 E2B.

The SAM 2 model is fully authored in C++ with the Tensor API. According to Google, a coding agent wrote it and evolved it from under 10 to over 30 frames per second at 512x512 on WebGPU. The full implementation and the web app are open source in the LiteRT Samples GitHub repo.

Google also ran the same C++ code on the GPUs of three devices: a macOS laptop (Metal), an iOS phone (Metal) and an Android phone (OpenCL). In its tests on a MacBook Pro, the model ran about 1.5x faster than Apple MLX (fp16, image encoder plus mask decoder). Google notes that pre- and post-processing add ~0.34 ms and that output masks match Meta's PyTorch reference pixel for pixel. The SAM 2.1 sample includes benchmark commands for reproducing the results.

Keeping the whole pipeline on the accelerator

Google notes that a real app needs more than the network itself. In the demo, turning a camera frame and a click into a mask takes four steps: pre-processing, the image encoder, the mask decoder and post-processing. In a typical app, only the model runs on the GPU, while the CPU resizes frames, hands the encoder output to the decoder and post-processes the mask. Each trip copies data and stalls the GPU, according to the post.

In the sam2 sample, all four steps are Tensor API graphs on the GPU. They are connected once at startup to share buffers and then run for each frame and click.

What it means for developers

Google is clear that conversion remains the best default for most models. The Tensor API is aimed at developers who need low-level control over the graph, or who want an entire pipeline to run end to end on the accelerator without rewriting it for each hardware target. The company also positions it as a programmatic foundation for AI coding agents that generate, rewire and tune model graphs automatically.