LiteRT Tensor API in C++: XNNPACK, GPU, NPU and Native Model Graphs

Evidence boundary — October 20, 2026: This guide reviews Google’s announcement, publicly documented Tensor API interfaces, and upstream examples. XBSTACK has not compiled the Tensor API sample on its development Mac or run it on an iPhone, Android handset, or dedicated NPU. Every benchmark mentioned as Google’s result belongs to its specified demonstration, not an XBSTACK experiment.

Short answer: LiteRT Tensor API is a C++ interface for authoring and composing tensor computation graphs. It is not a new foundation model, and it does not eliminate model training. Depending on the chosen backend, a graph can run directly via XNNPACK on CPU, be serialized as a .tflite for LiteRT CompiledModel, or follow a supported ML Drift GPU path. Real hardware compatibility must still be tested per backend, operation and device.

Why Google added graph authoring to LiteRT

For most on-device applications, the standard path remains the right starting point: train or obtain a model, convert and optimize it, then deploy the artifact with a supported runtime. Google explicitly preserves that recommendation in its October 19 announcement: Native Model Authoring for Edge.

Graph authoring fills a different engineering gap. A converter may not yet support a novel operation or a specific composition of layers. A real-time video application might spend substantial time resizing, normalizing, moving and compositing tensors around the model. An experimental architecture may require deterministic visibility into graph transformations. In those cases, directly expressing the graph in C++ can be more useful than introducing another conversion stage or maintaining separate backend-specific implementations.

The trade-off is real: C++ build tooling, library revisions, operator support, buffer interoperability, numerical testing and long-term maintenance now become explicit responsibilities of the application team.

Tensor API, CompiledModel, LiteRT-LM and ML Drift are different layers

Component	Main responsibility	Typical use	What it does not guarantee
Tensor API	Author and compose C++ tensor graphs	Custom graphs, preprocess/postprocess, experimental operations	Model training or universal acceleration
CompiledModel	Prepare and execute supported model artifacts	Deploying an existing model in an application	Writing the computation graph itself
LiteRT-LM	Engine and tooling for on-device generative language models	Local chat, streaming and model execution workflows	A one-to-one replacement for graph authoring
ML Drift	Supported GPU acceleration / lowering path	Selected device-resident GPU workloads	Identical operator support on every GPU

This distinction also matters when comparing Tensor API with Apple MLX or llama.cpp. All can involve native code and model computation, but they make different choices about graph authoring, artifacts, backend support and deployment. Performance claims require a fixed model, precision, hardware configuration, runtime version and benchmark workload.

How a LiteRT computation graph is built

A Tensor value here represents a node in a lazily recorded graph: constructors define shape, type and optional constants, while operators add directed dependencies. Execution happens only after selecting a runner or serializing the graph. This distinction lets you validate the same logical model through different supported backends without assuming equivalent performance or device coverage.

An intentionally small first test: add two vectors

This is a development checklist illustration, not an XBSTACK test log.

Before evaluating a complex attention graph or SAM 2 pipeline, start with a three-element addition.

Input a: [1, 2, 3]
Input b: [4, 5, 6]
Graph: Add(a, b)
Expected output: [5, 7, 9]

The essence of the documented C++ API looks like this. It is a reduced interface sketch, not a complete compile-ready file:

using namespace litert::tensor;
using CpuTensor = Tensor<XnnpackMixinTag>;

CpuTensor lhs({.name = "lhs", .type = Type::kFP32,
               .shape = {3}, .buffer = std::vector<float>{1, 2, 3}});
CpuTensor rhs({.name = "rhs", .type = Type::kFP32,
               .shape = {3}, .buffer = std::vector<float>{4, 5, 6}});
auto output = Add(lhs, rhs);
// Use XnnpackRunner to compile/run and ReadOutputAs<float> to inspect values.

For complete includes, Bazel dependencies, error-handling macros and runner calls, use the pinned version of Google’s Tensor API Quick Start. The README also provides the analogous TfLiteMixinTag and ML Drift variants.

Reproducible build setup

With the required C++ toolchain, Bazel/Bazelisk and repository dependencies installed:

git clone https://github.com/google-ai-edge/LiteRT.git
cd LiteRT

# Pin this revision in any published reproduction.
git rev-parse HEAD

# Build and test upstream's Tensor API targets.
bazel build //tensor/...
bazel test //tensor:all //tensor/backends/tflite:all

If you create your own add_xnnpack.cc, declare an explicit cc_binary and its dependencies on //tensor, //tensor:arithmetic, the XNNPACK arithmetic library, and the XNNPACK runner. Run that specific binary target and compare the output with the expected vector. The generic //tensor/... build alone does not create your custom target.

Actual XBSTACK environment: clang++ was available on the connected Mac at the time of review, but Bazel/Bazelisk and a prepared LiteRT workspace were not detected. Therefore the expected [5,7,9] output is a reference value from the example, not an independently captured XBSTACK execution result.

The three practical backend choices

Execution-path schematic only; accelerator availability is not implied.

The Tensor API README describes Tensor<Mixins...> as a lazy graph-building abstraction: declaring tensors and applying operations records a directed computation graph. The chosen mixin controls how that graph is lowered or executed.

1. XNNPACK: get numerical correctness on CPU first

A graph authored with XnnpackMixinTag can be lowered into XNNPACK and run through XnnpackRunner without first generating an intermediate model file. This is a sensible first checkpoint for many custom graphs because it separates shape and numerical problems from mobile accelerator packaging.

A CPU reference run is not proof that every operation maps to a phone’s GPU or NPU. It is the baseline against which optimized paths should be compared.

2. .tflite plus CompiledModel: prepare for device distribution

With TfLiteMixinTag, Tensor API can produce a LiteRT model graph. A resulting .tflite artifact can then be loaded by the appropriate LiteRT runtime and CompiledModel workflow. The consumer application does not always need the entire graph-authoring toolchain.

You still need to validate supported ops, shapes, quantization, accelerator binaries and fallback behavior. Having an NPU in the phone does not mean that an arbitrary exported graph executes on that NPU.

3. ML Drift: model graphs on supported GPUs

MlDriftMixinTag targets the supported ML Drift GPU flow. Depending on the platform and toolchain, this may use OpenCL or WebGPU; other pathways such as Metal also appear in Google’s end-to-end samples. The hardware API, shader compilation and runtime constraints are not magically identical across devices.

Choose this route after numerical equivalence is established, and only if the intended GPU backend supports your actual graph. The official Tensor API documentation provides the current backend headers and dependency targets.

Where zero-copy pipelines can help

Consider a real-time segmentation workload:

1. Acquire camera frames and convert their pixel format.
2. Resize and normalize tensors.
3. Run an encoder.
4. Execute prompt processing, mask decoding and temporal state updates.
5. Resize and composite the mask into the output frame.

If every stage copies data back to CPU memory and then transfers it to the accelerator again, the complete task latency can dominate the time spent in the neural network itself.

Tensor API’s ModelChain and LitertBuffer support composing multiple stages and negotiating compatible intermediate buffers. The goal is to avoid unnecessary copies when backend and buffer types support sharing. A particular device and pipeline still need evidence that those copies were actually removed.

Google’s SAM 2 demonstrations include browser WebGPU and mobile GPU execution scenarios. Google’s published frame rates and comparisons with MLX refer to Google’s own model, resolution, hardware and benchmark configuration. They are not a general 1.5× Tensor API advantage or XBSTACK data. Read Google’s demonstration scope.

Device compatibility and deployment boundaries:

A common trap is to confuse a portable graph description with portable accelerated execution. An exported model may load successfully while a specific operation remains unsupported by the GPU or NPU plugin, or a runtime may fall back to CPU without the intended performance benefit. Verify both the selected execution provider and an input/output correctness check on each target device. Consider accelerator library packaging, tensor memory alignment, buffer sharing, and synchronization separately; a desktop CPU run cannot establish mobile NPU readiness.

A production verification matrix

No performance or hardware-compatibility result is implied; all values require real testing.

Verification area	What to record	Common mistake
Reproducibility	LiteRT commit SHA, Bazel, compiler, OS, SDK and device	Citing a moving main branch as a stable benchmark
Numerical validity	Golden input/output, dtype, shape and error tolerance	Assuming matching shapes imply matching results
Operator support	Per-backend supported ops and actual fallbacks	Assuming that an NPU model option means NPU execution
End-to-end latency	Conversion, initialization, inference, post-processing and synchronization	Reporting only inner model latency
Memory and power	Startup/peak allocations, sustained usage and device conditions	Generalizing a brief desktop run to a phone
Device matrix	Exact iOS/Android model, driver, runtime and accelerator libraries	Extrapolating across unrelated hardware
Failure modes	Unsupported ops, dynamic shapes, memory pressure, recovery	Testing only one successful input

For serious on-device production work, collect P50/P95 latency, peak memory, numerical error, sustained thermal behavior, and actual selected backend on real target hardware. A faster operator microbenchmark is not necessarily a faster user task.

How Tensor API fits existing LiteRT-LM and on-device RAG systems

If a mobile application already uses LiteRT-LM for local language-model or embedding inference, Tensor API is not a mandatory replacement. Profile the complete task first: model-load time, KV-cache growth, vector search, embedding throughput, CPU/GPU transfers, and rendering may have very different bottlenecks. Introduce graph authoring only where custom operators, unsupported conversions or repeated pre/postprocessing transfers have a measurable impact. Keep the model runtime and retrieval index architecture independent when that is simpler to maintain.

When Tensor API is the wrong choice

Do not rewrite a working on-device inference stack merely because this API is new. If your model already converts correctly and CompiledModel achieves the required latency and accuracy, the extra C++ code and maintenance surface may not justify themselves.

The case becomes stronger when profiling reveals a genuine conversion limitation, an unsupported graph composition, or expensive pre/postprocessing buffer transfers — and the team can own device-specific regression testing.

Similarly, if you already operate LiteRT-LM to run local language models or embeddings, Tensor API is not a mandatory migration. Diagnose whether the bottleneck is graph operations, model loading, the KV cache, retrieval, network interaction or something else first.

Related XBSTACK material:

• EmbeddingGemma 2 on-device retrieval evaluation: an article that contains real XBSTACK-local evidence, unlike the upstream-only evaluation in this guide.
• Personal AI architecture: decide which tasks belong on-device and which require cloud infrastructure.
• AI Tools Lab: future independently verified tools and experiments.

FAQ

Does Tensor API replace model conversion? Not in general. The conversion path remains recommended for most established models. Tensor API is an alternative for cases that need direct graph authoring or integrated pipelines.

Can I run the same graph on CPU, GPU and NPU without changes? The graph authoring code can be reused across supported paths, but operator support, driver/runtime packaging and execution correctness must be checked per backend. There is no universal performance or compatibility guarantee.

Do I need a training framework? You may use existing compatible weights or build a graph from operators, but Tensor API does not perform the job of a complete training framework.

Are the benchmark numbers on this page XBSTACK’s? No. Any discussed SAM 2 comparison is explicitly identified as Google’s upstream report; XBSTACK has not run a matching experiment.

Primary sources and evidence limits

• Google: Native Model Authoring for Edge with LiteRT Tensor API, October 19, 2026
• LiteRT Tensor API: source README, backend tags, Bazel targets and Quick Start
• LiteRT Getting Started and CompiledModel guidance
• Google AI Edge sample applications and model recipes

Editorial verification note: API details and build commands were cross-checked against published first-party source material as of October 20, 2026. No unmeasured CPU/GPU/NPU speed, memory or power result is inferred from these documents.