cuTile Rust at 96% cuBLAS — Grout's engine wins need context

A new NVIDIA–Hugging Face paper makes a claim that GPU programmers have spent a decade assuming was impossible: you can carry Rust's borrow checker all the way onto the device and pay almost nothing for it.

What cuTile Rust Does: Borrow-Checker Rules Inside CUDA Tile Programs

cuTile Rust is a tile-based Rust DSL and runtime that extends Rust's ownership and borrow-checking rules into CUDA tile programs themselves, rather than wrapping CUDA calls from the CPU side. Its headline result, from the June 2026 arXiv paper "Fearless Concurrency on the GPU" (Elibol et al., NVIDIA and Hugging Face), is that a safe persistent f16 GEMM at M=N=K=8192 reaches 2.07 PFlop/s on a B200 — about 96.4% of cuBLAS and within 0.3% of the unsafe low-level Tile IR variant . The safety mechanism, in other words, carries no measurable runtime overhead, which is precisely the tax that historically made safe-language GPU programming a non-starter.

Quick Answer: cuTile Rust pushes Rust's borrow checker into CUDA device code, eliminating data races at compile time. The proof it works without a performance penalty: a safe f16 GEMM on the B200 hits 2.07 PFlop/s — 96.4% of cuBLAS — per arXiv:2606.15991 (NVIDIA/Hugging Face, June 2026).

The novelty is not that Rust can call CUDA — host-side Rust engines already do. It is that the aliasing-XOR-mutability invariant — a reference is either uniquely mutable or freely shared, never both — is enforced structurally inside the kernel. cuTile Rust partitions mutable output tensors into disjoint sub-tensors, one per tile program, while immutable reads are broadcast to every program . Generated launchers hold ownership of the data while GPU work is in flight, so the host cannot touch buffers a kernel still owns.

Ordering within a single program is handled by Tile IR "tokens," which sequence memory operations so reads and writes cannot be reordered into a race. Kernels lower through CUDA Tile IR, and the unusual rigor is that the paper proves data-race freedom for the safe API against Tile IR's weakly-ordered memory model, with the formal argument in Appendix A . Races are designed out at compile time, not caught after the fact at runtime.

"We map the borrow checker's aliasing-XOR-mutability invariant onto GPU execution and prove the safe API is data-race-free against Tile IR's weakly-ordered memory model," write the authors of arXiv:2606.15991 — Elibol, Roesch, Gelado and Garland (NVIDIA) with Eric Buehler (Hugging Face).

Two artifacts ship with the work: cuTile Rust itself, released as NVlabs/cutile-rs, and Grout, a Qwen3 inference engine built on top of it, published as huggingface/grout . The sections below test how far that 96% number actually reaches — and where the safety guarantee quietly hands off to cuBLAS.

GEMM and Bandwidth: cuTile Rust on B200 vs cuBLAS

On NVIDIA's B200, safe cuTile Rust kernels run within a few percent of the fastest unsafe code on the same hardware. A safe persistent f16 GEMM at M=N=K=8192 hits 2.07 PFlop/s — about 96.4% of cuBLAS and within 0.3% of the low-level Tile IR variant . The headline claim of the paper rides on that gap: that the borrow-checker safety carried across the launch boundary costs no measurable throughput for the tile abstraction.

Memory-bound work tells the same story. An element-wise add reaches 7.02 TB/s against a roughly 7.68 TB/s theoretical peak — about 91.4% of bandwidth — and the broader sweep lands near 2 PFlop/s on GEMM and roughly 7 TB/s element-wise . The committed benchmark folder in NVlabs/cutile-rs makes these reproducible against the paper's exact configuration, which is more transparency than most vendor benchmark claims ship with.

The detail worth pinning down is why the overhead is near zero, because the paper is careful here and a reader should be too. The result is empirical, not architectural . The tile abstraction operates at a granularity that avoids the fine-grained synchronization a thread-level safe model would need — sub-tensor partitioning and Tile IR token ordering do their work at compile time, so there is no runtime guard tax to pay. But the authors do not claim safety overhead is structurally impossible; they claim that for this specific abstraction, on these kernels, it measures as zero. That is a narrower and more honest statement than "safe is free."

So the 96.4% figure is real and well-instrumented, but it describes the kernel core, not an end-to-end engine. The GEMM that hits 2.07 PFlop/s is a clean, regular shape at one large size. The next sections follow the number outward — into Grout's decode loop — and find where the safe path quietly hands the largest GEMMs back to cuBLAS.

Grout Is a Qwen3 Decode Artifact, Not a Scheduler

Grout is a Qwen3 inference engine written entirely in Rust on top of cuTile Rust, published as huggingface/grout with just six commits and no tagged releases as of June 16, 2026 . Its own README scopes it as a test bed for exercising cuTile Rust kernels, not a serving stack — which is the right frame for reading every number attached to it.

"Grout is an LLM inference test bed, not a general-purpose serving engine," — project README, huggingface/grout.

That scoping matters because Grout is also not pure safe Rust end to end. It uses safe cuTile Rust kernels for the fused and model-specific operations that ring the GEMM core, but it hands the largest matrix multiplications back to cuBLAS and reaches for raw-pointer kernels and unchecked_accesses elsewhere . So the engine is a mixed-safety harness: the safe path covers the custom ops the paper wants to showcase, while the heavy linear algebra runs on the same vendor library every other engine uses.

The feature gaps are equally deliberate. Grout has no continuous batching, no prefix cache, no distributed parallelism, and no OpenAI-compatible endpoint — the scheduling machinery that defines vLLM and SGLang as production frameworks is simply absent . What remains is a single-request decode loop. The reported runs — 171 generated tokens/sec for Qwen3-4B on an RTX 5090 and 82 tokens/sec for Qwen3-32B on a B200 — are batch-1, single-request measurements .

Read that way, Grout is testing cuTile Rust ops in isolation, not scheduling or multi-user throughput. It answers "does a safe kernel keep up token-for-token in a clean decode loop?" and leaves "does the engine schedule a crowd?" untouched. That distinction sets up the next question: why the cross-engine comparison disabled the very caches that make vLLM and SGLang fast under load.

Why the Grout Decode Comparison Disabled Prefix Cache

The cross-engine numbers were collected with the rival engines' fastest paths switched off. vLLM 0.18.0 ran with CUDA graphs and prefix cache disabled, and SGLang 0.5.9 ran with RadixAttention and prefix cache disabled . That is by design, not an oversight: to ask whether a safe cuTile Rust kernel keeps pace with hand-tuned CUDA inside a decode loop, you have to strip out everything that isn't the kernel — caching, graph capture, and scheduler tricks that would otherwise mask kernel-level differences.

The metric makes the scope explicit. The cross-engine figure, request_gen_tps, is generated tokens divided by total end-to-end wall-clock request time . Each request is timed cold, one at a time. So the comparison deliberately excludes the machinery that actually decides production throughput:

• PagedAttention — vLLM's non-contiguous KV-cache allocation that lets many sequences share GPU memory .
• Continuous batching — interleaving new and in-flight requests so the GPU never idles between tokens.
• Prefix / cache reuse — RadixAttention and prefix caching that skip recomputation when prompts overlap.
• Cold-start amortization — costs that disappear once a server is warm and saturated with concurrent traffic.

With those features off, Grout, vLLM, and SGLang are reduced to roughly the same thing: a single request, batch size one, measuring how fast each emits tokens through its decode kernels. That is a fair test of kernel competitiveness, and it is the test the paper set out to run. It is not a scheduler test, and it is not a server-throughput test.

This is the lens through which "competitive with vLLM and SGLang" has to be read. The claim holds for batch-1 cold-start decode, where the kernel is the bottleneck and Grout's safe cuTile Rust ops land within a few percent of the incumbents . It says nothing about multi-user load, where PagedAttention and continuous batching are precisely what let vLLM and SGLang pull ahead. Reading the headline without the disabled-cache footnote turns a narrow, honest kernel result into a production-serving claim the paper never makes — and the next section's CSV margins show just how narrow that batch-1 window really is.

Grout Decode CSVs: RTX 5090 and B200 Margin Analysis

The committed CSVs quantify exactly how thin the batch-1 lead is: Grout wins, but by single-digit percentages. On a GeForce RTX 5090 running Qwen3-4B at pp=18/tg=36, median request_gen_tps is 170.52 tok/s for Grout versus 163.63 for vLLM and 162.52 for SGLang — a 4.2% edge over the nearest rival . On an HGX B200 running Qwen3-32B at the same config, Grout posts 80.85 against vLLM's 78.31 and SGLang's 75.49, a 3.2% margin . These were measured with Grout commit 4631fb0, vLLM 0.18.0 and SGLang 0.5.9 .

Median request_gen_tps (generated tokens / end-to-end request time), batch-1 decode .

Two patterns matter. First, the headline "82 tok/s for Qwen3-32B on B200" is not the pp=18/tg=36 figure — it tracks Grout's peak of about 81.56 tok/s at tg=512, which rounds to 82 . Second, the lead shrinks as sequences grow. At tg=8192 on the RTX 5090, Grout (154.75) and SGLang (154.26) sit within half a token per second, with vLLM at 151.26 .

The prefill picture is consistent with this: Grout's latency advantage is clearest at short prompts on the RTX 5090, while on the B200 the three engines converge at the longest sequences . Read literally, the CSVs say Grout is genuinely the fastest of the three in single-request decode — but the 4–7% margins are real and narrow, and they evaporate at long context. That is a defensible kernel-quality signal, not a serving-throughput verdict.

Fearless Concurrency on the GPU (arXiv:2606.15991) and the committed benchmark CSVs let anyone re-derive these numbers directly.

cuTile Rust v0.2.0: Race Freedom Stops at Large GEMMs and SIMT Ops

The safety guarantee is partial, and the boundary sits exactly where transformer compute is heaviest. Large-matrix GEMMs — the dominant operation in a transformer forward pass — fall back to cuBLAS rather than running through safe cuTile Rust kernels, so the formally proven data-race freedom does not cover the hottest path. Grout's own README confirms it mixes safe kernels with unchecked_accesses, raw-pointer kernels and cuBLAS GEMM fallbacks . cuTile Rust currently covers the custom fused and model-specific operations around the GEMM core, not the GEMM itself.

The abstraction also gives up SIMT-level control. Warp primitives and explicit shared-memory management sit outside the tile model, and unsupported lower-level cases use explicit opt-outs that are functionally equivalent to unsafe paths . For kernel authors who reach for warp shuffles or hand-tuned shared-memory staging, the tile DSL is not yet a replacement — it is a higher-level layer that trades that control for the structural race guarantee. Tensor API coverage is also incomplete, and there is no large-batch serving comparison or third-party reproduction yet.

Maturity is the other constraint. NVlabs labels cuTile Rust early-stage research with expected bugs, incomplete features and API breakage, and v0.2.0 was the latest release as of June 16, 2026 . The toolchain requirements are narrow:

• GPU: NVIDIA sm_80+ (Ampere or newer)
• CUDA: 13.3 recommended
• Rust: 1.89+
• OS: tested on Ubuntu 24.04 — no Windows or macOS support documented

Read together, these bounds explain why the 96.4% cuBLAS GEMM result and the partial safety story are not in tension: cuTile Rust proves race freedom for the tile API it covers, then hands the largest GEMMs back to cuBLAS where that proof does not extend. The guarantee is real within its scope; the scope is the thing to watch as the Tensor API and SIMT coverage fill in.

Should You Write Rust CUDA Ops With cuTile? The Honest Take

The answer depends entirely on who you are. For GPU kernel researchers and compiler authors, NVlabs/cutile-rs is worth tracking now: extending Rust's aliasing-XOR-mutability invariant across the host/device launch boundary, then proving data-race freedom for the safe API against Tile IR's weakly-ordered memory model, is a genuine contribution independent of Grout's decode margins . The 96.4% of cuBLAS GEMM result shows the safety tax can be near zero for the tile abstraction — that is the durable finding.

For production LLM serving, the recommendation is different: Grout is not a drop-in. The comparison left vLLM and SGLang's defining features — PagedAttention, continuous batching, prefix caching, and multi-GPU distribution — out of scope, so nothing here tests them. Grout's own authors are explicit about the framing.

"Grout is an inference test bed, not a general-purpose serving stack." — Grout authors (source: huggingface/grout README)

Do not conflate this work with rvLLM (m0at/rvllm), a separate Rust-native engine. Its "safety" is host-side engineering discipline — Result-based errors, no unwrap() — calling CUDA, not device-code ownership proofs. It targets a different design point, reporting 5,802 tok/s at batch 128 on an H100 against vLLM's 4,689 . That is large-batch throughput; the cuTile paper measures batch-1 decode. Both are credible Rust signals; they answer different questions.

What to watch over the next releases: v0.3.0 GEMM coverage in cuTile Rust, Tensor API completeness, and — most decisive — whether any third-party team reproduces Grout's margins with prefix cache enabled on both sides. Until then, the concrete takeaway: adopt cuTile Rust today for authoring fused, model-specific kernels where you want compile-time race freedom, keep cuBLAS for the largest GEMMs, and keep vLLM or SGLang for anything serving real traffic.

Frequently asked questions

Is Grout a drop-in replacement for vLLM or SGLang?

No. Grout's README explicitly scopes it as an inference test bed, not a general-purpose serving stack . It has no continuous batching, no prefix cache, and no multi-GPU parallelism — the production scheduling features that vLLM and SGLang are built around. The paper's benchmark ran vLLM with CUDA graphs and prefix cache disabled, and SGLang with RadixAttention disabled, so the comparison is batch-1, single-request decode, not an apples-to-apples production serving test . For real traffic, keep vLLM or SGLang.

What makes cuTile Rust different from host-side Rust CUDA wrappers?

Existing Rust-CUDA wrappers call device code from Rust but enforce nothing about memory safety inside the GPU tile programs themselves. cuTile Rust carries Rust's borrow-checker semantics across the host/device launch boundary: mutable output tensors are partitioned into disjoint sub-tensors, one per tile program, while immutable reads are broadcast to all programs . The result is that data races are eliminated structurally at compile time, and the paper formally proves data-race freedom for the safe API against Tile IR's weakly-ordered memory model in Appendix A .

Why does Grout fall back to cuBLAS for large GEMMs?

Because cuTile Rust's Tensor API is incomplete at v0.2.0, and large matrix multiplications — the dominant operation in transformer layers — are not yet covered by the safe tile abstraction . Grout therefore routes its largest GEMMs to cuBLAS and uses raw-pointer kernels and unchecked accesses elsewhere, so it is not pure safe Rust end-to-end . cuTile Rust covers the custom fused and model-specific operations around the GEMM core, while cuBLAS still handles the heavy multiply.

How does rvLLM relate to cuTile Rust and Grout?

They are separate projects with different safety models and should not be conflated. rvLLM (m0at/rvllm) is a host-side Rust engine that calls CUDA and needs no Docker, conda, PyTorch or vLLM; its safety comes from engineering discipline — no unwrap(), Result-based error handling — not from formal device-code ownership proofs . On an H100 it reports 5,802 tok/s at batch-128 versus vLLM's 4,689 . The NVIDIA/Hugging Face work, by contrast, concerns formal safety of GPU device code itself.

What hardware and software does cuTile Rust require?

cuTile Rust requires an NVIDIA sm_80+ GPU (Ampere or newer), with CUDA 13.3 recommended, Rust 1.89+, and testing done on Ubuntu 24.04; v0.2.0 was the latest release as of June 16, 2026 . The paper's headline benchmarks use Blackwell SXM hardware in the HGX B200 . NVlabs flags expected bugs, incomplete features and API breakage at this stage, so treat cuTile Rust as research-grade tooling rather than production-ready infrastructure.