Rust's borrow checker reaches CUDA — and the cost is zero

GPU kernels have long forced a hard trade: the raw speed of CUDA C++ or the memory safety Rust gives you on the CPU — rarely both at once. A new paper from NVIDIA and Hugging Face argues that trade is now optional.

What cuTile Rust Brings to CUDA Programming

cuTile Rust is a domain-specific language that lowers idiomatic Rust kernels to CUDA Tile IR while carrying Rust's &/&mut ownership contract across the CPU→GPU launch boundary. It was introduced in the arXiv paper "Fearless Concurrency on the GPU" (arXiv:2606.15991, posted June 14, 2026), authored by Melih Elibol, Jared Roesch, Isaac Gelado and Michael Garland of NVIDIA with Eric Buehler of Hugging Face . The claim is narrow but concrete: borrow-checked Rust GPU code can reach near-vendor-library performance with no measured overhead.

Quick Answer: cuTile Rust is an NVIDIA/Hugging Face DSL that extends Rust's borrow checker onto CUDA kernels, lowering them to Tile IR while preserving &/&mut contracts. In the June 2026 paper it hits ~96% of cuBLAS on GEMM and matches vLLM/SGLang at batch-1 decode — safety with no measured cost.

The core idea is structural. Each kernel runs with single-threaded semantics over fixed-size data tiles that the compiler maps to thread blocks . Branded partition indices and bounded iterators let the compiler prove disjoint memory access at compile time, which eliminates runtime bounds checks rather than just hiding them. Macro-generated host code (#[cutile::module]) splits a mutable output tensor into disjoint mutable sub-tensors, passes inputs as shared references, and JIT-compiles kernels to cubin — preserving ownership while GPU work is in flight. When you need lower-level control, explicit unsafe escape hatches remain.

What keeps this out of demo territory is the companion artifact. Grout, a Qwen3 inference engine built with cuTile Rust, is 100% Rust, Apache-2.0, and public at huggingface/grout . It supports CUDA-graph decode, device-side argmax token selection, and FP16 accumulation — an end-to-end decoder, not a microbenchmark wrapper.

The authors frame the contribution plainly: "memory-safe, data-race-free GPU code written in idiomatic Rust can reach near-vendor-library performance," per the team's writeup on the official Rust users forum . The sections below test how far that holds.

Carrying Rust Ownership Into CUDA: Partitions, Iterators, and JIT Launch

cuTile Rust carries Rust's ownership contract across the CPU→GPU launch boundary by proving disjoint memory access at compile time rather than checking it at runtime. The mechanism rests on two compiler-visible constructs: branded partition indices and bounded iterators. Each kernel runs with single-threaded semantics over fixed-size data tiles that the compiler maps to thread blocks, and the brands let the compiler verify that two partitions never alias — so the usual runtime bounds checks are eliminated, not merely optimized away . The result is that the safety reasoning happens entirely before the kernel ever launches.

The host side is where ownership is preserved while GPU work is in flight. The #[cutile::module] macro generates launch code that splits a mutable output tensor into disjoint mutable sub-tensors and passes inputs as shared references — the GPU equivalent of handing out one &mut per writer and & for every reader . Because the partitions are disjoint by construction, Rust's borrow checker accepts the split as sound, and the caller keeps ownership of the backing tensor for the duration of the launch. At launch the kernel is JIT-compiled to cubin, so the same source compiles for whatever target GPU is present rather than being baked at build time.

For a developer, the practical shape looks like this:

• No runtime bounds checks. Branded indices and bounded iterators discharge the disjointness proof at compile time, so the device code carries none of the per-access guard cost a naive safe abstraction would .
• Ownership survives the boundary. The macro-generated host code passes &/&mut sub-tensors, so aliasing bugs between concurrent writers are rejected by cargo build, not discovered in a profiler.
• Composable execution. Host-side launches compose across synchronous calls, asynchronous pipelines, and CUDA-graph replay, which is what lets an engine like Grout capture a decode step into a graph and replay it .

The model does not pretend the safe surface covers everything. Programmers keep unsafe escape hatches for the cases the safe API cannot express — explicit warp primitives, manual shared-memory management, and the lower-level control a SIMT CUDA kernel exposes directly . This is the same bargain Rust makes on the CPU: a checked default with a clearly marked door to unchecked control, rather than an all-or-nothing wall. The tile abstraction trades some of CUDA's lowest-level expressiveness for a default path the compiler can reason about, and the unchecked opt-out is there for the kernels that genuinely need it.

Two design choices are worth flagging for anyone evaluating this. First, the single-threaded-per-tile semantics is what makes the borrow reasoning tractable — concurrency lives in how tiles map to blocks, not in the kernel body the programmer writes. Second, JIT-to-cubin at launch is announced on the official Rust users forum as part of the workflow , which means iteration looks like ordinary Rust edit-compile-run rather than a separate device-toolchain step.

Safe Rust CUDA GEMM: 96% of cuBLAS, Zero Added Cost

The headline microbenchmark result is that cuTile Rust's safe kernels run at vendor-library speed with no measurable overhead. On an NVIDIA B200 in f16, element-wise operations reach about 7.02 TB/s — roughly 91% of peak DRAM bandwidth — and matrix multiply at M=N=K=8192 hits 2.07 PFLOP/s, reported as ~96% of cuBLAS in the paper abstract . The point is not the absolute figures but where they land: the borrow-checked path matches the raw-pointer and cuBLAS baselines, so the safety machinery described earlier carries no runtime tax in these tests .

For the bandwidth-bound case, the element-wise kernel matches cuTile Python within measurement noise . That equivalence matters because it shows Rust's `&`/`&mut` contract and the compile-time disjointness proofs add nothing the Python tile DSL doesn't already pay — the same generated Tile IR, the same memory traffic. There is one caveat worth flagging: the GitHub README frames the GEMM result as about 92% of dense f16 peak rather than 96% of cuBLAS . Those are two different denominators — peak hardware throughput versus the tuned vendor library — not a contradiction, but a reminder to read which baseline a percentage is measured against.

The more pointed evidence sits inside the GEMM comparison. The safe mapped kernel matches its raw-pointer variant, and the safe persistent GEMM lands within ~0.3% of the corresponding low-level Tile IR version . In other words, dropping to `unsafe` raw pointers buys essentially nothing here. That undercuts the usual assumption — borrow-checked or bounds-checked code is fine for orchestration but too slow for the inner loop — at least for the tile-mapped patterns these benchmarks exercise.

Source: "Fearless Concurrency on the GPU," arXiv:2606.15991 , and the cutile-rs repository .

The developer takeaway is concrete: the safety abstractions compile away. Branded partition indices, bounded iterators, and the disjoint-mutable-subtensor split all resolve at compile time, so the emitted device code is the same code you would have written by hand — without the proof. For GPU systems programmers, that reframes the trade-off. Compile-time guarantees against data races and out-of-bounds access stop being a feature you pay for in throughput, in these microbenchmarks at least. The open question is how far the result generalizes beyond the tile-mapped GEMM and element-wise shapes measured here — a boundary the authors are explicit about and one the next sections take up.

Grout on B200: 80 Tok/s for Qwen3-32B Decode

Grout, the Qwen3 inference engine the authors built on cuTile Rust, reaches 80.1 generated tokens/s on a B200 running Qwen3-32B at batch 1 — about 66.7% of the HBM roofline estimate — and 154.7 tok/s on an RTX 5090 running Qwen3-4B, roughly 74.7% of roofline . In the same Section 5.3 sweep at generated length tg=8192, vLLM 0.18.0 records 77.5 tok/s and SGLang 0.5.9 records 76.5 tok/s on the B200/Qwen3-32B case . That is single-stream decode parity with two engines that define the high-performance serving baseline, achieved by a 100% Rust stack .

Quick Answer: On a B200 running Qwen3-32B at batch 1, f16, with prefix caching disabled, Grout decodes at 80.1 tokens/s versus 77.5 for vLLM 0.18.0 and 76.5 for SGLang 0.5.9 — system-level latency parity, measured over 10 reps per cell with median and IQR shading.

The configuration matters as much as the numbers. Every run was f16, batch 1, prefix caching disabled — vLLM with CUDA graphs enabled, SGLang with RadixAttention turned off — so the comparison isolates single-stream decode latency rather than the high-concurrency throughput those systems are tuned for . Provenance is pinned: cuTile Rust 0.2.0, Grout commit 4631fb01, vLLM 0.18.0 and SGLang 0.5.9, with the RTX 5090 using 10 reps at shorter sizes and 3 reps at 2048/8192, and the B200 using 10 reps per cell .

One caveat keeps this honest: Grout's model GEMMs still fall back to cuBLAS . So the headline is not a pure Rust-versus-C++ kernel shootout. What the safe Rust path owns end to end is the engine — CUDA-graph decode, device-side argmax and token selection, FP16 accumulation, greedy or sampled generation over Qwen3 safetensors — while the densest matmuls delegate to NVIDIA's library. The result is a system-level decode-speed claim: a borrow-checked Rust inference loop orchestrating cuBLAS reaches batch-1 parity with mature serving stacks, the kind of local-LLM latency target practitioners increasingly benchmark on consumer cards like the RTX 5090 (video: Alex Ziskind).

"Grout is a minimal testbed, not a general-purpose serving-stack replacement; concurrent-batching throughput is left to future work," — Elibol, Roesch, Gelado, Garland and Buehler, "Fearless Concurrency on the GPU" (source: arXiv:2606.15991).

That framing is the point. The repo carries few commits and no tagged releases, and the headline batch-1 figures published elsewhere — 171 tok/s for Qwen3-4B on the RTX 5090 and 82 tok/s for Qwen3-32B on the B200 — describe single-stream latency, not multi-tenant server throughput. Read narrowly, the evidence is strong: safe Rust can drive a competitive decode loop today. Read broadly, it is unproven, and the authors say so.

How the Authors Bound the Claim

The parity claim is deliberately narrow: Grout matches vLLM and SGLang only at batch 1, single-stream decode — not the multi-tenant concurrency those engines are built around. The Section 5.3 sweep reports Grout at 80.1 tok/s on a B200 with Qwen3-32B versus 77.5 tok/s for vLLM 0.18.0 and 76.5 tok/s for SGLang 0.5.9, all at f16, batch 1, prefix caching disabled . The authors state plainly that Grout is not a general-purpose serving-stack replacement and that high-concurrency batched throughput — the regime where PagedAttention, continuous batching, and RadixAttention earn their keep — is separate future work . Single-stream latency parity is the result; server throughput parity is not claimed.

The safety model also costs surface area. The paper's limitations section notes that cuTile Rust exposes a reduced low-level-control surface compared with SIMT CUDA: explicit warp primitives and manual shared-memory management are not fully expressible in the safe API, and programmers fall back to unsafe escape hatches when they need that control . GEMM still has gaps at some sizes, and Grout's model GEMMs fall back to cuBLAS rather than running entirely on cuTile kernels .

The engineering maturity is early by the authors' own framing. Several constraints bound how far the result generalizes today:

• Scope is NVIDIA-only. The pipeline targets CUDA Tile IR and requires compute capability sm_80 or higher — A100, H100, B200 — with no portability story to other vendors .
• Model coverage is thin. Grout supports only Qwen3 directories stored as safetensors, with no speculative decoding and no multi-GPU execution .
• The testbed is minimal. The public Grout repository carries few commits, no tagged releases, and no benchmark numbers committed to the repo — the headline figures live in the paper, not the code .

Read against vLLM and SGLang's actual feature sets — continuous batching, prefill-decode disaggregation, and deployment claims spanning over 400,000 GPUs for SGLang — the gap is one of breadth, not peak single-stream speed. That framing is the authors' own, and it is why the borrow-checker-on-CUDA story is credible: the claim is small enough to be true, and the primary source draws the boundary itself.

cuTile Rust 0.2.0: Apache 2.0, sm_80+, and Uncovered Areas

cuTile Rust 0.2.0 is an early-stage, NVIDIA-only toolchain with concrete hardware and software floors: it requires NVIDIA GPUs of compute capability sm_80 or higher — A100, H100, or B200 — along with CUDA 13.3 and Rust 1.89+, and is tested on Ubuntu 24.04 . Version 0.2.0 landed on June 16, 2026, and the repository is explicitly labeled early-stage with expected API breakage, so anyone adopting it now is building on a moving target .

Licensing is split. The Rust crate at NVlabs/cutile-rs is Apache 2.0, while the bundled cuda-bindings fall under the NVIDIA Software License — a distinction that matters for teams with strict dependency-licensing review . Grout, the Qwen3 inference testbed, is separately Apache 2.0 at huggingface/grout and is 100% Rust .

On reproducibility, the authors publish the artifacts needed to re-run the paper's benchmarks but draw a deliberate line. The repository ships benchmark code and provenance — cuTile Rust 0.2.0, a pinned Grout commit, vLLM 0.18.0, and SGLang 0.5.9 — while raw profiling traces, model weights, build artifacts, and large generated outputs are intentionally omitted . That keeps the repo small but means a full bit-for-bit replay still requires supplying your own weights and profiling setup.

The known gaps are stated plainly rather than buried. Relative to SIMT CUDA, cuTile Rust exposes a reduced low-level-control surface: no explicit warp primitives and no manual shared-memory management, with remaining GEMM gaps at some sizes and a fallback to cuBLAS for Grout's model GEMMs . The tensor API is young, some raw-pointer use remains, and the scope is NVIDIA-only by construction since Tile IR and the sm_80+ floor are NVIDIA-specific .

One signal worth reading: the work was announced not only on arXiv but on the official Rust users forum, where the authors framed it as safe GPU kernels in Rust . Posting to users.rust-lang.org rather than only ML-infrastructure channels suggests intent to engage the broader Rust systems community, not just inference-engine teams — a positioning choice that fits a project still early enough to absorb language-level feedback.

What This Means for Rust ML Framework Authors

For Rust ML framework authors, the practical signal from cuTile Rust is that Rust is now viable below the host-side tensor API layer — not only as orchestration glue around C++ kernels, but for the CUDA code itself. The paper demonstrates safe, borrow-checked GPU kernels reaching about 96% of cuBLAS on an 8192³ GEMM and roughly 91% of peak DRAM bandwidth on element-wise work . That undercuts the long-standing assumption that compile-time safety on the GPU must cost throughput, and it opens a path to writing custom fused kernels without pervasive unsafe or hand-written CUDA C++.

The primary audience is GPU systems programmers who want compile-time race elimination at the kernel level. cuTile Rust extends Rust's &/&mut contract across the launch boundary, so the disjoint-access guarantees that make CPU Rust safe now apply to thread-block-mapped tiles . For teams maintaining Rust inference stacks, that means a custom attention or normalization kernel can carry the same ownership discipline as the surrounding host code.

The honest part of the story is the cuBLAS fallback. Grout still defers its model GEMMs to cuBLAS rather than running them through cuTile Rust end to end , which marks exactly where the safe Rust path is already production-grade and where it still defers to a vendor library. That candor matters more than a clean parity headline.

As the authors put it in the paper, the contribution is bounded but specific: "memory-safe, data-race-free GPU code written in idiomatic Rust" reaching near-vendor-library performance, per Melih Elibol and co-authors of NVIDIA and Hugging Face (source: arXiv:2606.15991) .

Two things are worth watching. First, whether API stabilization follows 0.2.0 — the project is explicitly labeled early-stage with expected breakage , so framework authors should pin versions and expect churn. Second, whether coverage expands beyond NVIDIA: today the work is Tile-IR-bound and requires sm_80 or higher , and AMD/ROCm or Apple Metal targets would change the portability calculus. The concrete takeaway: if you build Rust ML infrastructure, cuTile Rust is worth a prototype kernel today — just treat 0.2.0 as a research preview, not a stable dependency.

Frequently asked questions

What hardware does cuTile Rust require?

cuTile Rust runs only on NVIDIA GPUs of compute capability sm_80 or higher — A100, H100, and B200 — and needs CUDA 13.3, Rust 1.89+, with testing done on Ubuntu 24.04 . There is no AMD/ROCm or Apple Silicon path today; the model lowers to NVIDIA's CUDA Tile IR, so the scope is NVIDIA-only for now . Treat the supported matrix as narrow and version-pinned.

Does adding Rust borrow-checking to CUDA code cost runtime overhead?

Per the paper's microbenchmarks, no. The safety abstractions compile away: on GEMM (M=N=K=8192, NVIDIA B200, f16) the safe mapped kernel matches its raw-pointer variant, and the safe persistent GEMM lands within roughly 0.3% of the corresponding low-level Tile IR version . Element-wise operations reach about 7.02 TB/s, near 91% of peak DRAM bandwidth and on par with cuTile Python within measurement noise . The borrow-check contract is enforced at compile time, not at runtime.

Is Grout ready to replace vLLM or SGLang in production?

No. The authors explicitly describe Grout as an experimental testbed that demonstrates cuTile Rust end-to-end, not a general-purpose serving stack . It is batch-1, Qwen3-only, with no multi-GPU and no concurrent batching, and the public repo is minimal — few commits, no tagged releases, and no benchmark numbers committed . Production serving needs the high-concurrency throughput that vLLM and SGLang are built for and that Grout does not yet address.

How should I read the batch-1 comparison to vLLM and SGLang?

As single-stream latency parity, nothing broader. Under matched conditions (B200, Qwen3-32B, f16, batch 1, prefix caching disabled, tg=8192), Grout reaches 80.1 tok/s versus 77.5 tok/s for vLLM and 76.5 tok/s for SGLang . That verifies batch-1 decode parity against established engines, not toy baselines. Multi-tenant, high-concurrency throughput — where vLLM's PagedAttention and SGLang's RadixAttention are optimized — is untested here and flagged by the authors as future work .

What licenses cover cuTile Rust and Grout?

cuTile Rust is dual-licensed: the Rust crate (cutile-rs, version 0.2.0 as of June 16, 2026) is Apache 2.0, while its cuda-bindings fall under the NVIDIA Software License . Grout (huggingface/grout) is fully Apache 2.0 and 100% Rust . If you vendor either, note the split license boundary and that 0.2.0 is early-stage with expected API breakage.