A "354M softmax-free open-weights model with custom Triton tile-skip kernels" has been making the rounds — but the parameter count doesn't match anything you can actually download, and the part that's real isn't a checkpoint at all.
Does AdaSplash-2's softmax-free tile-skip hold up?
The work behind the chatter is AdaSplash-2, which replaces dense softmax attention with α-entmax — a normalizer that assigns exact zero probability to low-scoring tokens, so a Triton kernel can skip entire all-zero attention tile blocks instead of traversing the full score matrix. This is exact sparsity, not approximate pruning: blocks are skipped because they are genuinely empty. The paper (arXiv:2604.15180, submitted April 2026, BSD-3-Clause) trains 350M and 1B LLaMA-3-style causal LMs to test the idea at scale .
Quick Answer: AdaSplash-2 swaps softmax for α-entmax, which produces exact-zero attention weights so a Triton kernel skips empty tile blocks. At 1B scale with 32K context, α-entmax+NAPE beats softmax+NAPE on RULER at every length (54.7 vs. 48.1 at 4K). But no public 350M/354M checkpoint exists — only the kernel is released.
The quality signal is real. On RULER at 1B scale with 32K context extension, α-entmax+NAPE beat softmax+NAPE at every tested length — 54.7 vs. 48.1 at 4K and 39.4 vs. 36.8 at 32K . So the mechanism holds up where it matters: long-context retrieval.
The "354M open-weights" claim is what doesn't. Primary sources confirm no 350M or 354M causal-LM checkpoint: SARDINE Lab's Hugging Face org lists SparseModernBERT-alpha2.0 (~0.1B, MIT-licensed), not the scaled models from the paper . What you can actually use today is the kernel — pip-installable, BSD-3 — at github.com/deep-spin/adasplash. Treat this as research-grade infrastructure to drop into your own training run, not a ready-made model to load.
How softmax prevents exact sparsity
Softmax can never produce a true zero. Its exponential normalization divides each score's exponent by the sum of all exponents, so every key — however low its score — keeps a nonzero share of the probability mass. That is the structural reason a tile-skip kernel cannot trust a softmax matrix: there is no safe "this block is all-zero" condition to test, and pruning small values after the fact introduces numerical divergence because the surviving weights no longer sum to one. α-entmax breaks that guarantee by design. With α > 1, the normalizer solves a per-row optimization that maps low-scoring tokens to exactly zero at training time, yielding a hard sparse distribution rather than a rounded-down softmax .
The downstream consequence matters more than the math. Because the score matrix is genuinely sparse, AdaSplash-2's Triton kernel can build a bit-packed block mask and visit only nonzero query/key tiles on both the forward and backward passes . Exact zeros are the precondition for the skip; small-but-positive softmax weights would force the kernel to traverse every tile regardless.
Exact sparsity also dissolves a familiar softmax artifact. Attention sinks — the persistent high-score tokens that models lean on simply because softmax forbids assigning zero mass anywhere — are not patched but structurally removed: α-entmax eliminates the mechanism that creates them rather than suppressing the symptom.
"The differentiator is dynamic, exact sparsity exploited end-to-end in the kernel, including the backward pass," — the AdaSplash-2 authors, SARDINE Lab (source: arXiv:2604.15180).
α-entmax is not the only route to this property. Threshold Differential Attention (TDA), from Snap Research with collaborators at Oxford and CMU and accepted to ACL 2026 , reaches exact zeros by a different path — thresholding differential attention scores — and also ships fused Triton tile-skip kernels . The two efforts are independent and contemporaneous; they converge on the same kernel-level payoff from opposite ends of the normalizer design space.
The histogram threshold solver: 1–2 iterations
AdaSplash-2's central systems contribution is how it finds the per-row α-entmax threshold cheaply. Because α-entmax assigns exact zero mass below a per-row cutoff, every kernel pass must first solve for that cutoff — and the original AdaSplash (arXiv:2502.12082, 2025) did it with multiple streaming passes over the raw attention scores . At long context that repeated traversal is the bottleneck: the cost of locating the threshold scales with the same sequence length the sparsity is supposed to make cheap.
AdaSplash-2 collapses that work into a single pass. It streams the attention scores into an 8-bin histogram held in on-chip SRAM, derives a bounded lower estimate for the threshold directly from that histogram, then hands the estimate to a safeguarded hybrid solver that refines it . Starting the solver from a histogram-derived bound rather than a naive guess is what cuts convergence to typically 1–2 refinement iterations , instead of the multi-pass scan the 2025 version required.
The same kernel pass also builds the structure that makes tile-skipping possible. As scores stream through, AdaSplash-2 constructs a bit-packed binary block mask over the query/key tiles — default configuration is 8 histogram bins, 64-bit packing, and a 64-token key block, giving a stated capacity of 16,320 keys per query block before the kernel flushes . The mask is traversed with GPU-native find-next-set and population-count operations, so the kernel jumps straight to the next nonzero block rather than iterating over zeroed tiles.
Both directions reuse that mask. The forward and backward passes visit only the nonzero blocks recorded in the bit-packed layout, and the authors are explicit that the backward pass is where the wall-clock gap against dense attention is widest — gradient computation over a dense score matrix is the expensive half of training, so skipping its zero tiles end-to-end is what drives most of the training-time saving . The implementation is public under BSD-3-Clause at github.com/deep-spin/adasplash, requiring PyTorch, Triton, and CUDA to run the kernels.
TDA from Snap Research: a concurrent ACL 2026 approach
AdaSplash-2 is not the only softmax-free tile-skip kernel shipping this year. Threshold Differential Attention (TDA), from Snap Inc. with collaborators at Oxford and CMU, takes the same high-level path — drop softmax, produce exact zeros, fuse the result into a Triton kernel that skips zeroed tiles — but arrives at it through a different mechanism. TDA applies a learned hard threshold to differential attention scores, zeroing everything below the cutoff rather than solving an α-entmax per-row normalizer. It was submitted to arXiv in January 2026, revised to v2 in April 2026, and accepted to ACL 2026 . The code is public and MIT-licensed at github.com/snap-research/TDA .
The detail that matters for anyone tracking the "354M" figure: TDA's publicly benchmarked checkpoint is a GPT-2 variant at roughly 162M parameters . That does not match 354M, and no confirmed TDA release does. The 162M model is also where TDA's reported results live, so its numbers should not be borrowed to characterize AdaSplash-2's 350M/1B LLaMA-3-style runs.
Treat the two projects as concurrent and independent, not a single coordinated drop. They were submitted separately, are architecturally distinct — a learned threshold on differential scores versus a histogram-initialized α-entmax solver — and report different models at different scales. Conflating their parameter counts or benchmark tables produces claims neither paper supports. The shared caveat reinforces the point: as of the papers' submission dates, neither project exposes a confirmed causal-LM checkpoint at 350M-plus on Hugging Face, and TDA's GitHub shows the kernel and training code but no visible checkpoint link in the primary sources . What is genuinely released in both cases is the kernel, not the weights.
RULER at 32K: α-entmax vs. softmax head-to-head
On RULER long-context evaluation at 1B scale, α-entmax with NAPE positional encoding beat the softmax+NAPE baseline at every measured context length — 54.7 vs. 48.1 at 4K, 49.9 vs. 46.4 at 8K, 45.4 vs. 43.3 at 16K, and 39.4 vs. 36.8 at 32K . The margin holds across the range rather than appearing only at the extremes, which is the cleaner signal: exact sparsity does not trade away quality at the lengths where it should help most.
| Context length | α-entmax + NAPE | Softmax + NAPE | Margin |
|---|---|---|---|
| 4K | 54.7 | 48.1 | +6.6 |
| 8K | 49.9 | 46.4 | +3.5 |
| 16K | 45.4 | 43.3 | +2.1 |
| 32K | 39.4 | 36.8 | +2.6 |
The training recipe behind these numbers matters for anyone trying to reproduce them. Both the 350M and 1B LLaMA-3-style models were trained on 50B DCLM-Edu tokens across 4× NVIDIA H100s, with a 4096-token base context, a roughly 524k-token effective batch, and a WSD schedule over 100k total steps . The 32K results come from a ProLong-style context-extension phase applied in the final 20% of steps — the RULER scores reflect that extended model, not the base 4K run.
Efficiency tracks block sparsity directly. At head dimension 64 in bf16, AdaSplash-2 matches FlashAttention-2 at low block sparsity and pulls ahead at moderate-to-high sparsity, with the crossover cited at roughly 60% block sparsity and a claimed greater-than-2× speedup in the high-sparsity limit . Below that crossover you pay the threshold-solve overhead without recovering enough skipped tiles to win, so this is not a free swap at short context.
One caveat the authors raise themselves: α-entmax models lag softmax on LAMBADA perplexity, and the paper flags it as an open problem rather than a solved one . Read the RULER table as evidence of long-context parity-or-better, not as blanket quality parity across every benchmark. The reference kernels are public under BSD-3-Clause , so the sparsity-vs-speed crossover is something you can measure on your own workload instead of taking on the paper's word.
The BSD-3 license is out — but where is the 350M checkpoint?
Installing the kernels is the easy part. Run pip install 'adasplash>=0.2.0' and supported default attention calls route to the AdaSplash-2 Triton path automatically . It depends on PyTorch and Triton, and needs CUDA to execute — the paper's runs are on NVIDIA H100 . So you can wire the kernel into a training loop today; what you cannot yet do is download the headline model.
SARDINE Lab's Hugging Face org does publish usable artifacts: SparseModernBERT-alpha1.5 and alpha2.0, an MIT-licensed ~0.1B masked-LM that swaps softmax for AdaSplash via Triton . Those are public and runnable, but note the mismatch with the topic: ~0.1B not 350M, and a masked encoder rather than a causal decoder. They prove the mechanism ships in weights — they are not the 350M/1B causal-LM checkpoints the paper trained on 50B DCLM-Edu tokens .
As of publication, no 350M (or "354M") or 1B AdaSplash-2 causal checkpoint is confirmed on the org page or in the deep-spin/adasplash repo . If you intend to cite an open-weights AdaSplash-2 model in a README or dependency list, verify the current availability yourself first — treat the "354M open weights" framing as unconfirmed until a card appears.
The same caveat applies to the concurrent work: TDA's repository exposes its fused kernel under MIT, but the primary sources reviewed surfaced no Hugging Face link or downloadable checkpoint . For both projects, the kernel is the deliverable you can actually depend on; the weights are not.
Under what conditions AdaSplash-2 outpaces dense attention
AdaSplash-2 outpaces dense attention only when block sparsity is high enough to pay back its bookkeeping cost. The paper reports it matches or improves per-step training time versus FlashAttention-2 once block sparsity is moderate-to-high — with about 60% cited as an example threshold — and reaches more than 2× speedup in the high-sparsity limit at head dimension 64 in bf16 . Below that, the histogram threshold solve and mask construction become overhead the dense kernel never pays, so the two roughly break even.
Sparsity, in turn, is driven by context length. The 1B context-scaling runs used block-sparsity ratios learned by a 1B Transformer and evaluated lengths up to 128K, with the authors arguing sparsity grows as sequences lengthen . The practical reading: the efficiency case is strongest for long-context training at 32K and beyond, and weakest for short-context fine-tuning, where there are few all-zero tiles to skip.
Where the gap actually opens is the backward pass. The authors attribute much of the training-time gain to backward-pass tile-skipping, which means pure inference at short context — a forward-only, low-sparsity regime — sees limited benefit. One more constraint: the kernels require PyTorch plus Triton and CUDA to run, and were benchmarked on 4× NVIDIA H100 . Neither AdaSplash-2 nor the concurrent TDA kernel is validated on CPU or AMD backends , so non-CUDA stacks should treat portability as unproven.
| Condition | Verdict vs. dense attention |
|---|---|
| Block sparsity < ~60% | Roughly breaks even; histogram overhead is the cost |
| Block sparsity > ~60% | Matches or beats FlashAttention-2; up to 2×+ in the high-sparsity limit |
| Long-context training (32K+) | Strongest case — sparsity grows with sequence length |
| Short-context / inference-only | Limited gain — little to skip, backward pass not in play |
| CPU / AMD backends | Out of scope — Triton + CUDA only, H100-tested |
The takeaway: adopt AdaSplash-2 when you are training long-context models on CUDA hardware and your attention matrices are genuinely sparse. In that regime the open BSD-3 kernel is a credible drop-in. Outside it — short context, inference-bound, or non-CUDA — dense FlashAttention-2 remains the safer default, and the missing 350M checkpoint gives you no shortcut around running the numbers on your own workload.
Frequently asked questions
What is α-entmax and how does it differ from softmax?
α-entmax is a sparse attention normalizer that, with α greater than 1, assigns exact zero probability to keys scoring below a learned per-row threshold — whereas softmax gives every key nonzero mass and so never produces a true zero . The distinction matters operationally: α-entmax yields a genuinely sparse probability distribution at training time, not a post-hoc pruning of a dense one. Because those zeros are exact, a Triton kernel can skip the resulting all-zero tiles losslessly rather than traversing every block, which is the mechanism behind AdaSplash-2's tile-skip path .
Is the AdaSplash-2 350M or 354M causal-LM checkpoint publicly available to download?
Not as confirmed by the primary sources located as of April 2026. SARDINE Lab's Hugging Face org hosts SparseModernBERT-alpha2.0, an MIT-licensed ~0.1B masked-LM that replaces softmax with AdaSplash via Triton, but the visible org page does not list a 350M or 354M causal-LM checkpoint . What is confirmed available is the kernel itself: the pip-installable adasplash>=0.2.0 package, released under the BSD-3-Clause license at deep-spin/adasplash . So the "354M open-weights" framing of the topic is not substantiated; treat the figure as unconfirmed.
How do I drop AdaSplash-2's tile-skip kernel into an existing training loop?
Install it with pip install 'adasplash>=0.2.0', which routes supported default attention calls to the AdaSplash-2 Triton path automatically . You need PyTorch, Triton, and a CUDA GPU to run the kernels — the package will not execute on CPU or non-CUDA backends . The standard path's defaults — an 8-bin histogram threshold initializer and a 64-token key block mask with 64-bit packing — are on by default and need no manual configuration; that default block geometry gives a stated capacity of 16,320 keys per query block before flushing .
Does AdaSplash-2 improve inference throughput or only training speed?
The gains are concentrated in training, and specifically in the backward pass, which the authors say drives much of the speedup . Forward-only inference at short context sees limited benefit because block sparsity at shorter sequences is materially lower than at the 32K+ context lengths where the method shines — sparsity is reported to increase with context length, with context-scaling experiments evaluated up to 128K . In the high-sparsity limit the paper cites more than a 2× speedup over FlashAttention-2, but at short context the histogram overhead has little block-skipping to amortize against.
How does AdaSplash-2 relate to TDA from Snap Research — are they the same release?
No. They are independent early-2026 softmax-free papers that both ship Triton tile-skip kernels, but the mechanisms and released scales differ. AdaSplash-2 uses α-entmax and trains 350M and 1B LLaMA-3-style models , while Snap Research's Threshold Differential Attention, accepted to ACL 2026, uses a different threshold differential mechanism and releases a roughly 162M GPT-2 variant at snap-research/TDA . The "354M" figure in the topic line matches no confirmed release from either project, which is why it should be read as the most likely point of conflation rather than a real checkpoint.
