quicktok: 11× on tiktoken, author-reported, no README

A new open-source project called quicktok claims it can tokenize text token-for-token identically to OpenAI's tiktoken while running several times faster, and it gets there not by changing the algorithm, but by gutting the part of the pipeline most people overlook. The trick is the pretokenizer.

How quicktok's SIMD pretokenizer replaces the regex engine

quicktok is a C++20 BPE tokenizer with no external dependencies and Python bindings packaged via Maturin, designed to hand-optimize OpenAI's specific tokenization pipeline rather than re-implement byte-pair encoding generically . Its core design move is to replace the general-purpose regex engine that splits raw text into pretokens with handwritten SIMD scanners compiled by hand from the fixed cl100k and o200k split patterns . Because those regexes never change at runtime, the author treats them as constants and turns them into vectorized byte scanners instead of feeding them to a regex VM on every call.

The pretokenizer carries several supporting optimizations, all aimed at avoiding repeated work:

• 2-byte trie with a zero-lookup direct table for CJK: Chinese, Japanese, and Korean byte sequences resolve without a hash probe .
• Dense "memo" validity caches (~2 MB) for 17-bit token IDs, which let recurring sequences skip repeated BPE merge lookups .
• Single-pass "product machines" for ASCII paths, handling the common case in one scan .

What is not being changed is the merge logic itself. The encoding step still performs exact backtracking BPE (not an approximation), so the output token IDs are intended to match tiktoken exactly rather than land "close enough" . That distinction is the whole pitch: the speedup is claimed to come from a faster front end (pretokenization and cache reuse), while the algorithmically load-bearing back end stays identical. If that holds, you get the same numbers out, faster, without the correctness risk that usually accompanies a "faster tokenizer."

The library covers OpenAI's cl100k_base, o200k_base, and o200k_harmony, plus several open-model vocabularies: Llama-3, Qwen2.5/3, and the gated Llama-4 set . The byte-identical guarantee is strongest for the OpenAI families, where the split patterns are the fixed targets the SIMD scanners were written against. Whether that parity actually holds, and whether the speed survives outside the author's machine, is what the rest of this analysis tests.

The byte parity guarantee and its one caveat

quicktok's core promise is correctness, not approximation: its token IDs are stated to be byte-identical to tiktoken for OpenAI's cl100k_base and o200k_base family, with the encoding step still performing exact backtracking BPE rather than a faster heuristic . The claim is enforced programmatically: an import-time self-test refuses to load on a single token mismatch, so parity is checked at runtime rather than asserted in a README . That is a stronger posture than a trust-me benchmark; if the ports drift, the package fails closed instead of silently returning wrong counts.

The author surfaces one caveat himself. Some ported open-model (non-OpenAI) encodings show roughly 99.9998% agreement, with the residual disagreements landing on rare rank-vs-merge split cases . The byte-identical guarantee is therefore cleanest for the core OpenAI cl100k/o200k encodings (the fixed split patterns the SIMD scanners were written against) and weakest for the open-model vocabularies layered on top.

For most developer workflows the practical implication is narrow and favorable. Context-window checks and API cost estimation against OpenAI models depend on exact encoding output and on correct model-to-encoding resolution: gpt-5, gpt-4.1, gpt-4o, o1, o3, and o4-mini currently route to o200k_base, while GPT-4/GPT-3.5 map to cl100k_base . If quicktok matches there, your token math matches. But if any non-OpenAI vocabulary sits in your pipeline, verify parity separately before relying on the count.

One framing caution: a credible byte-identical claim covers more than .encode() on clean prose. It must also reproduce model-to-encoding resolution, special-token handling, decode semantics, and UTF-8 boundary behavior, plus error and batch-API behavior . The self-test is reassuring; it is not a substitute for testing those surfaces against the exact tiktoken version you run.

The corroboration gap: no README, no reproducible methodology

The published package does not carry the evidence the claims need. On PyPI, quicktok's latest release is 0.2.0, uploaded January 6, 2026, with prior 0.1.0/0.1.1 from February 2024, maintainer krzysztofwos, Python >=3.8, and CPython/PyPy/Rust classifiers, but no project description at all . The single repository link on that page resolves to openai/tiktoken, not a quicktok-specific tree . So the canonical install surface points readers at the reference implementation it claims to beat.

What the metadata does confirm is supply-chain hygiene, not parity. The 16.7 kB sdist carries a recorded SHA256 (0482ae1a…a648bd8), and the artifacts were published via Maturin 1.11.2 using Trusted Publishing . That is a clean, verifiable build pipeline. It says nothing about whether the token IDs match tiktoken or whether any MB/s ratio holds. Provenance and correctness are separate questions, and only the first is answered here.

Every performance and parity figure traces to two author-controlled sources: a GitHub README and a Hugging Face Forums announcement . No third-party reproduction, no public CI run, and no benchmark script inside the published sdist were found. The most-cited result (quicktok at 92.8 MB/s versus tiktoken's 12.6 MB/s on cl100k_base over The Pile) is qualified only as Apple M1, single-thread . The tiktoken version, Python version, and OS are not stated, so reproducing the number means filling those gaps yourself.

This is precisely the bar adjacent research clears. Peek2, for instance, publishes its algorithm, datasets, baselines, and stated limits rather than a single headline ratio . As the OpenAI engineering write-up on faster BPE tokenization puts it, a speed claim is only legible when pinned to versions and workload:

"tiktoken 0.2.0 is 3–6× faster than a comparable open source tokenizer," per GitHub Engineering, benchmarked with tokenizers==0.13.2 and transformers==4.24.0 over 1 GB of text (source: GitHub Blog).

That sentence names the version, the comparators, and the corpus size. quicktok's public record does not, which leaves its numbers author-reported and single-machine until someone independent re-runs them.

The reported MB/s figures and the conditions behind them

The headline number is a single-thread, Apple M1 run on cl100k_base over The Pile, where quicktok is reported at 92.8 MB/s against tiktoken's Python binding at 12.6 MB/s (roughly 7.4×) . The same run places the Rust comparators in between: bpe-openai at 29.8 MB/s, tiktoken-rs at 13.6 MB/s, and the separate TokenDagger project at 9.7 MB/s . So the 7× figure is measured against Python tiktoken specifically; against a native Rust tokenizer the gap narrows to about 3×.

The single number hides a wide spread. Across three 25 MB corpora (The Pile, code, and Common Crawl), quicktok is reported between 55.6 and 114.9 MB/s, summarized as 2–3.4× over bpe-openai and 3.5–11× over Python tiktoken depending on the corpus . Code and natural-language text do not tokenize at the same rate, so a builder should treat the high end as workload-specific, not a guarantee.

One condition matters more than the corpus: the build. Portable PyPI wheels are reported only 1.1–1.6× faster than bpe-openai ; the dramatic ratios depend on a native build compiled for the target CPU. A plain pip install without a source build yields a much smaller delta than the headline suggests, because the SIMD scanners need the host's instruction set to pay off.

The batch path is profiled separately and reads far higher: about 706 MB/s native on 8 cores, around 550 MB/s through Python encode_batch (framed as roughly 24× tiktoken's batch API), and encode_to_numpy() reaching near-native ~92 MB/s from Python . Those are different code paths with different overheads; single-call, batch, and numpy-output each deserve their own measurement before you size a pipeline around any one of them.

tiktoken 0.13.0 and the moving baseline

Any "× faster than tiktoken" ratio is only as precise as the tiktoken version it was measured against, and that baseline keeps moving. The current PyPI release, 0.13.0 (May 15, 2026), updated fancy-regex for "significantly increased performance" and changed a BPE behavior to fix slow handling of unusual inputs . So a benchmark run against an older build is comparing against a slower reference than the one most teams now pip install .

The trajectory is longer than one release. tiktoken's own changelog records v0.3.0 (+5–20%), v0.6.0 (a regex optimization adding roughly +20%), v0.7.0 (GPT-4o support), and v0.12.0 (Python 3.14 wheels with free-threaded support and GPT-5 recognition) . A library that has shipped several compounding speedups makes any undated "vs tiktoken" comparison ambiguous: quicktok's reported 3.5–11× span almost certainly straddles different reference versions, and without a pinned version the reader cannot tell which.

Correctness has a moving target too. A drop-in replacement has to reproduce tiktoken's model-to-encoding resolution exactly, not just its .encode() output. Today that mapping sends gpt-5, gpt-4.1, gpt-4o, o1, o3, and o4-mini to o200k_base; the older GPT-4 and GPT-3.5 families to cl100k_base; and gpt-oss-* models to o200k_harmony .

That resolution layer matters because different encodings split the same string differently, which changes both token counts and downstream API cost . A library that gets .encode() byte-identical but resolves a model name to the wrong encoding will silently miscount context-window fit and mis-estimate cost, a failure that never raises an exception. When you validate parity, pin the tiktoken version and test the full encoding_for_model() path, not only raw encoding output.

Reproducing the stated gain on your own hardware

Treat every quicktok figure as a hypothesis to re-test, not a result to inherit. The reported numbers are single-machine and author-reported, so the only ratio that matters for your decision is the one you measure against the exact tiktoken wheel you ship, on your corpus, on your CPU. Start by pinning the baseline: tiktoken's v0.13.0 release (May 15, 2026 ) updated fancy-regex for a significant performance increase and changed BPE behavior to fix unusual-input handling , so any speedup ratio derived against an older wheel may not hold against the current one.

The need to state your exact comparison conditions is not new. tiktoken's own README historically claimed version 0.2.0 ran 3–6× faster than a comparable GPT-2 tokenizer, but only with the caveat that it used tokenizers==0.13.2 and transformers==4.24.0 on 1 GB of text :

"tiktoken is 3-6x faster than a comparable open source tokeniser": the published number was always bound to specific pinned versions and a 1 GB workload (source: GitHub Engineering).

Then build your test around how you actually feed text:

• Use your corpus, not The Pile. The reported 3.5–11× range came from large English-heavy corpora ; code-heavy input and short sequences under 2,000 tokens produce materially different ratios.
• Validate token-ID identity on representative samples: UTF-8 boundary cases, special tokens (BOS/EOS/pad), and max-context sequences. The library's strict import self-test validates its internal corpus, not yours.
• Profile encode() and encode_batch() separately. The ~24× batch figure is a distinct code path from the single-thread ratio ; conflating them is a common source of inflated apparent comparisons.

If your production path resolves model names, exercise the full encoding_for_model mapping, not raw encoding alone; a silent miscount there never raises an exception.

Competing approaches: GPUTOK, Peek2, and Incremental BPE

Three peer-reviewed tokenizer projects from 2026 show what a credible speed claim looks like, and the contrast with quicktok is instructive: each names its dataset, states its hardware, pins a versioned baseline, and declares the regime where it loses. GPUTOK (submitted March 3, 2026) reports about 1.67× over tiktoken on 131K-token WikiText103 sequences (53.4 ms versus 598.1 ms), but is explicitly slower than tiktoken below roughly 2,000 tokens . It tells you exactly when not to use it.

Peek2 (submitted January 9, 2026; ACL SRW 2026) takes a different target: it replaces the cl100k-style regex pretokenizer with a linear-time algorithm, reaching 2.48× on microbenchmarks and 1.14× end-to-end while matching the regex baseline's output on all tested inputs . That is the same pretokenization bottleneck quicktok's SIMD scanners attack, but Peek2 publishes the parity check and the end-to-end number, which is usually far lower than the microbenchmark.

Incremental BPE Tokenization (submitted May 29, 2026; ICML 2026 Spotlight) bounds the worst case at O(log²t) per byte and targets pathological long inputs, reporting latency reductions over tiktoken with explicit sequence-length conditions rather than a single headline ratio .

The pattern is the scaffolding: named corpus, declared hardware, versioned comparison, and a published failure regime. Quicktok's reported figures may well hold (its approach is plausible and its parity guard is strict), but its public record currently lacks that scaffolding, so the burden sits with you. The concrete takeaway: treat quicktok as a promising candidate, not a verified result. Pin the exact tiktoken version, run its strict import check against your encodings, and benchmark single-thread and batch paths separately on your own corpus before any speedup reaches production.

Frequently asked questions

Is quicktok safe to swap into a production pipeline today?

Not without your own verification. quicktok ships a strict import-time self-test that refuses to load on a single token mismatch, which gives a programmatic parity guarantee for the OpenAI cl100k_base and o200k_base families . But no independent party has reproduced either the parity or the speed claims, and the PyPI package (latest 0.2.0, Jan. 6, 2026) carries no README or benchmark report . Run a token-ID diff against your exact tiktoken version on a representative sample before moving any cost-sensitive or context-window-sensitive workload.

What does 'byte-identical' mean for a BPE tokenizer?

It means the output token IDs are the same integer sequence as the reference implementation for the same input bytes, matching exactly rather than approximately. quicktok's encoding step still performs exact backtracking BPE rather than an approximation, so IDs should equal tiktoken's . This precision matters because token counts determine whether input fits a model's context window and how much an API call costs; any ID difference changes those calculations . The author notes one caveat: some ported non-OpenAI encodings show only ~99.9998% agreement on rare split cases, so the clean guarantee is strongest for cl100k/o200k .

Why does the quicktok PyPI page link to openai/tiktoken instead of its own repo?

This is most likely a metadata misconfiguration set at upload time. The PyPI listing for quicktok 0.2.0 has no project description, and its only visible repository link resolves to openai/tiktoken rather than a quicktok-specific tree . The wheels and the 16.7 kB sdist were uploaded via maturin/1.11.2 using Trusted Publishing, which verifies the upload source but says nothing about semantic equivalence or speed . The practical effect: a PyPI visitor has no direct path from the package page to the project's own code or documentation.

How much quicker is a native build versus the portable PyPI wheel?

The headline gains depend on compiling from source. Author-reported figures put portable PyPI wheels at only ~1.1–1.6× over bpe-openai (Rust), while the 3.5–11× over tiktoken figures require a native build tuned for the target CPU architecture . A plain pip install without a source build therefore produces a much smaller delta than the announcement suggests. If you cannot build for your hardware, benchmark the portable wheel specifically rather than assuming the native numbers transfer.

How does quicktok differ from tiktoken-rs and bpe-openai?

The three projects make different generality trade-offs. tiktoken-rs is a Rust wrapper around tiktoken's underlying logic, and bpe-openai is GitHub's 2024/2025 Rust tokenizer . quicktok instead hand-compiles SIMD scanners for the specific fixed cl100k/o200k regex patterns from scratch, plus a 2-byte trie and dense validity caches, trading generality for throughput on those particular vocabularies . In the author's single-thread Apple M1 run on cl100k_base over The Pile, quicktok reported 92.8 MB/s versus 29.8 for bpe-openai, 13.6 for tiktoken-rs, and 12.6 for tiktoken (Python) , figures that remain author-reported and single-machine.