A Rust Tokenizer Claims 1,268x HuggingFace. Independent Testing Found 26x.

GigaToken, a new open source Rust tokenizer, claims to be dramatically faster than HuggingFace's Python tokenizer. The author's own benchmarks put it at up to 1,268x faster on Apple silicon. An independent verification site reproduced a real speedup, just a much smaller one: about 26x.

The author's numbers

Marcel Roed, the developer behind GigaToken, published throughput comparisons across three machines. On an AMD EPYC 9565 server chip, GigaToken hit 24.53 GB/s against HuggingFace's 24.8 MB/s, a 989x gap. On an Apple M4 Max, it reached 8.79 GB/s against 6.9 MB/s, the 1,268x figure that circulated on Hacker News. On a Ryzen 9800X3D, the gap narrowed to 106x: 6.27 GB/s against 59 MB/s. All three numbers come from the same author, on hardware and workloads he chose himself. The speedup comes from writing the tokenizer in Rust with SIMD-accelerated pretokenization, caching pretokens aggressively, and minimizing the round-trips into Python that a pure HuggingFace pipeline pays for on every call.

What independent testing found

KrabArena, a site built specifically to independently verify benchmark claims, ran GigaToken on a 174MB slice of OpenWebText on a generic 4-vCPU cloud VM, hardware picked to be reproducible rather than favorable. GigaToken processed the slice at 277.77 MB/s, against tiktoken's 10.62 MB/s and HuggingFace's 3.33 MB/s. That works out to roughly 26x over tiktoken, not 1,268x over HuggingFace. Both numbers are real. They were produced on different hardware, different datasets, and against different baseline tokenizers, which accounts for most of the gap between them.

Whether tokenization speed even matters

The Hacker News thread on GigaToken, at 1.2k stars and 44 forks and still climbing, argued past the benchmark numbers into whether the whole premise matters. Tokenization is typically under 0.1% of total inference time, so by Amdahl's Law, making it 1,000x faster barely moves end-to-end latency. Roed's counter, backed by his own time-to-first-token measurements, is that the real-world gain lands at 5-10%, smaller than the headline multiplier but not nothing for latency-sensitive request routing or for tokenizing large pretraining datasets on a deadline.

Why a build studio cares

A benchmark number on a GitHub README is a claim, not a spec. Before we wire a dependency's claimed speedup into a client build, we reproduce it on our own hardware against our own workload, because the honest answer is almost always smaller than the biggest number in the README. The gap between 1,268x and 26x is exactly the kind of thing that should change how much weight a claim gets before it ships, and it's a cheap check that costs a lot less than finding out in production that a dependency's real-world number was 26x, not 1,268x. It's also a reminder to ask the second question the Hacker News thread raised: even a confirmed speedup only matters if the piece it speeds up was actually the bottleneck in your pipeline, not just the easiest one to benchmark in isolation.

Next step: see GigaToken on GitHub and KrabArena's independent reproduction. If you're evaluating a claimed speedup for a real pipeline before you commit to it, write to hello@gattyworks.com.