--- title: "A Simple Look at TokPress: A Compressor That Uses an LLM's Tokenizer" author: "Lê Đức Minh" published: "2026-09-06T12:30:04.854Z" updated: "2026-09-01T07:46:42.409Z" url: "https://zyvop.com/a-simple-look-at-tokpress-a-compressor-that-uses-an-llm-s-tokenizer-60243" canonical: "https://zyvop.com/a-simple-look-at-tokpress-a-compressor-that-uses-an-llm-s-tokenizer-60243" tags: [] --- # A Simple Look at TokPress: A Compressor That Uses an LLM's Tokenizer ![Cover image]() If you've ever compressed a bunch of small JSON log lines, you've probably noticed something annoying: gzip often makes them _bigger_, not smaller. A 200-byte log line has so little content that the compressor spends more bytes on its own bookkeeping than it saves. TokPress is a tiny weekend project that tries a different trick: instead of compressing raw bytes, it first runs the text through the same tokenizer that OpenAI's models use (`o200k_base`), then compresses the _tokens_. That's the whole idea. An LLM tokenizer has already learned which pieces of text are common — words, punctuation, code patterns. So the compressor gets a much better alphabet to work with for free. ## What it actually does Three simple steps: 1. **Tokenize** — turn the input into token ids using `o200k_base`. 2. **Compress the tokens** — run a simple LZ77 pass over the token ids. 3. **Entropy-code** — finish with rANS, an efficient entropy coder. The result is a `.tokz` file that decompresses back to the exact original bytes — even binary data, since the tokenizer works on bytes, not just text. ```python import tokpress compressed = tokpress.compress(payload) original = tokpress.decompress(compressed) # byte-exact ``` ## The trick that makes it useful: a shared dictionary The real win comes when you have _many records that look the same_ — log lines, API responses, telemetry. TokPress can train a small dictionary on a sample of your records, then every future record compresses against it: ```bash tokpress train-dict mydict.tokdict samples.jsonl tokpress compress new_record.json --dict mydict.tokdict -o new_record.tokz ``` With a trained dictionary, structured-log records went from a ratio of **0.800 to 0.2565** — about 3× smaller. And that's on records the dictionary had never seen. There's an even simpler mode that needs no training at all: compress many records together as one stream, and the model learns as it goes: ```python packed = tokpress.compress_many(records) records = tokpress.decompress_many(packed) ``` On 150 schema-similar JSON records, per-record compression summed to ratio **1.19** — the data literally grew. As one stream, it hit **0.0875**. Same bytes, one header instead of 150. ## Where it stands (honestly) - **On prose** it beats gzip and ties/beats zstd on some files (a 152KB prose file: 0.298 vs zstd's 0.324). - **It still loses to zstd's trained dictionary** by about 1.3×. zstd has had years of polish on dictionary training; this is a weekend project. - **It's pure Python**, so it's slow to compress. Decompression is fast (thousands of records/sec); compression is not. - **Tokenization isn't magic.** The tokenizer just reshapes the data; the real savings come from the entropy coding and the trained dictionary. ## Try it ```bash git clone https://github.com/LakoreAI/tokpress cd tokpress && pip install -e . python -c "import tokpress; print(len(tokpress.compress(b'{\"a\": 1}')))" ``` It's ~2900 lines, has 93 tests, and comes with an honest benchmark script. If you have a folder of repetitive logs or JSON, that's exactly the case it was built for.