{"schemaVersion":"1.0","type":"Article","types":["Article"],"slug":"a-simple-look-at-tokpress-a-compressor-that-uses-an-llm-s-tokenizer-60243","url":"https://api.zyvop.com/a-simple-look-at-tokpress-a-compressor-that-uses-an-llm-s-tokenizer-60243","title":"A Simple Look at TokPress: A Compressor That Uses an LLM's Tokenizer","subtitle":null,"tldr":"TokPress compresses tiny JSON log lines by tokenizing with OpenAI's o200k_base tokenizer, then applying LZ77 and rANS for smaller files.","keywords":["Case Studies"],"entities":["Lê Đức Minh","AI Engineer","Case Studies","ZyVOP"],"keyTakeaways":["If you've ever compressed a bunch of small JSON log lines, you've probably noticed something annoying: gzip often makes them bigger, not smaller.","A 200-byte log line has so little content that the compressor spends more bytes on its own bookkeeping than it saves.","TokPress is a tiny weekend project that tries a different trick: instead of compressing raw bytes, it first runs the text through the same tokenizer that OpenAI's models use (o200k_base), then compresses the tokens."],"headings":["What it actually does","The trick that makes it useful: a shared dictionary","Where it stands (honestly)","Try it"],"outboundLinks":[],"contentText":"If you've ever compressed a bunch of small JSON log lines, you've probably noticed something annoying: gzip often makes them bigger, not smaller. A 200-byte log line has so little content that the compressor spends more bytes on its own bookkeeping than it saves. TokPress is a tiny weekend project that tries a different trick: instead of compressing raw bytes, it first runs the text through the same tokenizer that OpenAI's models use (o200k_base), then compresses the tokens. That's the whole idea. An LLM tokenizer has already learned which pieces of text are common — words, punctuation, code patterns. So the compressor gets a much better alphabet to work with for free. What it actually does Three simple steps: Tokenize — turn the input into token ids using o200k_base. Compress the tokens — run a simple LZ77 pass over the token ids. Entropy-code — finish with rANS, an efficient entropy coder. The result is a .tokz file that decompresses back to the exact original bytes — even binary data, since the tokenizer works on bytes, not just text. import tokpress compressed = tokpress.compress(payload) original = tokpress.decompress(compressed) # byte-exact The trick that makes it useful: a shared dictionary The real win comes when you have many records that look the same — log lines, API responses, telemetry. TokPress can train a small dictionary on a sample of your records, then every future record compresses against it: tokpress train-dict mydict.tokdict samples.jsonl tokpress compress new_record.json --dict mydict.tokdict -o new_record.tokz With a trained dictionary, structured-log records went from a ratio of 0.800 to 0.2565 — about 3× smaller. And that's on records the dictionary had never seen. There's an even simpler mode that needs no training at all: compress many records together as one stream, and the model learns as it goes: packed = tokpress.compress_many(records) records = tokpress.decompress_many(packed) On 150 schema-similar JSON records, per-record compression summed to ratio 1.19 — the data literally grew. As one stream, it hit 0.0875. Same bytes, one header instead of 150. Where it stands (honestly) On prose it beats gzip and ties/beats zstd on some files (a 152KB prose file: 0.298 vs zstd's 0.324). It still loses to zstd's trained dictionary by about 1.3×. zstd has had years of polish on dictionary training; this is a weekend project. It's pure Python, so it's slow to compress. Decompression is fast (thousands of records/sec); compression is not. Tokenization isn't magic. The tokenizer just reshapes the data; the real savings come from the entropy coding and the trained dictionary. Try it git clone https://github.com/LakoreAI/tokpress cd tokpress && pip install -e . python -c \"import tokpress; print(len(tokpress.compress(b'{\\\"a\\\": 1}')))\" It's ~2900 lines, has 93 tests, and comes with an honest benchmark script. If you have a folder of repetitive logs or JSON, that's exactly the case it was built for.","contentHash":"sha256:9938372fb02e8607673206189186dd4859b6fa09eb94b0f86d86a6d29e4cc84f","authorName":"Lê Đức Minh","authorUrl":"https://api.zyvop.com/author/l445","authorSameAs":["https://minlee0210.github.io","https://github.com/MinLee0210"],"category":"Case Studies","tags":[],"audience":"Developers, software engineers, and students learning Case Studies","tone":"Professional, ai engineer perspective","readingTimeMinutes":2,"wordCount":479,"faqs":null,"primaryTopic":"Case Studies","publishedAt":"2026-09-06T12:30:04.854Z","updatedAt":"2026-09-01T07:46:42.409Z","canonicalUrl":"https://api.zyvop.com/a-simple-look-at-tokpress-a-compressor-that-uses-an-llm-s-tokenizer-60243"}