{"schemaVersion":"1.0","type":"Article","types":["Article"],"slug":"the-real-vietnamese-llm-tokenizer-cost-it-s-not-4-5x-krxzs","url":"https://api.zyvop.com/the-real-vietnamese-llm-tokenizer-cost-it-s-not-4-5x-krxzs","title":"The Real Vietnamese LLM Tokenizer Cost (It's Not 4.5x)","subtitle":null,"tldr":"Discover why Vietnamese tokenizers cost only 1.05‑2.14× English tokens, not 4.5×, and how updating tokenizers can cut LLM expenses by a third.","keywords":[],"entities":["Lê Đức Minh","AI Engineer","ZyVOP"],"keyTakeaways":["The 4.54x statistic applies only to obsolete 2019 tokenizers.","Modern production APIs tokenize Vietnamese between 1.1x and 2.1x relative to English.","Upgrading from cl100k_base to o200k_base saves ~37% in token overhead for Vietnamese text.","Specialized Vietnamese models (PhoBERT) achieve 0.87x parity, proving vocabulary design is what matters."],"headings":["Where the \"4.54x Token Tax\" Number Came From","The Real Numbers Across 13 Tokenizers","The ~37% Free Efficiency Win","Frontier Does Not Mean Token-Efficient","Benchmark Script","Experiment","Summary","References","Appendix: Full Script"],"outboundLinks":["https://medium.com/@minhle_0210/why-your-coding-agents-bill-grows-faster-than-the-chat-2d9ecb37423d","https://ai.plainenglish.io/i-spent-my-weekend-benchmarking-vietnamese-bert-models-so-you-dont-have-to-0d19aa280736","https://ai.plainenglish.io/weekend-nlp-fine-tuning-transformers-for-language-detection-with-huggingface-4041ac9ca20e","https://ssrn.com/abstract=7336938","https://openai.com/research/better-language-models","https://huggingface.co/vinai/phobert-base","https://github.com/openai/tiktoken","https://huggingface.co/Qwen/Qwen3-8B","https://www.linkedin.com/in/minhle007/","https://github.com/MinLee0210"],"contentText":"The moment real traffic hits your LLM endpoint, every token turns into a line item on your cloud bill — and that line item doesn't stop growing once the prompt itself stops changing, as I found out the hard way chasing down why a coding agent's bill kept climbing faster than the conversation did. A prompt that looks concise in text can balloon once the tokenizer processes it, especially for non-English languages — something I first ran into while benchmarking Vietnamese BERT models. When building a social listening pipeline that processed thousands of Vietnamese comments per hour, my dashboard costs quickly exceeded initial projections. Searching online, I found the number everyone repeats: Vietnamese requires up to 4.54x more tokens than English. It sounds scary. It makes for great social media talking points. It is also measuring tokenizers from 2019 that nobody uses in production today. I benchmarked 13 tokenizers on matched English and Vietnamese text. Modern tokenizers hover between 1.05x and 2.14x (with PhoBERT achieving parity at 0.87x). Upgrading tokenizer generations alone can reduce your token bill by over a third without changing your models. Where the \"4.54x Token Tax\" Number Came From The 4.54x statistic traces back to blog posts evaluating GPT-2 and RoBERTa-style tokenizers. These 2019 byte-pair encoders had English-only vocabularies with zero Vietnamese diacritics. As a result, common words like \"được\", \"nghĩa\", and \"những\" were split into multiple sub-byte pieces. That wasn't a property of the Vietnamese language. It was a limitation of five-year-old English tokenizers. The Real Numbers Across 13 Tokenizers I ran a matched paragraph of natural English and Vietnamese through 13 tokenizers across three generations: Tokenizer Category Ratio (VI Tokens / EN Tokens) gpt2 Legacy English-only 4.03x roberta-base Legacy English-only 3.98x phobert-base* Vietnamese-specific 0.87x gpt2-vietnamese Vietnamese-specific 1.05x xlm-roberta-base Modern Multilingual 1.14x mmBERT-base Modern Multilingual 1.24x qwen2.5-7b Modern Multilingual 1.26x qwen3-8b Modern Multilingual 1.26x o200k_base (GPT-4o / GPT-4.1) Modern Multilingual 1.34x deepseek-v3.2 Frontier Reasoning 1.88x cl100k_base (GPT-3.5 / GPT-4) Modern Multilingual 2.14x llama-3-8b Modern Multilingual Gated (no token) gemma-2-9b-it Modern Multilingual Gated (no token) * PhoBERT expects word-segmented input; raw text was fed here, making 0.87x a conservative baseline. xychart-beta horizontal title \"Vietnamese / English token ratio by tokenizer\" x-axis [\"gpt2\", \"roberta\", \"cl100k (GPT-4)\", \"deepseek-v3.2\", \"o200k (GPT-4o)\", \"qwen3-8b\", \"qwen2.5\", \"mmBERT\", \"xlm-roberta\", \"gpt2-vn\", \"phobert\"] y-axis \"Ratio (VI / EN)\" 0 --> 4.5 bar [4.03, 3.98, 2.14, 1.88, 1.34, 1.26, 1.26, 1.24, 1.14, 1.05, 0.87]The ~37% Free Efficiency Win The most actionable finding is the gap between OpenAI generations: cl100k_base (GPT-3.5 and GPT-4): 2.14x token ratio. o200k_base (GPT-4o and GPT-4.1): 1.34x token ratio. Switching from a GPT-4 endpoint to a GPT-4o endpoint automatically cuts your Vietnamese token inflation by roughly 37%. You don't need to rewrite prompts or change architecture; the tokenizer vocabulary expansion simply covers Vietnamese syllables more efficiently. Frontier Does Not Mean Token-Efficient Another surprise: DeepSeek-V3.2, a frontier reasoning model, tokenizes Vietnamese at 1.88x. By contrast, mmBERT (a smaller multilingual encoder) sits at 1.24x, and Qwen3 lands at 1.26x. Model intelligence and tokenizer efficiency are completely independent dimensions. Meanwhile, Vietnamese-specific tokenizers like PhoBERT and gpt2-vietnamese match or beat English token counts. The \"Vietnamese tax\" is fundamentally a choice of tokenizer, not an inherent cost of the language — the same vocabulary-design lesson I ran into fine-tuning transformers for language detection. These numbers align closely with an independent production study published by engineers at Techcombank in Hanoi (The Tokenization Tax), which confirmed that vocabulary coverage, rather than model vendor, dictates production token consumption. Benchmark Script try: if spec[\"loader\"] == \"hf\": from transformers import AutoTokenizer, PreTrainedTokenizerFast try: tok = AutoTokenizer.from_pretrained(spec[\"id\"]) except Exception: tok = PreTrainedTokenizerFast.from_pretrained(spec[\"id\"]) en_ids = tok.encode(EN_TEXT) vi_ids = tok.encode(VI_TEXT) else: import tiktoken enc = tiktoken.get_encoding(spec[\"id\"]) en_ids = enc.encode(EN_TEXT) vi_ids = enc.encode(VI_TEXT) entry[\"ratio_vi_over_en\"] = round(len(vi_ids) / len(en_ids), 3) except Exception as e: entry[\"status\"] = \"skipped\" entry[\"error\"] = str(e)Experiment Here's the actual run, step by step. The complete script is in the appendix at the end of this post. 1. Write matched English/Vietnamese paragraphs for this benchmark specifically — not lifted from an external source, so token counts reflect genuinely parallel content rather than sentences that just happen to differ in length: EN_TEXT = ( \"Running a language model in production is nothing like running it in a \" \"notebook. The moment real traffic hits your endpoint, every token you \" \"send and receive turns into a line item on your bill. ...\" ) VI_TEXT = ( \"Chạy một mô hình ngôn ngữ trong môi trường production hoàn toàn khác với \" \"chạy trong notebook. Ngay khi traffic thật đổ vào endpoint, mỗi token \" \"bạn gửi đi và nhận về đều trở thành một dòng trong hóa đơn. ...\" )2. List every tokenizer to benchmark, tagged by generation and loader. Here are 3 of the 13: TOKENIZER_SPECS = [ {\"name\": \"gpt2\", \"family\": \"legacy-english-only\", \"loader\": \"hf\", \"id\": \"gpt2\"}, {\"name\": \"phobert-base\", \"family\": \"vietnamese-specific\", \"loader\": \"hf\", \"id\": \"vinai/phobert-base\"}, {\"name\": \"o200k_base (GPT-4o / GPT-4.1)\", \"family\": \"modern-multilingual\", \"loader\": \"tiktoken\", \"id\": \"o200k_base\"}, ]3. Print each result as it's computed, and record failures honestly instead of estimating them: status_word = entry[\"status\"] detail = ( f\"ratio={entry.get('ratio_vi_over_en')}\" if status_word == \"ok\" else f\"error={entry.get('error')}\" ) print(f\"[{status_word:7s}] {spec['name']:32s} {detail}\", flush=True)That's why llama-3-8b and gemma-2-9b-it show up as \"Gated (no token)\" in the table above instead of a made-up ratio. 4. Write the full payload — both texts, their word/char counts, and every tokenizer's result — to disk for the post: payload = { \"en_text\": EN_TEXT, \"vi_text\": VI_TEXT, \"en_word_count\": len(EN_TEXT.split()), \"vi_word_count\": len(VI_TEXT.split()), \"en_char_count\": len(EN_TEXT), \"vi_char_count\": len(VI_TEXT), \"results\": results, }Run it yourself: uv run python tokenizer_cost_benchmark.py. Summary The 4.54x statistic applies only to obsolete 2019 tokenizers. Modern production APIs tokenize Vietnamese between 1.1x and 2.1x relative to English. Upgrading from cl100k_base to o200k_base saves ~37% in token overhead for Vietnamese text. Specialized Vietnamese models (PhoBERT) achieve 0.87x parity, proving vocabulary design is what matters. References Better Language Models and Their Implications (GPT-2) — OpenAI's original GPT-2 release, the tokenizer this post traces the \"4.54x\" myth back to. PhoBERT — VinAI's Vietnamese-specific tokenizer/model, the 0.87x baseline in this post. tiktoken — OpenAI's BPE library implementing cl100k_base and o200k_base, the two tokenizers compared in the \"37% free win\" section. Qwen3-8B — one of the modern multilingual tokenizers benchmarked. \"The Tokenization Tax\" — the independent production study (Techcombank) this post cites as corroboration. What tokenizer ratios have you observed in your production workloads? Let's discuss in the comments. 👉 Follow my work: LinkedIn | GitHub Appendix: Full Script For anyone who wants the complete, runnable file: #!/usr/bin/env python3 \"\"\"Measure real token-count inflation for Vietnamese vs English across tokenizer generations (legacy English-only BPE, Vietnamese-specific, modern multilingual). No fabricated numbers: every tokenizer that fails to load (gated repo, network error, missing files) is recorded as skipped with the real error, not estimated. \"\"\" import json import sys import time from pathlib import Path OUT_DIR = Path(\"content/2026-09-01/tokenizer-cost/scratch\") # Matched-content EN/VI pair, first-person voice, written for this benchmark # (not lifted from an external source) so token counts reflect genuinely # parallel content rather than independently-varying sentence structure. EN_TEXT = ( \"Running a language model in production is nothing like running it in a \" \"notebook. The moment real traffic hits your endpoint, every token you \" \"send and receive turns into a line item on your bill. A prompt that \" \"looked short on screen can balloon once the tokenizer gets its hands on \" \"it, especially if your users aren't writing in English. I learned this \" \"the hard way while building a social listening pipeline that ingested \" \"thousands of Vietnamese comments an hour. The dashboard cost numbers \" \"didn't match my mental model at all, and it took a real benchmark, not \" \"a blog post, to figure out why.\" ) VI_TEXT = ( \"Chạy một mô hình ngôn ngữ trong môi trường production hoàn toàn khác với \" \"chạy trong notebook. Ngay khi traffic thật đổ vào endpoint, mỗi token \" \"bạn gửi đi và nhận về đều trở thành một dòng trong hóa đơn. Một prompt \" \"nhìn có vẻ ngắn trên màn hình có thể phình to đáng kể sau khi qua tay \" \"tokenizer, nhất là khi người dùng của bạn không viết bằng tiếng Anh. \" \"Mình nhận ra điều này một cách khá đau đớn khi xây dựng pipeline social \" \"listening xử lý hàng nghìn bình luận tiếng Việt mỗi giờ. Con số chi phí \" \"trên dashboard không khớp với mô hình mình nghĩ trong đầu chút nào, và \" \"phải chạy benchmark thật sự, chứ không phải đọc blog, mới tìm ra lý do.\" ) TOKENIZER_SPECS = [ {\"name\": \"gpt2\", \"family\": \"legacy-english-only\", \"loader\": \"hf\", \"id\": \"gpt2\"}, {\"name\": \"roberta-base\", \"family\": \"legacy-english-only\", \"loader\": \"hf\", \"id\": \"FacebookAI/roberta-base\"}, { \"name\": \"phobert-base\", \"family\": \"vietnamese-specific\", \"loader\": \"hf\", \"id\": \"vinai/phobert-base\", \"note\": \"PhoBERT's BPE expects word-segmented Vietnamese input (e.g. via VnCoreNLP); \" \"raw unsegmented text was used here, so this number is a lower bound on \" \"PhoBERT's real efficiency, not its best case.\", }, {\"name\": \"gpt2-vietnamese\", \"family\": \"vietnamese-specific\", \"loader\": \"hf\", \"id\": \"NlpHUST/gpt2-vietnamese\"}, {\"name\": \"xlm-roberta-base\", \"family\": \"modern-multilingual\", \"loader\": \"hf\", \"id\": \"FacebookAI/xlm-roberta-base\"}, { \"name\": \"mmBERT-base\", \"family\": \"modern-multilingual\", \"loader\": \"hf\", \"id\": \"jhu-clsp/mmBERT-base\", \"note\": \"ModernBERT-architecture encoder trained on 3T tokens across 1,833 languages \" \"(JHU CLSP, ICML 2026) — first model to meaningfully beat XLM-R on multilingual \" \"benchmarks; encoder, not a causal LM, but the tokenizer comparison is still valid.\", }, {\"name\": \"qwen2.5-7b\", \"family\": \"modern-multilingual\", \"loader\": \"hf\", \"id\": \"Qwen/Qwen2.5-7B\"}, {\"name\": \"qwen3-8b\", \"family\": \"modern-multilingual\", \"loader\": \"hf\", \"id\": \"Qwen/Qwen3-8B\"}, {\"name\": \"deepseek-v3.2\", \"family\": \"modern-multilingual\", \"loader\": \"hf\", \"id\": \"deepseek-ai/DeepSeek-V3.2\"}, {\"name\": \"llama-3-8b\", \"family\": \"modern-multilingual\", \"loader\": \"hf\", \"id\": \"meta-llama/Meta-Llama-3-8B\"}, {\"name\": \"gemma-2-9b-it\", \"family\": \"modern-multilingual\", \"loader\": \"hf\", \"id\": \"google/gemma-2-9b-it\"}, {\"name\": \"cl100k_base (GPT-3.5 / GPT-4)\", \"family\": \"modern-multilingual\", \"loader\": \"tiktoken\", \"id\": \"cl100k_base\"}, {\"name\": \"o200k_base (GPT-4o / GPT-4.1)\", \"family\": \"modern-multilingual\", \"loader\": \"tiktoken\", \"id\": \"o200k_base\"}, ] def run(): results = [] for spec in TOKENIZER_SPECS: entry = {\"name\": spec[\"name\"], \"family\": spec[\"family\"], \"id\": spec[\"id\"]} t0 = time.time() try: if spec[\"loader\"] == \"hf\": from transformers import AutoTokenizer, PreTrainedTokenizerFast try: tok = AutoTokenizer.from_pretrained(spec[\"id\"]) except Exception: # Some very new model configs (e.g. deepseek_v32) aren't fully # recognized by this transformers version's AutoConfig yet. # The tokenizer files themselves are still standard, so load # them directly, bypassing model-aware AutoConfig entirely. tok = PreTrainedTokenizerFast.from_pretrained(spec[\"id\"]) entry[\"loaded_via\"] = \"PreTrainedTokenizerFast fallback (AutoTokenizer failed on model config)\" en_ids = tok.encode(EN_TEXT) vi_ids = tok.encode(VI_TEXT) else: import tiktoken enc = tiktoken.get_encoding(spec[\"id\"]) en_ids = enc.encode(EN_TEXT) vi_ids = enc.encode(VI_TEXT) entry[\"en_tokens\"] = len(en_ids) entry[\"vi_tokens\"] = len(vi_ids) entry[\"ratio_vi_over_en\"] = round(len(vi_ids) / len(en_ids), 3) entry[\"status\"] = \"ok\" if \"note\" in spec: entry[\"note\"] = spec[\"note\"] except Exception as e: # noqa: BLE001 - deliberately broad, this is a survey script entry[\"status\"] = \"skipped\" entry[\"error\"] = str(e).splitlines()[0][:300] entry[\"load_and_encode_seconds\"] = round(time.time() - t0, 2) status_word = entry[\"status\"] detail = ( f\"ratio={entry.get('ratio_vi_over_en')}\" if status_word == \"ok\" else f\"error={entry.get('error')}\" ) print(f\"[{status_word:7s}] {spec['name']:32s} {detail}\", flush=True) results.append(entry) payload = { \"en_text\": EN_TEXT, \"vi_text\": VI_TEXT, \"en_word_count\": len(EN_TEXT.split()), \"vi_word_count\": len(VI_TEXT.split()), \"en_char_count\": len(EN_TEXT), \"vi_char_count\": len(VI_TEXT), \"results\": results, } OUT_DIR.mkdir(parents=True, exist_ok=True) out_path = OUT_DIR / \"benchmark_results.json\" out_path.write_text(json.dumps(payload, ensure_ascii=False, indent=2)) print(f\"\\nWrote {out_path}\", flush=True) return payload if __name__ == \"__main__\": run()","contentHash":"sha256:39484af62d993f8743b47679a3f36a7e103a2ee5b0a64d33235c096d702fffaf","authorName":"Lê Đức Minh","authorUrl":"https://api.zyvop.com/author/l445","authorSameAs":["https://minlee0210.github.io","https://github.com/MinLee0210"],"category":null,"tags":[],"audience":"Software engineers and developers building applications with software development","tone":"Professional, ai engineer perspective","readingTimeMinutes":8,"wordCount":1793,"faqs":null,"primaryTopic":null,"publishedAt":"2026-09-08T12:30:00.208Z","updatedAt":"2026-09-05T04:39:11.930Z","canonicalUrl":"https://api.zyvop.com/the-real-vietnamese-llm-tokenizer-cost-it-s-not-4-5x-krxzs"}