The Real Vietnamese LLM Tokenizer Cost (It's Not 4.5x)

The moment real traffic hits your LLM endpoint, every token turns into a line item on your cloud bill — and that line item doesn't stop growing once the prompt itself stops changing, as I found out the hard way chasing down why a coding agent's bill kept climbing faster than the conversation did.

A prompt that looks concise in text can balloon once the tokenizer processes it, especially for non-English languages — something I first ran into while benchmarking Vietnamese BERT models. When building a social listening pipeline that processed thousands of Vietnamese comments per hour, my dashboard costs quickly exceeded initial projections.

Searching online, I found the number everyone repeats: Vietnamese requires up to 4.54x more tokens than English.

It sounds scary. It makes for great social media talking points.

It is also measuring tokenizers from 2019 that nobody uses in production today.

I benchmarked 13 tokenizers on matched English and Vietnamese text. Modern tokenizers hover between 1.05x and 2.14x (with PhoBERT achieving parity at 0.87x). Upgrading tokenizer generations alone can reduce your token bill by over a third without changing your models.

Where the "4.54x Token Tax" Number Came From

The 4.54x statistic traces back to blog posts evaluating GPT-2 and RoBERTa-style tokenizers.

These 2019 byte-pair encoders had English-only vocabularies with zero Vietnamese diacritics. As a result, common words like "được", "nghĩa", and "những" were split into multiple sub-byte pieces.

That wasn't a property of the Vietnamese language. It was a limitation of five-year-old English tokenizers.

The Real Numbers Across 13 Tokenizers

I ran a matched paragraph of natural English and Vietnamese through 13 tokenizers across three generations:

* PhoBERT expects word-segmented input; raw text was fed here, making 0.87x a conservative baseline.

xychart-beta horizontal
    title "Vietnamese / English token ratio by tokenizer"
    x-axis ["gpt2", "roberta", "cl100k (GPT-4)", "deepseek-v3.2", "o200k (GPT-4o)", "qwen3-8b", "qwen2.5", "mmBERT", "xlm-roberta", "gpt2-vn", "phobert"]
    y-axis "Ratio (VI / EN)" 0 --> 4.5
    bar [4.03, 3.98, 2.14, 1.88, 1.34, 1.26, 1.26, 1.24, 1.14, 1.05, 0.87]

The ~37% Free Efficiency Win

The most actionable finding is the gap between OpenAI generations:

• cl100k_base (GPT-3.5 and GPT-4): 2.14x token ratio.

• o200k_base (GPT-4o and GPT-4.1): 1.34x token ratio.

Switching from a GPT-4 endpoint to a GPT-4o endpoint automatically cuts your Vietnamese token inflation by roughly 37%. You don't need to rewrite prompts or change architecture; the tokenizer vocabulary expansion simply covers Vietnamese syllables more efficiently.

Frontier Does Not Mean Token-Efficient

Another surprise: DeepSeek-V3.2, a frontier reasoning model, tokenizes Vietnamese at 1.88x.

By contrast, mmBERT (a smaller multilingual encoder) sits at 1.24x, and Qwen3 lands at 1.26x. Model intelligence and tokenizer efficiency are completely independent dimensions.

Meanwhile, Vietnamese-specific tokenizers like PhoBERT and gpt2-vietnamese match or beat English token counts. The "Vietnamese tax" is fundamentally a choice of tokenizer, not an inherent cost of the language — the same vocabulary-design lesson I ran into fine-tuning transformers for language detection.

These numbers align closely with an independent production study published by engineers at Techcombank in Hanoi (The Tokenization Tax), which confirmed that vocabulary coverage, rather than model vendor, dictates production token consumption.

Benchmark Script

try:
    if spec["loader"] == "hf":
        from transformers import AutoTokenizer, PreTrainedTokenizerFast
        try:
            tok = AutoTokenizer.from_pretrained(spec["id"])
        except Exception:
            tok = PreTrainedTokenizerFast.from_pretrained(spec["id"])
        en_ids = tok.encode(EN_TEXT)
        vi_ids = tok.encode(VI_TEXT)
    else:
        import tiktoken
        enc = tiktoken.get_encoding(spec["id"])
        en_ids = enc.encode(EN_TEXT)
        vi_ids = enc.encode(VI_TEXT)
    entry["ratio_vi_over_en"] = round(len(vi_ids) / len(en_ids), 3)
except Exception as e:
    entry["status"] = "skipped"
    entry["error"] = str(e)

Experiment

Here's the actual run, step by step. The complete script is in the appendix at the end of this post.

1. Write matched English/Vietnamese paragraphs for this benchmark specifically — not lifted from an external source, so token counts reflect genuinely parallel content rather than sentences that just happen to differ in length:

EN_TEXT = (
    "Running a language model in production is nothing like running it in a "
    "notebook. The moment real traffic hits your endpoint, every token you "
    "send and receive turns into a line item on your bill. ..."
)

VI_TEXT = (
    "Chạy một mô hình ngôn ngữ trong môi trường production hoàn toàn khác với "
    "chạy trong notebook. Ngay khi traffic thật đổ vào endpoint, mỗi token "
    "bạn gửi đi và nhận về đều trở thành một dòng trong hóa đơn. ..."
)

2. List every tokenizer to benchmark, tagged by generation and loader. Here are 3 of the 13:

TOKENIZER_SPECS = [
    {"name": "gpt2", "family": "legacy-english-only", "loader": "hf", "id": "gpt2"},
    {"name": "phobert-base", "family": "vietnamese-specific", "loader": "hf", "id": "vinai/phobert-base"},
    {"name": "o200k_base (GPT-4o / GPT-4.1)", "family": "modern-multilingual", "loader": "tiktoken", "id": "o200k_base"},
]

3. Print each result as it's computed, and record failures honestly instead of estimating them:

status_word = entry["status"]
detail = (
    f"ratio={entry.get('ratio_vi_over_en')}"
    if status_word == "ok"
    else f"error={entry.get('error')}"
)
print(f"[{status_word:7s}] {spec['name']:32s} {detail}", flush=True)

That's why llama-3-8b and gemma-2-9b-it show up as "Gated (no token)" in the table above instead of a made-up ratio.

4. Write the full payload — both texts, their word/char counts, and every tokenizer's result — to disk for the post:

payload = {
    "en_text": EN_TEXT, "vi_text": VI_TEXT,
    "en_word_count": len(EN_TEXT.split()), "vi_word_count": len(VI_TEXT.split()),
    "en_char_count": len(EN_TEXT), "vi_char_count": len(VI_TEXT),
    "results": results,
}

Run it yourself: uv run python tokenizer_cost_benchmark.py.

Summary

• The 4.54x statistic applies only to obsolete 2019 tokenizers.

• Modern production APIs tokenize Vietnamese between 1.1x and 2.1x relative to English.

• Upgrading from cl100k_base to o200k_base saves ~37% in token overhead for Vietnamese text.

• Specialized Vietnamese models (PhoBERT) achieve 0.87x parity, proving vocabulary design is what matters.

References

• Better Language Models and Their Implications (GPT-2) — OpenAI's original GPT-2 release, the tokenizer this post traces the "4.54x" myth back to.

• PhoBERT — VinAI's Vietnamese-specific tokenizer/model, the 0.87x baseline in this post.

• tiktoken — OpenAI's BPE library implementing cl100k_base and o200k_base, the two tokenizers compared in the "37% free win" section.

• Qwen3-8B — one of the modern multilingual tokenizers benchmarked.

• "The Tokenization Tax" — the independent production study (Techcombank) this post cites as corroboration.

What tokenizer ratios have you observed in your production workloads? Let's discuss in the comments.

👉 Follow my work: LinkedIn | GitHub

Appendix: Full Script

For anyone who wants the complete, runnable file:

#!/usr/bin/env python3
"""Measure real token-count inflation for Vietnamese vs English across tokenizer
generations (legacy English-only BPE, Vietnamese-specific, modern multilingual).

No fabricated numbers: every tokenizer that fails to load (gated repo, network
error, missing files) is recorded as skipped with the real error, not estimated.
"""
import json
import sys
import time
from pathlib import Path

OUT_DIR = Path("content/2026-09-01/tokenizer-cost/scratch")

# Matched-content EN/VI pair, first-person voice, written for this benchmark
# (not lifted from an external source) so token counts reflect genuinely
# parallel content rather than independently-varying sentence structure.
EN_TEXT = (
    "Running a language model in production is nothing like running it in a "
    "notebook. The moment real traffic hits your endpoint, every token you "
    "send and receive turns into a line item on your bill. A prompt that "
    "looked short on screen can balloon once the tokenizer gets its hands on "
    "it, especially if your users aren't writing in English. I learned this "
    "the hard way while building a social listening pipeline that ingested "
    "thousands of Vietnamese comments an hour. The dashboard cost numbers "
    "didn't match my mental model at all, and it took a real benchmark, not "
    "a blog post, to figure out why."
)

VI_TEXT = (
    "Chạy một mô hình ngôn ngữ trong môi trường production hoàn toàn khác với "
    "chạy trong notebook. Ngay khi traffic thật đổ vào endpoint, mỗi token "
    "bạn gửi đi và nhận về đều trở thành một dòng trong hóa đơn. Một prompt "
    "nhìn có vẻ ngắn trên màn hình có thể phình to đáng kể sau khi qua tay "
    "tokenizer, nhất là khi người dùng của bạn không viết bằng tiếng Anh. "
    "Mình nhận ra điều này một cách khá đau đớn khi xây dựng pipeline social "
    "listening xử lý hàng nghìn bình luận tiếng Việt mỗi giờ. Con số chi phí "
    "trên dashboard không khớp với mô hình mình nghĩ trong đầu chút nào, và "
    "phải chạy benchmark thật sự, chứ không phải đọc blog, mới tìm ra lý do."
)

TOKENIZER_SPECS = [
    {"name": "gpt2", "family": "legacy-english-only", "loader": "hf", "id": "gpt2"},
    {"name": "roberta-base", "family": "legacy-english-only", "loader": "hf", "id": "FacebookAI/roberta-base"},
    {
        "name": "phobert-base",
        "family": "vietnamese-specific",
        "loader": "hf",
        "id": "vinai/phobert-base",
        "note": "PhoBERT's BPE expects word-segmented Vietnamese input (e.g. via VnCoreNLP); "
                "raw unsegmented text was used here, so this number is a lower bound on "
                "PhoBERT's real efficiency, not its best case.",
    },
    {"name": "gpt2-vietnamese", "family": "vietnamese-specific", "loader": "hf", "id": "NlpHUST/gpt2-vietnamese"},
    {"name": "xlm-roberta-base", "family": "modern-multilingual", "loader": "hf", "id": "FacebookAI/xlm-roberta-base"},
    {
        "name": "mmBERT-base",
        "family": "modern-multilingual",
        "loader": "hf",
        "id": "jhu-clsp/mmBERT-base",
        "note": "ModernBERT-architecture encoder trained on 3T tokens across 1,833 languages "
                "(JHU CLSP, ICML 2026) — first model to meaningfully beat XLM-R on multilingual "
                "benchmarks; encoder, not a causal LM, but the tokenizer comparison is still valid.",
    },
    {"name": "qwen2.5-7b", "family": "modern-multilingual", "loader": "hf", "id": "Qwen/Qwen2.5-7B"},
    {"name": "qwen3-8b", "family": "modern-multilingual", "loader": "hf", "id": "Qwen/Qwen3-8B"},
    {"name": "deepseek-v3.2", "family": "modern-multilingual", "loader": "hf", "id": "deepseek-ai/DeepSeek-V3.2"},
    {"name": "llama-3-8b", "family": "modern-multilingual", "loader": "hf", "id": "meta-llama/Meta-Llama-3-8B"},
    {"name": "gemma-2-9b-it", "family": "modern-multilingual", "loader": "hf", "id": "google/gemma-2-9b-it"},
    {"name": "cl100k_base (GPT-3.5 / GPT-4)", "family": "modern-multilingual", "loader": "tiktoken", "id": "cl100k_base"},
    {"name": "o200k_base (GPT-4o / GPT-4.1)", "family": "modern-multilingual", "loader": "tiktoken", "id": "o200k_base"},
]


def run():
    results = []
    for spec in TOKENIZER_SPECS:
        entry = {"name": spec["name"], "family": spec["family"], "id": spec["id"]}
        t0 = time.time()
        try:
            if spec["loader"] == "hf":
                from transformers import AutoTokenizer, PreTrainedTokenizerFast
                try:
                    tok = AutoTokenizer.from_pretrained(spec["id"])
                except Exception:
                    # Some very new model configs (e.g. deepseek_v32) aren't fully
                    # recognized by this transformers version's AutoConfig yet.
                    # The tokenizer files themselves are still standard, so load
                    # them directly, bypassing model-aware AutoConfig entirely.
                    tok = PreTrainedTokenizerFast.from_pretrained(spec["id"])
                    entry["loaded_via"] = "PreTrainedTokenizerFast fallback (AutoTokenizer failed on model config)"
                en_ids = tok.encode(EN_TEXT)
                vi_ids = tok.encode(VI_TEXT)
            else:
                import tiktoken
                enc = tiktoken.get_encoding(spec["id"])
                en_ids = enc.encode(EN_TEXT)
                vi_ids = enc.encode(VI_TEXT)
            entry["en_tokens"] = len(en_ids)
            entry["vi_tokens"] = len(vi_ids)
            entry["ratio_vi_over_en"] = round(len(vi_ids) / len(en_ids), 3)
            entry["status"] = "ok"
            if "note" in spec:
                entry["note"] = spec["note"]
        except Exception as e:  # noqa: BLE001 - deliberately broad, this is a survey script
            entry["status"] = "skipped"
            entry["error"] = str(e).splitlines()[0][:300]
        entry["load_and_encode_seconds"] = round(time.time() - t0, 2)
        status_word = entry["status"]
        detail = (
            f"ratio={entry.get('ratio_vi_over_en')}"
            if status_word == "ok"
            else f"error={entry.get('error')}"
        )
        print(f"[{status_word:7s}] {spec['name']:32s} {detail}", flush=True)
        results.append(entry)

    payload = {
        "en_text": EN_TEXT,
        "vi_text": VI_TEXT,
        "en_word_count": len(EN_TEXT.split()),
        "vi_word_count": len(VI_TEXT.split()),
        "en_char_count": len(EN_TEXT),
        "vi_char_count": len(VI_TEXT),
        "results": results,
    }
    OUT_DIR.mkdir(parents=True, exist_ok=True)
    out_path = OUT_DIR / "benchmark_results.json"
    out_path.write_text(json.dumps(payload, ensure_ascii=False, indent=2))
    print(f"\nWrote {out_path}", flush=True)
    return payload


if __name__ == "__main__":
    run()