I Tested 3 LLMs on Vietnamese Idioms. They Got It Backwards.

A paper from LREC 2026 presented an intriguing finding, one that fits a pattern I've seen since benchmarking Vietnamese BERT models: state-of-the-art LLMs score under 50% accuracy on VIVID, a benchmark of 1,636 authentic Vietnamese idioms and proverbs.

Even specialized Vietnamese models struggled. Few-shot prompting barely helped.

I wanted to see these failure modes firsthand.

Unlike other benchmarks with private answer keys, VIVID publishes ground-truth meanings on GitHub under an MIT license. I sampled 6 idioms flagged with complex traits (Sino-Vietnamese roots, archaic vocabulary, folk lore) and asked Qwen3-8B, Llama-3.1-8B-Instruct, and gpt-oss-20b to explain each one.

The result? Across 18 model-idiom explanations, not a single answer was completely correct.

Worse, models delivered their explanations with total grammatical fluency and zero hedging, even when stating the exact opposite of the idiom's actual meaning.

The Scorecard

for idiom in SELECTED_IDIOMS:
    prompt = f'Giải thích ý nghĩa của thành ngữ/tục ngữ tiếng Việt sau bằng 1-2 câu ngắn gọn: "{idiom}"'
    # Evaluated directly against VIVID ground truth meanings

Out of 18 responses: 0 fully correct, 6 partial, 2 empty completions, and 10 completely incorrect. Three answers directly inverted the moral lesson.

The Two Most Striking Failures

Two responses stood out for their confident fabrication:

1. "Võng tía lọng đào"

• Actual meaning: A purple hammock and peach-colored parasol, representing aristocratic honor and prestige.

• Llama-3.1's answer: "People talking about something without reason, just because they want to talk."

It is fluent, grammatically clean Vietnamese with zero connection to reality.

2. "Năng may hơn dày giẻ"

• Actual meaning: A metaphor comparing a hardworking tailor to one who cheats customers by padding fabric with rags. The moral: honest diligence brings sustainable success over deceptive shortcuts.

• gpt-oss-20b's answer: "Luck determines success more than effort; without luck, effort brings disappointing results."

The model inverted the entire ethical lesson of the proverb, delivering it with the authoritative tone of a teacher.

Why Confident Errors Are Dangerous

An AI that responds with "I don't know" is safe.

An AI that confidently explains a cultural idiom backwards is a liability.

In education, customer support, or translation tools, end users who lack deep cultural background cannot distinguish a hallucinated explanation from a correct one.

The VIVID paper categorizes these failures into four main drivers — the same category of gap I ran into fine-tuning transformers for language detection:

1. Literal over-interpretation: Taking figurative allegories at face value.

2. Archaic vocabulary gaps: Failing on Sino-Vietnamese roots.

3. Cultural disconnection: Lacking grounding in Vietnamese folk traditions.

4. Pragmatic flattening: Defaulting to generic modern tropes (like "luck vs. effort") when confused.

Experiment

Here's the actual run, step by step. The complete script is in the appendix at the end of this post.

1. Load the dataset and index it by idiom text so each row's ground truth is a dict lookup away:

def load_rows():
    rows = {}
    with open(DATASET_PATH, encoding="utf-8-sig") as f:
        r = csv.DictReader(f)
        for row in r:
            rows[row["Idiom_Proverb"].strip()] = row
    return rows

2. Call the model, keeping the reasoning trace alongside the answer — same reason as the V-Bench script: a None answer needs to be distinguishable from a token-exhausted one:

def call_model(model, prompt):
    resp = requests.post(
        ROUTER_URL,
        headers={"Authorization": f"Bearer {HF_TOKEN}", "Content-Type": "application/json"},
        json={"model": model, "messages": [{"role": "user", "content": prompt}],
              "max_tokens": 600, "temperature": 0.2},
        timeout=120,
    )
    resp.raise_for_status()
    data = resp.json()
    choice = data["choices"][0]
    msg = choice["message"]
    return {
        "content": msg.get("content"),
        "reasoning": msg.get("reasoning_content") or msg.get("reasoning"),
        "finish_reason": choice.get("finish_reason"),
    }

3. For each of the 6 idioms, pull its taxonomy tags and theme, then ask all 3 models to explain it:

for idiom in SELECTED_IDIOMS:
    row = rows.get(idiom)
    taxonomy_flags = [TAXONOMY_NAMES[i] for i in range(5)
                       if row.get(f"Linguistic_Complexity_Taxonomy_{i+1}", "0") == "1.0"]
    theme = THEME_NAMES.get(int(float(row["Semantic_Theme"])), "unknown")
    entry = {
        "idiom": idiom, "ground_truth_meaning": row["Meaning"].strip(),
        "taxonomy": taxonomy_flags, "theme": theme, "models": {},
    }
    prompt = f'Giải thích ý nghĩa của thành ngữ/tục ngữ tiếng Việt sau bằng 1-2 câu ngắn gọn: "{idiom}"'
    for model in MODELS:
        try:
            out = call_model(model, prompt)
            entry["models"][model] = out
        except Exception as e:
            entry["models"][model] = {"error": str(e)[:300]}
    results.append(entry)

Summary

• On the challenging tail of Vietnamese idioms, current open LLMs produced 0/18 fully correct explanations.

• Inverted meanings and hallucinations were delivered with complete fluency and no uncertainty markers.

• VIVID provides an open, reproducible dataset for testing cultural reasoning.

• Never rely on an LLM's confidence as a proxy for cultural accuracy.

References

• VIVID dataset repository — the idiom benchmark's GitHub repo (MIT-licensed ground truth).

• VIVID_Dataset.csv — the raw ground-truth file used for scoring.

• Qwen3-8B

• Llama-3.1-8B-Instruct

What Vietnamese idioms have you seen AI models struggle with? Share your examples in the comments.

👉 Follow my work: LinkedIn | GitHub

Appendix: Full Script

For anyone who wants the complete, runnable file:

#!/usr/bin/env python3
"""Ask 3 open-weight LLMs to explain real Vietnamese idioms from the VIVID benchmark
dataset (github.com/ReML-AI/VIVID, MIT licensed), and compare their explanations
against the dataset's own ground-truth meaning.

Unlike the V-Bench post, VIVID's public dataset DOES ship with ground truth (the
`Meaning` column), so this is a real correctness check, not just a qualitative read.
We do NOT reproduce VIVID's own LLM-as-a-judge aspect-based scoring methodology
(Cohen's kappa 0.792 against human judges) — that's a more elaborate rubric than
this script implements. This is a simpler, direct "does the model's explanation
match the dataset's meaning" comparison, judged by this session reading Vietnamese,
stated as such rather than presented as a reproduction of the paper's own scores.
"""
import csv
import json
import os
import time
from pathlib import Path

import requests

DATASET_PATH = Path("../vivid-idioms/scratch/VIVID_Dataset.csv")
OUT_PATH = Path("..

💡 TL;DR & Key Takeaways:
**TL;DR**
State‑of‑the‑art LLMs, even Vietnamese‑specialized ones, score below 50 % on the VIVID benchmark of 1,636 authentic Vietnamese idioms and fail completely on a hand‑picked set of six complex idioms. The models produce fluently written, confident explanations that are frequently wrong or even opposite to the true meanings, exposing a serious gap in idiom comprehension.

- VIVID provides open‑source ground‑truth meanings (MIT‑licensed) for all idioms, enabling transparent evaluation.
- In a test of Qwen3‑8B, Llama‑3.1‑8B‑Instruct, and gpt‑oss‑20b on 6 idioms (18 responses), **0** explanations were fully correct.
- Responses were grammatically perfect and unqualified, yet often hallucinated, inverted, or entirely unrelated, highlighting the models’ inability to handle culturally nuanced language.

/vivid-idioms/scratch/vivid_qualitative_results.json")

MODELS = [
    "Qwen/Qwen3-8B",
    "meta-llama/Llama-3.1-8B-Instruct",
    "openai/gpt-oss-20b",
]

# Idiom text -> which taxonomy column made it interesting, for narrative labeling
SELECTED_IDIOMS = [
    "Gội gió tắm mưa",
    "Đàn ông là nhà, đàn bà là cửa",
    "Võng tía lọng đào",
    "Phúc đức khán tử tôn",
    "Tiếng bấc tiếng chì",
    "Năng may hơn dày giẻ",
]

HF_TOKEN = os.environ["HF_TOKEN"]
ROUTER_URL = "https://router.huggingface.co/v1/chat/completions"

TAXONOMY_NAMES = ["literal", "sino_vietnamese", "uncommon_vocab", "folk_knowledge", "pragmatic_nuance"]
THEME_NAMES = {1: "Love", 2: "Virtues", 3: "Criticism", 4: "Work and nature", 5: "Society", 6: "Life Lessons", 7: "Others"}


def load_rows():
    rows = {}
    with open(DATASET_PATH, encoding="utf-8-sig") as f:
        r = csv.DictReader(f)
        for row in r:
            rows[row["Idiom_Proverb"].strip()] = row
    return rows


def call_model(model, prompt):
    resp = requests.post(
        ROUTER_URL,
        headers={"Authorization": f"Bearer {HF_TOKEN}", "Content-Type": "application/json"},
        json={"model": model, "messages": [{"role": "user", "content": prompt}],
              "max_tokens": 600, "temperature": 0.2},
        timeout=120,
    )
    resp.raise_for_status()
    data = resp.json()
    choice = data["choices"][0]
    msg = choice["message"]
    return {
        "content": msg.get("content"),
        "reasoning": msg.get("reasoning_content") or msg.get("reasoning"),
        "finish_reason": choice.get("finish_reason"),
    }


def run():
    rows = load_rows()
    results = []
    for idiom in SELECTED_IDIOMS:
        row = rows.get(idiom)
        if row is None:
            print(f"[skip] '{idiom}' not found in dataset")
            continue
        taxonomy_flags = [TAXONOMY_NAMES[i] for i in range(5)
                           if row.get(f"Linguistic_Complexity_Taxonomy_{i+1}", "0") == "1.0"]
        theme = THEME_NAMES.get(int(float(row["Semantic_Theme"])), "unknown")
        entry = {
            "idiom": idiom,
            "ground_truth_meaning": row["Meaning"].strip(),
            "taxonomy": taxonomy_flags,
            "theme": theme,
            "models": {},
        }
        prompt = (
            f'Giải thích ý nghĩa của thành ngữ/tục ngữ tiếng Việt sau bằng 1-2 câu ngắn gọn: "{idiom}"'
        )
        for model in MODELS:
            t0 = time.time()
            try:
                out = call_model(model, prompt)
                out["latency_s"] = round(time.time() - t0, 2)
                entry["models"][model] = out
                print(f"[ok] idiom='{idiom}' model={model} finish={out['finish_reason']}")
            except Exception as e:  # noqa: BLE001
                entry["models"][model] = {"error": str(e)[:300]}
                print(f"[FAIL] idiom='{idiom}' model={model} error={e}")
        results.append(entry)
    OUT_PATH.write_text(json.dumps(results, ensure_ascii=False, indent=2))
    print(f"\nWrote {OUT_PATH}")


if __name__ == "__main__":
    run()