BM25 Beat the RAG Frameworks on Vietnamese Law

💡 TL;DR & Key Takeaways
In the ViLegalExpert benchmark for Vietnamese legal retrieval, plain BM25 outperformed advanced RAG frameworks like LightRAG and RAPTOR on F1, MRR, and precision, with a hybrid BM25 approach achieving the highest score of 18.06 F1. The benchmark evaluates retrieval and question answering as separate tasks by providing models with gold legal context, meaning it does not test the full pipeline where retrieval errors impact final answers.

• Sparse BM25 beat all four RAG-framework baselines on retrieval metrics, though the F1 margin over LightRAG was negligible (0.07), while the MRR gap was more significant.

• Fine-tuned Vietnamese seq2seq models like BARTpho outscored instruction-tuned LLMs on abstractive QA metrics, a result the author attributes to metric biases favoring length and phrasing match rather than inherent superiority.

• The paper contains inconsistent dataset size figures (172K, 177,158, and 117,158 samples), and the author recommends verifying data sizes against released files and establishing a BM25 baseline before deploying complex graph-based retrieval systems.

The newest, most elaborate retrieval methods in a new Vietnamese legal benchmark lost to a keyword scorer. Then I read how the benchmark measures them, and the surprise got smaller and more useful.

In ViLegalExpert, a new Vietnamese legal benchmark of real citizen-lawyer questions, plain BM25 outscored LightRAG, RAPTOR, HippoRAG and MiniRAG on F1, MRR and precision, and hybrid BM25 with dense retrieval did best at 18.06 F1. Absolute scores stay low, and the QA tasks hand models the gold legal text, so they never test retrieval.

• Retrieval of the right legal provision is the hard part: the best F1 in the paper is 18.06.

• Sparse BM25 beats all four RAG-framework baselines on all three retrieval metrics. On F1 the margin over LightRAG is 0.07, so read that one as a tie.

• QA models receive the gold legal context, so the benchmark scores retrieval and answering separately and never as one pipeline.

• The paper gives three different sizes for the same dataset: 172K, 177,158 and 117,158.

I did not run this benchmark. Every number below is from the paper (arXiv 2609.39189, UIT, Vietnam National University HCMC, 30 Sep 2026), whose full text I read. Where I add an interpretation, I say so.

What is ViLegalExpert, and why does it matter?

It is a Vietnamese legal retrieval and QA benchmark built from real online consultations, where citizens ask questions and lawyers answer. Legal experts then checked whether the provisions the lawyers cited are valid and relevant, and kept only the links that held up.

The corpus behind it has 9,563 legal documents: the Constitution, laws, decrees, circulars. The questions are short (average 21.08 tokens) and the lawyer answers are long (average 380.51 tokens), across 34 legal domains.

That combination is the point. A citizen types an everyday sentence, and the system has to land on an article of a formal document written in different words. The authors say earlier Vietnamese resources were largely built for shared tasks, manual annotation or synthetic scenarios. Per the paper's own comparison table, VLegal-Bench has 10,450 samples and VLQA has 24,012 across 27 domains.

Which retrieval method won, and by how much?

Source: Table 5 of the paper. I reordered the rows by F1.

Sparse BM25 is ahead of every RAG framework on all three metrics. The F1 gap to LightRAG is 0.07 points, which I would not call a win. The MRR gap is real: 23.23 against LightRAG's 20.53 and RAPTOR's 21.44. HippoRAG's MRR of 12.63 sits well below its F1 and its precision is the lowest in the table. My read: it surfaces some relevant provisions but ranks them poorly.

Two caveats from the pages I read. The paper does not state the cut-off k behind F1 and precision, so I cannot tell you what "precision" is measured over. And the row labelled "BM25 (Dense)" cites a dense passage retriever replication; I read it as a dense retriever, but the text does not name the encoder.

Why would a keyword scorer beat graph-based RAG here?

The authors' explanation is that legal text uses standardized terminology and recurring phrases, which gives exact-match signals that semantic representations can lose. Their hybrid result fits: sparse catches terminology, dense catches paraphrase, and together they beat either.

My own reading, and it is only that: LightRAG, HippoRAG and RAPTOR build graphs or summary trees over a corpus, which pays off when a question needs several passages stitched together. This task is closer to lookup. One question maps to a specific article. If that is right, the extra structure adds cost and indexing time without adding signal. The benchmark does not test that explanation, so treat it as a hypothesis for you to check on your own corpus.

I wrote about what graph-based pipelines cost to build in Building a GraphRAG Pipeline: What It Really Costs, Pt 2. Whatever that bill is for you, a graph method should beat BM25 on your own queries before you pay it.

Does the benchmark test a full RAG pipeline?

No, and this is the part I would have missed without reading the setup section.

Diagram: ViLegalExpert scores legal retrieval and question answering as two separate tasks; the QA task receives the gold legal context, so the two are never connected.

For QA, the paper says models are provided with the question and its associated legal context. Retrieval errors never reach the answer. So the QA table tells you how well a model reads a correct provision, not how well a system behaves when retrieval hands it the wrong one.

In a deployed legal assistant, the wrong-provision case is the dangerous one. That failure is not in this benchmark.

Do instruction-tuned LLMs really lose to BARTpho?

On abstractive QA, they do:

Source: Table 4. Qwen2.5-14B scores below its 7B sibling on all three metrics.

I would not read this as "small Vietnamese seq2seq beats LLMs for legal answers". Three reasons, all mine:

1. The metric rewards length and phrasing match. ROUGE-L and METEOR compare against lawyer answers that average 380 tokens. A fluent, shorter, differently worded chat answer can be correct and still score low.

2. The LLMs are 2024-era. Llama3, Llama3.1, Qwen2.5 and Mistral-7B-v0.3 were the paper's open baselines. The paper says instruction-tuned models were evaluated with a consistent prompting template. It does not mention fine-tuning them, while trainable models were optimized on the training set. I could not confirm the exact setup from the text I read.

3. No human or grounded-correctness check. Nothing here measures whether a generated answer cites the right article.

The defensible claim is narrower: with this prompting and these overlap metrics, the fine-tuned Vietnamese seq2seq models score higher.

Which numbers in the paper don't agree?

The dataset size appears three ways:

Table 2 also lists 87,695 IR samples. I would trust 177,158 because the tables agree with each other, but I would check the released data before quoting a size. Cite the number from the file you actually load.

What I would do before trusting a Vietnamese legal RAG demo

These are rules I am taking from this paper, not results I measured.

• Build the BM25 baseline first. If your fancy retriever cannot beat it on your queries, you have learned something cheap.

• Try hybrid before graph. Sparse plus dense gave the best score in the one place anyone measured it here.

• Score retrieval and answering separately, then together. A benchmark that hands over the gold context is hiding your worst failure.

• Do not judge a generator on ROUGE alone. Check that it cites the right article.

• Expect low absolute numbers. An F1 of 18 on real citizen questions is the state of the art in this paper, not a bug in your setup.

The same pattern shows up on the encoder side. In ViLegalExpert's extractive QA, ViDeBERTa scored 62.79 ROUGE-L, mBERT 62.36, and PhoBERT 50.68, so the Vietnamese-specific model did not win there. The authors suggest Vietnamese-only pretraining does not automatically help legal answer extraction. For a broader encoder comparison, see my Vietnamese BERT benchmark.

How to check this yourself

• Paper: arXiv 2609.39189, "ViLegalExpert: A Large-Scale Benchmark for Vietnamese Legal Retrieval and Question Answering from Real-World Consultations". Tables 3, 4 and 5 hold every number above.

• I did not run the benchmark and have not checked whether the data is public.

• Open questions I could not settle from the text: the retrieval cut-off k, the dense encoder behind "BM25 (Dense)", and whether any LLM baseline was fine-tuned.

If you have run any of these retrievers on Vietnamese legal text, I would like to hear whether BM25 held up on your corpus.

Share your thoughts in the comments — I'd love to hear how this technology is impacting your industry.

Follow me: LinkedIn | GitHub