Read next
LREC-COLING 2024Teaching machines to translate medical text between Vietnamese and English10 min read
LREC 2026Catching the English words in Vietnamese medical speech11 min readAll publications · Get in touch about this research
EMNLP 2026 · Interactive explainer
English biomedical text comes with hundreds of thousands of expert-labeled terms. Vietnamese had no corpus labeled with the same standard. We built one, then tested whether today’s AI models can manage without it.
What should a model find in this sentence?
A real title from the corpus, with its expert labels. Tap any word to see its tag.
Three medical things are named here: a gene, an infection and a disease. This sentence follows us through the whole page.
This task is called , or NER. A model reads a sentence and marks the spans that name medical things: a gene, a disease, a body part, a procedure. Medical search engines, clinical decision tools and knowledge graphs all start with this step.
Think of it asA careful reader going through a text with a set of highlighters, one colour for each kind of medical term.
The labels are codes. T038 means Biologic Function in the , the U.S. National Library of Medicine’s master vocabulary for biomedicine. A code means the same thing in any language. That is the idea behind En-ViMedNER: give Vietnamese sentences the exact labels that English sentences already carry.
The standard English resource is MedMentions: 4,392 PubMed abstracts in which experts linked every medical mention to a UMLS concept. Its ST21pv version keeps 21 clinically useful .
Vietnamese datasets exist, but they are smaller, cover narrow domains such as COVID-19 or spoken transcripts, and each uses its own label set. None used UMLS types, so a Vietnamese model could not be compared with, or learn from, the large English resources.
How much bigger is it than the last Vietnamese medical NER set?
Each dot is 100 sentences.
44,892 sentence pairs · 21 UMLS types · PubMed abstracts
9,000 sentences · 18 entity types · spoken medical transcripts
About five times as many sentences, and labels that line up with the English resources.
The 21 types are far from even. Biologic functions such as diseases, and chemicals such as drugs and genes, make up nearly 40% of all mentions. Body systems and foods appear only a few hundred times each. That imbalance comes straight from real biomedical writing.
Which kinds of medical terms are most common?
Entity mentions by semantic type, full corpus, 202,949 mentions. The two types in our example sentence are colored.
Biologic functions and chemicals alone make up 38.8% of all mentions; body systems make up 0.2%.
Labeling 200,000 Vietnamese entities by hand from scratch would take years. We reused the English labels instead and moved them across, a step called . Follow the example sentence through the pipeline.
Think of it asTwo editions of the same book, one in English and one in Vietnamese. Someone has highlighted the English edition, and you copy each highlight onto the matching words in the Vietnamese edition.
How does an English label end up on Vietnamese words?
Pick a step, or use the buttons below.
Every one of the 44,892 sentence pairs went through these four steps; only 1.77% needed a second look by people.
We drew 450 test sentences at random, holding 2,085 entity mentions. Two annotators independently checked every projected span against the English source and the Vietnamese text, before anything was corrected. Their agreement was high ( = 0.83).
How often did the moved labels land on the wrong words?
Each square is one entity mention. Pick an error type to find it in the grid.
35 of 2,085 mentions (1.7%) were wrong before correction, and most of those were off by a word at one edge.
| Type | C1 Boundary | C2 Projection | C3 Untranslated | C4 Abbreviation | C5 Terminology | Total |
|---|---|---|---|---|---|---|
| T017 | 3 | 3 | 0 | 0 | 0 | 6 |
| T058 | 5 | 0 | 0 | 0 | 1 | 6 |
| T082 | 3 | 0 | 0 | 1 | 1 | 5 |
| T170 | 2 | 0 | 0 | 1 | 1 | 4 |
| T033 | 0 | 1 | 1 | 0 | 1 | 3 |
| T038 | 2 | 1 | 0 | 0 | 0 | 3 |
| T098 | 1 | 0 | 2 | 0 | 0 | 3 |
| T103 | 2 | 0 | 0 | 0 | 0 | 2 |
| T062 | 2 | 0 | 0 | 0 | 0 | 2 |
| T204 | 0 | 0 | 1 | 0 | 0 | 1 |
| Total | 20 | 5 | 4 | 2 | 4 | 35 |
Of the 20 boundary errors, 12 (60.0%) were over-projection, where the span included extra words or modifiers, and 8 (40.0%) were under-projection. The 35 errors fall in 30 of the 450 audited sentences. Overall annotator agreement: κ = 0.8315.
Most misses happen at the edges. In 20 of the 35 cases the projection found the right term but took a word too many or one too few. Vietnamese makes this easy to do: small words such as này (“this”), khác (“other”) or mỗi (“per”) sit right beside a term and can slip into its span. Our automatic checks look for exactly these words.
After the audit, all 35 errors were corrected by consensus, so this 450-sentence mini-test is . The rest of the corpus is : projected, checked by rules and repaired where flagged, but not read twice by people.
can tag a sentence without any training, if you describe the task in a prompt. If that worked well, a dataset like this would matter less. So we compared three approaches on the same Vietnamese sentences: small models on Vietnamese, the same kind of model fine-tuned on English only, and prompted LLMs given 0, 1 or 3 similar examples ().
Think of it asFine-tuning is giving a generalist a specialist course. Prompting is handing the same generalist a few worked examples and hoping they pick it up on the spot.
First, how the score works. NER is graded with , which combines two questions: of the spans the model marked, how many were right, and of the real entities, how many did it find?
What does an F1 score count?
You play the model on our example sentence. Mark which real entities it found, and add wrong guesses. Higher is better.
The 3 real entities
Spans the model marked that are not entities
F1 = 2 × precision × recall ÷ (precision + recall). An entity counts only if both its span and its type are exactly right.
At real scale: ViPubMedDeBERTa-base on the full test set, per 100
The game is illustrative; the wrong-guess spans are words from the sentence that carry no label. The bottom rows use the paper’s numbers (Table 10). An F1 near 53 means the model gets about half of everything right.
Make a guess. A fine-tuned model with 22 million parameters scores 53.8. Where does GPT-5.4 land, with three worked examples in its prompt?
Score is entity-level F1 on the mini-test. An entity counts only if its span and its type are both exactly right.
Can prompted LLMs match a small model trained on Vietnamese?
on the 450-sentence mini-test. Higher is better. Dashed line: best fine-tuned model (53.78).
No. With three examples, the best prompted model trails the best fine-tuned one by 17.31 F1.
Every model trained on Vietnamese lands between 51 and 54. Train XLM-R on English and test it on Vietnamese, and it loses about 15 points. Prompting improves a lot with examples: Gemini goes from 12.70 with none to 36.47 with three. The best prompted model still trails the best fine-tuned one by 17.31 points. The small open models mostly fail to produce usable tags at all.
The same pattern holds on the full test set of 8,947 sentences:
Vietnamese supervision is worth 16.16 points over English transfer.
Among the fine-tuned models, the one pre-trained on Vietnamese biomedical text beats general models 25 times its size. Square area is proportional to parameter count.
Does a bigger model tag better?
Square area is proportional to parameter count. Higher F1 is better.
Not here. The 86M-parameter model pre-trained on Vietnamese biomedical text beats the 550M general model. Micro-F1 on the full test set, all trained on Vietnamese.
T033 Finding has 3,108 test mentions and still scores lowest of the ten types below. Findings overlap with other categories and are easy to confuse. Rare but distinctive types, like T031 Body Substance, score well above the average.
Do common entity types always score higher?
ViPubMedDeBERTa-base, full test set. Five most frequent and five least frequent types.
No. T033 Finding is the fifth most common type and scores lowest; rare T031 and T005 score above the average.
Most medical research is published in English. A Vietnamese clinician would want a translation and the tagged entities together: English in, tagged Vietnamese out. Because En-ViMedNER pairs every English sentence with a labeled Vietnamese one, models can learn this directly. There are three ways to wire it up.
Think of it asAn interpreter who, while translating a medical article aloud, also points out every drug, gene and disease as they go.
Which way of wiring translation and tagging works best?
Pick a pipeline to highlight its scores. NLLB-200 (600M) is a translation model; umT5-base is a general text-to-text model.
It depends on the score: one step (M1) wins on Judge-F1, tag-then-translate (M3) on Exact-F1. : translation quality, higher is better. Judge-F1: entities judged correct in meaning and type. Exact-F1: entities that match the reference word for word. ● marks the best pipeline per metric. Mini-test.
Doing everything in one step (M1) scores best on the lenient Judge-F1. Tagging first and translating second (M3) scores best on the strict Exact-F1, for both models. Neither design wins everywhere; the better one depends on how you count.
A correct translation can use different words from our reference translation. Exact-F1 counts that as a miss. Judge-F1 asks a separate model, gpt-5.4-mini, whether each predicted entity means the same thing and carries the same type. On 100 sampled outputs its verdicts agreed with human judges at κ = 0.87.
When does a differently worded entity still count?
Illustrative predictions written for this explainer. Pick one.
Exact-F1 only accepts the reference wording. Judge-F1 also accepts a different wording with the same meaning and type, but neither accepts the wrong type or half a span.
A model can translate fluently and still drop or mislabel the entities. Qwen-SEA-LION-8B, a model adapted for Southeast Asian languages, reaches a BLEU of 52.89 with three examples, higher than any fine-tuned model. Its Judge-F1 is only 18.43. Fine-tuned NLLB with 600 million parameters beats prompted GPT-5.4 on entities: 45.44 against 42.64.
Does good translation mean good tagging?
Mini-test. LLMs shown with 3 examples. Tap a point for its numbers.
No. SEA-LION-8B translates about as well as the best systems but finds far fewer entities; fine-tuned NLLB does well on both.
To count as an answer, an output must contain the Vietnamese sentence with valid entity tags. The open-source LLMs frequently returned something unparseable, or text with no entities at all. Those outputs score zero. The fine-tuned models and the closed LLMs never did.
How often do small open LLMs fail to return usable tags?
Each square is one of the 450 mini-test sentences. Red: no usable tags.
Often: up to 292 of 450 outputs for SEA-LION-8B with no examples. The closed LLMs never failed.
Read next
LREC-COLING 2024Teaching machines to translate medical text between Vietnamese and English10 min readLREC 2026Catching the English words in Vietnamese medical speech11 min readAll publications · Get in touch about this research