Read next
EMNLP 2026Teaching machines to read medicine in Vietnamese13 min read
LREC 2026Catching the English words in Vietnamese medical speech11 min readAll publications · Get in touch about this research
LREC-COLING 2024 · Interactive explainer
Almost all of Vietnam’s medical universities and hospitals teach and work in Vietnamese, while most specialist material for students, doctors and nurses is in English. Translation models had little medical data to learn from. MedEV adds 358,796 sentence pairs, and a model fine-tuned on it beats Google Translate.
What does a medical translator have to do?
A real sentence pair from the MedEV test set. Tap a word to see its partner.
The model reads the top line and must write the bottom one. Word order flips: Vietnamese says “concentrations dopamine plasma” and “patients schizophrenic”. Word links are drawn for this explainer; this pair follows us through the page.
A model learns from : the same sentence written in both languages. Two large, high-quality Vietnamese–English datasets were public before this work. PhoMT has no medical sentence pairs. MTet has 13,410. The COVID-19 Health Wikipedia set adds 4,273 pairs of COVID-19 news.
That is too little to train a strong medical translator. MedEV was built to close the gap.
Think of it asA translation model is an interpreter, and parallel text is its bilingual textbook: the same page in English and in Vietnamese, side by side. Until now, the medical chapters were missing.
How much medical parallel text existed before MedEV?
Each square is about 1,000 sentence pairs, rounded to the nearest thousand.
358,796 pairs
13,410 pairs
4,273 pairs
no medical pairs
Very little: MedEV alone has about 20 times the medical pairs of the two earlier sources combined.
MedEV starts from documents that already exist in both languages. The work is finding which English sentence matches which Vietnamese sentence, called , then throwing out the pairs that don’t really match.
How do 27,218 documents become 358,796 sentence pairs?
Pick a step, or use the buttons below.
Most sentences survive: about 390K aligned pairs shrink to 358,885 after filtering, and hand checks remove only 89 more, leaving 358,796.
The sources differ in who wrote them and how technical they are. Article abstracts from Vietnamese medical journals make up more than half the pairs. The MSD Manuals, a medical reference written by doctors, are the second largest source.
Where do the sentence pairs come from?
Bar width is proportional to sentence pairs. Pick a genre for its details.
Journal abstracts supply 54.7% of the pairs. Average sentence length in the training split. English is counted in words, Vietnamese in syllables, which is why Vietnamese sentences look longer.
The study compares tools as they come, without any training on MedEV: Google Translate, ChatGPT (gpt-3.5-turbo), two massively multilingual models (SeamlessM4T and M2M100), and two Vietnamese–English translation models (vinai-translate and envit5-translation).
Then four models are on MedEV’s training split. Two are the translation models above. The other two are the general pre-trained models those were built from, mBART and envit5-base. This shows how much a model gains from MedEV when it starts with, or without, general translation training.
Think of it asFine-tuning is sending a general interpreter on a course in medical language, using MedEV as the course book.
Where do the fine-tuned models come from?
BLEU on the MedEV test set. Higher is better.
MedEV lifts vinai-translate from 44.60 to 52.14 BLEU (English to Vietnamese). Fine-tuning mBART directly on MedEV gets close, at 51.18.
Translations are scored with . Before the scoreboard, try the score on our example sentence.
What does a BLEU score count?
Translate the Vietnamese sentence into English, or pick a version. BLEU counts how many of your words and word sequences also appear in the reference. Higher is better.
VIETNAMESE SOURCENghiên cứu đặc điểm lâm sàng và nồng độ dopamin huyết tương ở bệnh nhân tâm thần phân liệt
ENGLISH REFERENCE, FROM MEDEVResearch for clinical characteristics and plasma dopamine concentrations in schizophrenic patients
A correct translation in different words scores low. That is why short sentences, where one synonym is a large share of the words, get lower BLEU (Part 5). The “different wording” version is written for this explainer. The score here is sentence-level BLEU computed in your browser; the paper’s scores are pooled over the whole test set.
Modified n-gram precision for n = 1–4 with clipped counts, geometric mean, brevity penalty exp(1 − r/c) when the translation is shorter than the reference, and exponential smoothing for orders with zero matches; orders with no n-grams are skipped (effective order). Punctuation is split off and matching is case-sensitive, as in SacreBLEU. The paper reports case-sensitive corpus BLEU from SacreBLEU.
Make a guess. Google Translate scores 47.86 BLEU translating English medical sentences into Vietnamese. Where does ChatGPT land, with no examples in its prompt?
Now that you know what BLEU counts: MedEV test set, English to Vietnamese. Higher is better.
Among the tools used as they come, Google Translate scores best on every metric, in both directions. The authors suggest it may have been trained on some medical parallel data. The two Vietnamese–English models come next, ChatGPT is lower in most cases, and SeamlessM4T and M2M100 trail far behind.
After fine-tuning on MedEV, all four models pass Google Translate, on both the validation and test sets and in both directions. vinai-translate fine-tuned on MedEV is best overall.
Which system translates medical text best?
Higher is better. Dashed line: Google Translate.
vinai-translate fine-tuned on MedEV, on every metric and in both directions. counts the edits needed to turn the output into the reference, so lower is better. also credits matching word stems and synonyms, on a 0–1 scale.
ChatGPT was asked to translate with 0, 1, 8, 16 or 32 example pairs drawn at random from the training set and placed in its prompt (). Scores generally rise with more examples, but even the best setting stays more than 11 BLEU points behind Google Translate in English to Vietnamese.
Do more examples in the prompt help ChatGPT?
Test-set BLEU, higher is better. Pick a point or a setting to see the prompt the paper used.
A little: from 34.45 with none to 36.37 with 32 examples (English to Vietnamese), still far below Google Translate and the fine-tuned model.
The authors fine-tuned vinai-translate on growing slices of MedEV’s training set. Slide to see how quickly it learns, and when it passes Google Translate.
How much medical data does it take to beat Google Translate?
vinai-translate fine-tuned on growing slices of MedEV. Validation-set BLEU, higher is better. Dashed lines: Google Translate on the same validation set.
Not much: 10K pairs are enough from English to Vietnamese, 20K from Vietnamese to English.
The first 10,000 pairs do the most: about 4 BLEU points in each direction, from 44.24 to 48.23 and from 33.28 to 37.97. The remaining 330,000 or so pairs add about 4 more, reaching 52.21 and 42.66. A small medical dataset already helps a lot, and a large one keeps helping.
The MSD Manuals, written by doctors about common diseases classified under ICD-10, get the highest BLEU. Article abstracts, which include titles and keywords, carry the most medical terminology and get the lowest.
Which kinds of medical text are hardest to translate?
Test set. Darker means a higher score. Outlined: highest in each column.
Article abstracts score lowest for nearly every system. The MSD Manuals score highest for Google Translate and all four fine-tuned models.
Fine-tuned models have the highest score in six of the eight genre columns. Google Translate still leads the other two: Article Translations from English to Vietnamese (52.79) and Thesis Summaries from Vietnamese to English (44.82).
BLEU generally rises with sentence length. The authors explain that a longer sentence gives more chances for common words to match the reference. In a short sentence, a correct synonym or alternative medical term that differs from the reference costs a large share of the score.
Do short sentences get lower scores?
BLEU by reference sentence length, test set, higher is better. Length counts words including punctuation. Grey lines: every other system.
Mostly yes, and most sharply for vinai-translate before fine-tuning. Our example’s English reference has 11 words, so from Vietnamese to English it falls in the 10–19 bucket, which holds 39.85% of test sentences.
The short-sentence effect is sharpest for vinai-translate before fine-tuning. Translating English to Vietnamese, it scores 31.53 on sentences under 10 words and 43.07 on the next bucket. After fine-tuning on MedEV, the same bucket scores 48.64.
Next, the authors plan to combine MedEV with the general-domain PhoMT and MTet datasets.
Read next
EMNLP 2026Teaching machines to read medicine in Vietnamese13 min readLREC 2026Catching the English words in Vietnamese medical speech11 min readAll publications · Get in touch about this research