How do you find English medical terms in Vietnamese speech?
Pick a step, or use the buttons below.
Each step narrows the pool: 3,203 terms searched, 13,000+ videos, 700+ hours transcribed, 16,576 clips kept.
For researchers: the transcription prompt
Gemini 2.5 Pro was given this instruction (from the paper):
Transcription Task: You are an advanced transcription assistant for Vietnamese medical audio. Do the following:
1) Transcribe the Vietnamese speech.
2) Segment into sentences of approximately 5–15 seconds.
3) For each segment, output start_time, end_time, and text.
4) Detect segments that contain any non-Vietnamese terms (e.g., English, Chinese, technical product/brand names).
5) Return the complete set of segments, and separately the subset of code-switch segments. For each code-switch segment, list the non-Vietnamese terms that appear.
All output must be a single valid JSON object with keys:
• "segments": an array of all segment objects {start_time, end_time, text}.
• "code_switch_segments": an array of code-switch segment objects {start_time, end_time, text, cs_term: [...]}.
Return only JSON; do not include any additional prose.
Because spans returned by the LLM may not exactly match dictionary entries, the Levenshtein distance between each dictionary item and each sentence is computed, and the closest canonical entry is assigned to the detected span. Surface forms are normalized (orthography, hyphenation, casing, common variants) to keep term identity consistent.
How much of the audio made it into the dataset?
Each square is one hour of audio. Over 700 hours were transcribed; about 5% survived the filters as valid code-switched medical speech.
About 5%: 34.6 of the 700+ hours transcribed.
Every clip contains at least one code-switched term. Some contain several:
To check quality, two annotators independently reviewed 500 utterances, about one hour sampled across all five topics. They agreed substantially ( = 0.65). Most disagreements came from clip boundaries that started or ended a little early, so we added an automatic boundary-refinement step to the pipeline.
A long tail of rare terms
Medical vocabulary is lopsided. A few terms come up constantly, and many appear only once. The dataset keeps that shape on purpose.
How often does each medical term come up?
Each dot is one distinct term. Pick a group.
889 distinct terms. Almost half appear five times or fewer; that long tail is what the Hard split tests.
The rarest terms get their own test. The holds only terms that occur once or twice in the whole collection, and every occurrence of them is kept out of training. A model can only get them right by generalizing, not by memorizing.
Which topics and splits does the audio cover?
Hours of audio by topic and by split.
Five topics
Four splits
Medical Sciences is nearly half of all audio. Right-hand column: utterances. For splits, the count of distinct code-switched terms is shown after the bar label. The Hard split shares no terms with Train, Dev or Test.
Who hears the English best?
We first tested seven ready-made systems , without any training on our data. Some are built for Vietnamese: wav2vec2-base-vi, PhoWhisper and VietASR. Others are : MMS and OpenAI’s Whisper, trained on dozens to over a thousand languages.
Make a guess. Which model makes the fewest errors on the English medical terms?
Scored by CS-WER on the test set, before any fine-tuning.
The model that is best on English terms is not the best overall. VietASR, trained for Vietnamese, has the lowest WER of all seven at 27.56, but a CS-WER of 58.38. On the test set, in each matched pair, the Vietnamese version lowers overall WER and the multilingual version does as well or better on the terms. The gap is widest for the largest models: Whisper-Large-v3 beats PhoWhisper-Large on terms by more than 8 points.
Can one model get both the sentence and the terms right?
Lower is better on both axes. Dashed lines join each multilingual model to its Vietnamese counterpart.
Out of the box, no: Vietnamese models sit further left, the best multilingual one lower. Fine-tuning moves both small models down and to the left on the test set.
For researchers: all zero-shot scores (Table 4)
Hints at decoding time, or teach the model?
There are two broad ways to help a model with specialist words. You can hand it a list of terms while it transcribes, which is called . Or you can it on code-switched speech. We tried both on one backbone, PhoWhisper-Small, using four families of methods.
Think of it asContextual biasing is handing the typist a list of tricky words before they start. Fine-tuning is sending the typist on a course in medical vocabulary.
Which fix works best: a term list or fine-tuning?
Eight methods on PhoWhisper-Small. Pick a metric and a split. Lower is better. Dashed line: the frozen model with no adaptation.
Fine-tuning wins. Attention Guide drops test CS-WER from 62.55 to 19.50; term lists at decoding time barely move it.
For researchers: how the methods were set up
- Backbone: PhoWhisper-small, chosen because Whisper-style models are the prevailing substrate for recent code-switching ASR and PhoWhisper is a Vietnamese-optimized one of practical size.
- For DV, RS and AdaCS, the bias list is built from the code-switched medical terms present in the split being decoded (train, test or hard).
- Attention Guide uses bilingual Vietnamese–English prompts to reflect the code-switching setting.
- Because AdaCS runs after decoding, it is also tested as a hybrid on top of LoRA and AG.
- The Whisper-Small vs PhoWhisper-Small comparison (Table 6) is in the chart above under “Add fine-tuned models”.
Handing the decoder a term list barely moved anything: Dynamic Vocabulary and Rank & Selection stay within 2.5 points of the frozen model on test CS-WER. AdaCS, which corrects terms after decoding, nearly halves CS-WER on the test set but pushes overall WER up from 36.31 to 39.40.
Fine-tuning with adapters such as improves every metric. does best. It finds the attention heads that track which language is being spoken and steers them during training. It brings test CS-WER from 62.55 to 19.50, and WER from 36.31 to 23.67. In our example sentence, CS-WER looks only at quercetin, prostan and leukotriene; at 19.50, roughly one term word in five is still wrong.
Turn on the fine-tuned models in the chart above to see the backbone question. Both Whisper-Small and PhoWhisper-Small improve sharply, and the Vietnamese-optimized PhoWhisper ends up ahead overall. Starting from Vietnamese and teaching the English terms works better than the reverse.
The rare terms are still hard
Switch either chart to the Hard set. The gains mostly disappear. Terms the model never heard in training stay difficult for every method.
No method in the study gets Hard-set CS-WER below 56. Recognizing a medical term the model has never heard remains an open problem, and the Hard split is there to measure progress on it.
Some topics are harder than others
Before adaptation, Treatments is the most error-prone topic. After it, the gap narrows. Nutrition is easiest throughout. Diagnostics barely moves with LoRA and stays the worst topic after adaptation.
Which medical topics stay hard after fine-tuning?
PhoWhisper-Small. Lower is better.
Diagnostics barely moves with LoRA and is the hardest topic after Attention Guide (44.40); Nutrition is easiest throughout.
Four takeaways
- A first benchmark for Vietnamese medical code-switching. 34.6 hours and 16,576 utterances, each with at least one English medical term, plus a Hard split of rare terms.
- A clear trade-off out of the box. Vietnamese-optimized models get the sentence right; multilingual models get the English terms right. Whisper-Large-v3 has the lowest zero-shot CS-WER, VietASR the lowest WER.
- Adapters beat term lists. Attention Guide on PhoWhisper-Small gives the best balance: test WER 23.67 and CS-WER 19.50, down from 36.31 and 62.55.
- Unseen terms are the open problem. On the Hard split, CS-WER stays above 56 for every method tested.
Notes and limitations
- Audio comes from public YouTube medical videos, is used for research only, and contains no private patient information.
- Transcripts were produced with Gemini 2.5 Pro and checked by people on a 500-utterance sample (κ = 0.65).
- Recordings come from public videos rather than clinical consultations.
- The misrecognized spellings in the opening figure and the WER game are illustrative; every score comes from the paper.
Contact: tung.nx@vinuni.edu.vn · nhu.vd@vinuni.edu.vn
Read next
LREC-COLING 2024Teaching machines to translate medical text between Vietnamese and English10 min read
EMNLP 2026Teaching machines to read medicine in Vietnamese13 min readAll publications · Get in touch about this research