Picking 100 million words that stand for Vietnamese

A pile of text is not a corpus

Language technology learns from examples. To find how Vietnamese is really used, which words are common or how sentences are built, researchers count things in a corpus. We follow Sinclair: a corpus is a set of text samples chosen systematically, by stated criteria, so that it stands for a particular kind of language.

The key words are “chosen” and “stands for”. A collection scraped from one news website is large, but it speaks only like that website. A general corpus has to be representative and balanced.

Think of it asAn opinion poll. A poll of 1,000 people can describe a country only if those 1,000 are picked to match it: regions, ages, jobs. A corpus is a poll of a language’s texts.

The model for this kind of design is the British National Corpus (BNC): 100 million words of late-20th-century British English, written and spoken, built from 1991 by Oxford University Press, Longman and partners.

Vietnamese had big collections, but none built to be balanced

The paper lists five earlier Vietnamese corpora. Some are huge; Le and Quasthoff’s holds 4.05 billion words. But their text comes mostly from online newspapers and the web, they count size in different units, and none describes itself as a balanced, general corpus or includes speech.

Seven decisions to make before collecting a single word

The core of the paper is a checklist. It gathers advice from corpus linguistics (Sinclair, Biber, Atkins and colleagues, Wynne’s guide) into seven criteria, and says how each was applied to VnC 1.0.

• 1Text selection
• 2Size
• 3Representativeness
• 4Balance
• 5Topic
• 6Homogeneity
• 7External vs internal

Start with a few simple, feasible dimensions

First come general criteria that describe any text: its mode (written or spoken), genre, domain, where it comes from, its language, and when it was made. The advice is to start simple, drop criteria that overlap, and check that each one can be collected in practice. Spoken text, for example, is only possible if someone can record and transcribe it.

Each criterion splits the corpus into components, and every component needs enough text of its own. That is where size comes in.

Representativeness is a six-step plan, and the last step matters most

No corpus can contain a whole language, so “representative” has to be planned. The paper adopts a six-step procedure from Wynne’s guide to good practice.

Balance means more than mixing genres

Balance is even vaguer than representativeness, and the paper notes that many general corpora fall short simply because they hold too little speech. Beyond mode and genre, it names four axes to keep in proportion:

Balanced does not mean equal shares

A common misreading is that a balanced corpus gives every category the same percentage. The paper does not define balance that way. It asks that balance be checked across several factors: who writes and who reads, content, time of production and region (§III.A.4, citing Wynne [14]). Its representativeness plan then sets a target size for each text type from that type’s importance and how practical it is to collect (§III.A.3, step 4 [14]). Shares produced by these rules are not equal. The equal-share comparison below is our own illustration.

Topic is hard to fix in advance. We suggest starting from common topic lists and adjusting to what can be collected. For academic writing in VnC 1.0 we aimed at humanities, medicine, natural sciences, education, law, politics, social sciences, engineering, informatics and technology.

Homogeneity: a stack of score tables can skew a whole genre

Within a genre, pick typical texts and avoid odd ones. We give an example from our own collecting: some sports-news pages list only football scores. Collect too many of those, and a study of “sports news” would conclude that number words are its most frequent nouns, instead of words like trận đấu (match), trọng tài (referee) and vận động viên (athlete).

Choose by the label on the cover, not by the words inside

External criteria describe a text’s situation and can be judged without reading it: the author’s gender, age or job, the context. Internal criteria are about the language itself: vocabulary, syntax. Choosing only by internal features loses the link between language and context; choosing only by external ones can hide linguistic differences. The paper’s advice for a new corpus is to select mainly by external criteria, then add internal ones later. VnC 1.0 does exactly that.

From a checklist to a hundred million words

With the criteria set, the team picked the simple, feasible ones and collected. The plan: about 90 million written and 10 million spoken words, from Vietnamese speakers in all three regions, with about 90% of the text from 2000 onwards.

The plan held. Adding up the two tables gives 89,790,952 written and 10,130,306 spoken words, so speech makes up 10.14% of the 99,921,258 words counted. The paper does not give a word count by period, so the 90% post-2000 target cannot be checked from the tables.

Fiction and unlabelled text fill most of the written side

The written part is split into 31 groups by genre and domain. Before looking, make a guess.

Two groups dominate. Novels and short stories give 22.66% of written words from just 618 texts, long works sampled in part. “Other texts”, a catch-all not assigned to a specific genre, gives 38.84% of words and almost two-thirds of all written texts. Specialised documents in medicine, natural science or technology are each under 0.4%.

Speech: plenty from the North and South, almost none from the Centre

Speech was transcribed from real video and audio. To keep dialect features intact, each region’s recordings went to transcribers from that region. We name the Central region as our difficulty: it is the smallest spoken group by far.

Spoken sentences are short

The tables also let us compare sentence length, something the paper does not discuss. Dividing words by sentences, transcribed speech averages 6.3 words per sentence and written text 16.2.

What VnC 1.0 still lacks

We are direct about the gaps. Some written genres were hard to collect in practice: local newspapers, magazines, secondary-school students’ essays, personal and business letters. Speech covers fewer situations than they want. Our next steps are to keep updating the existing genres and to add the missing ones.

Four takeaways

1. A corpus is a designed sample, not a pile. The paper sets out seven criteria for choosing texts: text selection, size, representativeness, balance, topic, homogeneity, and external versus internal criteria, and recommends starting with simple, external, feasible ones. Balance means shares set on purpose and recorded, not equal ones: VnC plans 90% written and 10% spoken (§IV).
2. VnC 1.0 hits its planned shape. 89,790,952 written and 10,130,306 spoken words, 146,052 texts in 37 groups; speech is 10.14% of the total, against a 10% target.
3. Two groups dominate the written side. “Other texts” holds 38.84% of written words and novels and short stories 22.66%; the next largest, newspapers’ “other material”, has 6.36%.
4. Speech is the hardest part to balance. Central Vietnam has 13 transcribed texts and 6,930 words, against 2,542,896 words from the South and 1,458,930 from the North.

We plan to add spoken discussion programmes, counselling, documentaries, news, lectures, debates and meetings, and written local newspapers, magazines, school essays and letters, to raise the corpus’s balance, size and representativeness.