Vietnamese Dictionary & Encyclopedia 2020 · Interactive explainer
A dictionary is only as good as the evidence behind it. This paper sets out how to build a Vietnamese text collection that fairly stands for the language, then shows what its first version, 100 million words, gives back to the people who write dictionaries.
Which example should go in the dictionary entry for “gió chướng”?
Pick one of the three. The entry card shows what a reader would learn from it.
Candidate examples
Dictionary entry
The made-up sentence is the paper’s own counter-example (Section 5.2). The two others are real sentences from the VnC 1.0 corpus (Figure 1 of the paper). English glosses are ours; they translate “gió chướng” as headwind and “mùa gió chướng” as headwind season, following the ZIM Vietnamese–English Dictionary (“In October, the headwind season keeps small boats from sailing.”). “gió chướng” is the running example on this page.
A dictionary writer needs examples, definitions and a sense of which words matter. One way is to trust their own feeling for the language. The other is to look it up in a : a large, planned collection of real texts that a computer can search. This paper is about the second way, for Vietnamese.
Think of it asA dictionary writer working from a library of everything people actually wrote and said, instead of from memory.
John Sinclair, who started corpus-based , put the contrast plainly: in dictionary-making there is “a marked contrast between the data collected by computer and that collected by human readers exercising their judgment on what should or should not be selected.”
Working from a corpus makes a dictionary more objective, gives the writer evidence for every claim about a word, and makes it faster to add new words. English took this route decades ago. In Vietnam the approach was still rare in 2020, because few corpora met the requirements of a “living source” for dictionary work.
When did dictionaries start being built from corpora?
Tap a milestone. Blue = English, grey = other languages, red = Vietnamese.
English led from 1987; German and Portuguese followed. Vietnamese work began with resources and tools in 2011 and 2013. The milestones are the ones the paper cites in Section 3.
Following Wolfgang Teubert, the paper treats language as a social fact that shows up in texts. Ideally you would collect every text a language community has ever written or said. That is impossible, so you choose criteria (time, domain, text type…), sample texts for each at a sensible size, and aim for a corpus that is and .
Think of it asAn opinion poll. You cannot ask everyone, so you design who to ask, and how many of each kind, before the first call.
What are the steps from an idea to a usable corpus?
Pick a step, or use the buttons. The red line follows “gió chướng” through each one.
Sections 4.1–4.4 and 5 of the paper. Planning and sampling get the most space, because they decide whether the corpus can be trusted at all.
One planning decision is easy to get wrong: how long each text sample should be. The paper warns in both directions.
How long should each text sample be?
Choose a sample size. Dashed outlines show how much text of each kind can actually be collected. Illustrative amounts
The rule (Section 4.1): too small and a sample cannot show how the language behaves; too large and the kinds of text cannot be kept in balance, because each kind can be collected in different amounts. Printed works also raise copyright questions.
Collecting is always a fight with cost and time. Texts come from four places: files that are already digital, printed pages retyped or scanned with , transcribed recordings, or data bought in form. Whatever the route, what goes in is decided by sampling criteria.
The paper separates two kinds. can be judged without reading a word of the text. are about the language inside it. Try sorting them.
Can you tell a criterion about the author from one about the language?
Tap each card to mark it External or Internal, then check.
The paper’s advice: when starting out, select samples mainly by external criteria. They are cheap to check and do not assume in advance what the language looks like.
A community does not only write. It also reads, listens and speaks. The paper asks for samples of both what people take in (reception) and what they put out (production), and sets different rules for written and spoken texts.
What counts as a text worth sampling?
Switch between written and spoken texts.
Section 4.2.3. Pick the typical text of a genre, not an unusual one: one fashion magazine cannot stand for all magazines.
Before sampling, the paper lists six general criteria to settle. Each one splits the language into more cells to fill, which is why it recommends starting with simple criteria and checking that texts for each cell can really be found.
How fast do sampling criteria multiply the work?
Set each general criterion to describe one text. The counter shows how many cells these options create. Illustrative options
The six criteria and the 1945 and 1975 milestones come from Section 4.2.1; the option lists are our simplification. VnC 1.0’s spoken part is contemporary, mostly from 2000 on (Section 5).
A corpus is when what you find in it holds for the language beyond it. The paper gives six steps for getting there, and calls the last, plain record-keeping, the most important.
How do you make a sample stand for a whole language?
Step through the six steps of Section 4.2.4.
Steps 1–4 design the corpus, step 5 keeps it on plan, and step 6 lets users explain surprising results later.
is an even vaguer idea than representativeness. The paper notes that most popular corpora are unbalanced simply because they lack enough spoken language.
How is VnC 1.0 split between writing and speech?
Each square is 1 million words. Toggle a part, or tap a balance factor to read what it protects.
Balance depends on
Split from Section 5; the six factors from Section 4.2.5. The 10% spoken share is the part most corpora lack.
Section 4.2.6: the minimum size depends on (1) the queries users are expected to run, now or later (e.g. word-class lists sorted by frequency), and (2) the methods they will use to study the material. There is no maximum. The larger the corpus, the less its statistics deviate from reality; material should also be added over time for new research needs.
Collected text is messy. It has to be cleaned, marked up and annotated before a dictionary writer can search it. Vietnamese adds one step English barely needs: spaces separate syllables, not words, so gió chướng is one word written as two. joins them.
Think of it asPreparing a library: throw out the torn pages, put labels on the shelves, then index every word in every book.
What does one sentence look like after each stage?
Step through the layers of a real VnC 1.0 sentence.
The word boundaries are exactly as printed in the paper’s Figure 1. The raw noise, the markup tags and the part-of-speech and entity labels are our illustrations: the paper says VnC 1.0 carries word boundaries, parts of speech and entities, but does not print the tags.
The basic tools for mining a corpus are a word-frequency list, a , part-of-speech counts and . The paper uses them for three jobs: finding real examples, gathering every context of a word, and writing definitions in words simpler than the headword.
Here is the concordance the paper prints for our running example. Each row is a real sentence, cut so that the keyword lines up in the middle.
What do real sentences reveal about “gió chướng”?
Tap a row for its English gloss. Sort by the word before or after, or filter by what the line tells you.
The 8 lines of the paper’s Figure 1. Lines 3 and 7 are the two the paper quotes as better examples (Section 5.2). The “hints at” tags and English glosses are our reading, not the paper’s; the glosses translate “gió chướng” as headwind (“mùa gió chướng” = headwind season), following the ZIM Vietnamese–English Dictionary.
Eight lines are only a sample. Searching the written part of the corpus found every sentence that uses the word, so a writer can see all of its uses before choosing.
How do 100 million words narrow down to two examples?
Step through the search.
Numbers from Sections 5, 5.2 and 5.3 and Figure 1. From these 109 sentences a writer can pick uses of “gió chướng” about the climate and life of southern Vietnam, mainly the Mekong Delta.
A corpus also yields a : how often each word and each use of it appears, overall, in speech versus writing, and by domain. The paper uses it two ways: include the common words, and list each word’s senses from most to least frequent. It also catches a habit of Vietnamese explanatory dictionaries: defining an everyday word with harder ones.
Is this definition harder than the word it defines?
The entry for “đường” (sugar) in a Vietnamese dictionary. Check it against corpus frequency.
Section 5.1. The paper states the two words are “certainly” much less frequent than “đường” but gives no counts, so none are shown here.
Read next
FAIR 2020Picking 100 million words that stand for Vietnamese14 min read
Linguistic Journal 2022Teaching real Vietnamese to foreigners with a 100-million-word corpus14 min readAll publications · Get in touch about this research