Read next
PACLIC 34 · 2020Naming the writer of a Vietnamese text from its word habits13 min read
FAIR 2020Picking 100 million words that stand for Vietnamese14 min readAll publications · Get in touch about this research
NICS 2020 · Interactive explainer
Linguists have measured how stories differ from news, reports and essays in English, Mandarin, Russian and other languages. Vietnamese had very little of this work. We built two equal-sized Vietnamese corpora of 24 million words each, one of stories and one of everything else, and counted which words, word classes and grammatical features each one uses and which of them travel together.
Ten words, one corpus. Is it stories or not?
These are the ten most common words in one of our two 24-million-word corpora. Guess which one.
Mystery corpus
Non-narrative corpus (news, reports, academic…)
Normalized frequency as reported in the paper (Table II). Words tagged “not in other top 10” do not appear in the other corpus’s top ten.
That is the whole method in miniature. Take a large collection of real texts, a , count what is in it, and compare. When the counts are big enough, the style of a genre shows up as numbers: stories talk about tôi “I”, anh “he/you”, and what someone did; reports talk about các “the (plural)”, của “of” and trong “in”.
Think of it asTelling a novel from a bank report with your eyes closed, just from a tally sheet of the words someone read aloud to you.
In 1988 the linguist Douglas Biber proposed the : count dozens of grammatical features across many texts, see which ones rise and fall together, and name each group a of variation. Since then the method has been applied to English, Mandarin, Russian, Somali, Korean, Tuvaluan, Taiwanese and Spanish, and used for stylometry, genre classification and even spotting deceptive writing.
Thirty years of the method, and where Vietnamese stood
Tap a year to see one study the paper cites.
Prior studies as described in the paper’s introduction. Applying the method to Vietnamese has been held back because Vietnamese and English differ in ways that make building and tagging Vietnamese corpora hard.
Biber found five main dimensions in English. This paper looks at the second one, which he named “Narrative versus Non-Narrative Concerns”. Narrative texts in English pile up past-tense verbs, third-person pronouns, perfect-aspect verbs, , synthetic negation and present-participle clauses, and these features tend to appear together.
Not all of them make sense for Vietnamese. Vietnamese verbs do not change form, and negation is mostly a separate word, không “no(t)”, so there is neither synthetic nor analytic negation. We kept five features that fit Vietnamese grammar and that a computer could extract automatically from a tagged corpus.
Which of Biber’s features could we count in Vietnamese?
Tap a feature. Solid red = counted in this study; dashed = not used.
Biber (1988) used many more features. This study keeps five; the running example on this page is third-person pronouns. Example words in this figure are illustrative, not taken from the paper.
Both corpora come from the Vietnamese Corpus 1.0 (VnC), a collection of 100 million words of contemporary spoken and written Vietnamese. Every word was tagged automatically with its word boundaries and its by CLC_VN_Toolkit, a tool built at the Computational Linguistics Center, University of Science, VNU Ho Chi Minh City.
The narrative corpus holds novels, short stories, biographies, excerpts from modern drama scripts and poetry collections; 90% of the short stories and novels were published between 2000 and 2020, the rest between 1930 and 2000. The non-narrative corpus was crawled from the web: academic prose, administrative and regulatory texts, commerce, finance and economics, parliamentary proceedings (quochoi.vn), company annual reports, online newspapers (tuoitre.vn) and other non-academic texts, cleaned of comments, menus and adverts.
Same size in words, but how many texts did each take?
Pick a measure.
1,197 texts
41,163 texts
Each dot is about 100 texts. A novel can run to tens of thousands of words; a news story is a few hundred to a few thousand. To keep the two corpora equal in words, the narrative side needed fewer than 1,200 texts and the non-narrative side over 41,000. Source: Table I.
With the corpora in place, the study follows the five steps of Biber’s method (the paper’s Figure 1). Follow third-person pronouns through each step.
How does a pile of text become a table of correlations?
Pick a step, or use the buttons below.
The method of the paper’s Figure 1: corpus → feature selection → frequency extraction → normalization → correlation. The statistics were run in Python.
A 20,000-word novel will contain more of almost everything than a 500-word news item, simply because it is longer. Comparing raw counts would mostly measure length. So every count is turned into a rate, the : how many times the feature appears per 1,000 words of that text.
Can a short text use a feature as often as a long one?
Illustrative texts. Move the sliders and compare raw counts with the rate per 1,000 words.
Formula (1) in the paper: nf = (f ÷ l) × 1,000, with f the count and l the text length. For scale, the real corpus means are 12.09 (narrative) and 3.60 (non-narrative) third-person pronouns per 1,000 words (Table V).
Before looking at grammar features, we simply counted words. The ten most frequent words already split the two genres.
Which words top each corpus?
Tap a word to see where it ranks on the other side.
Narrative corpus
Non-narrative corpus
Tap any word.
Words tagged “not in other top 10” do not appear in the other corpus’s top ten. tôi “I” is number one in stories and absent from the non-narrative top ten. Source: Table II (normalized frequency).
Tags from CLC_VN_Toolkit: Pp pronoun, R adjunct, Nq quantifier, Vc copula, Cm/Cp conjunctions, Ve existential verb, Nn noun. Glosses for là, có, anh, đã, lại, trong are ours.
The non-narrative list is built from the glue of formal writing: the conjunction và “and” is first, của “of” second, and three , các, một and những, all make the top ten. The stories list keeps only một. Grouping words by part of speech makes the contrast larger.
Which word classes do stories use far more?
Pick a word class. Bars are raw counts in each corpus (both about 24 million words).
Source: Table III. Because the two corpora are about the same size in words, raw counts can be compared directly here.
Many Vietnamese words can act as more than one word class without changing form: tuyển can be the verb “recruit” or a noun. Which reading wins depends on the genre.
Think of it asThe English word “run”: in a sports story it is mostly something people do; in a factory report it is mostly a noun, “a production run”.
Does the same word change its job between genres?
Pick a word. Each bar shows how its uses split across word classes in each corpus.
Noun/verb words such as tuyển, qui định, cấu tạo and biến đổi lean towards verbs in stories and nouns elsewhere. English glosses are the paper’s where it gives one. Source: Table IV (%).
Now the five Dimension 2 features. For each text we computed the rate per 1,000 words, then summarised every feature across all the texts of a corpus with four numbers: the mean, the minimum and maximum, the range, and the .
How often does each feature appear per 1,000 words?
Pick a feature. The thick tick is the mean, the shaded band is mean ± one standard deviation (cut at 0), the thin tick is the highest text.
Every minimum is 0, so each range equals the maximum. Values exactly as printed; the standard deviations for present tense and public verbs are printed identically (2.26 and 3.35).
Third-person pronouns show the clearest gap: a mean of 12.09 per 1,000 words in stories against 3.60 elsewhere. Stories also vary much more among themselves (standard deviation 12.41, about twice the non-narrative 5.86), yet the paper finds no significant difference between the highest values on the two sides. Public verbs such as “say” and “tell” are about one and a half times as common in stories on average, but the most extreme non-narrative text uses three times as many as the most extreme story.
Averages say how often a feature appears. Biber’s idea is about what appears together. For every pair of features we computed a across the texts of a corpus. A positive value means texts with more of one tend to have more of the other; a negative value means one tends to show up where the other is scarce; −1 would mean they never share a text, which Biber calls complementary distribution.
Think of it asUmbrellas and raincoats: on days you see many of one you see many of the other (positive). Umbrellas and sunglasses mostly come out on different days (negative).
Which pairs of features rise and fall together?
Tap any cell to compare the same pair in both corpora. Cells involving adjectives are outlined.
Source: Figure 2 (Pearson correlations between the five features, per corpus). The largest value off the diagonal is 0.21, so these are weak tendencies, not rules.
Formula (2): rxy = Σ(xi − x̄)(yi − ȳ) / √(Σ(xi − x̄)² Σ(yi − ȳ)²), where xi, yi are the normalized frequencies of the two features in text i, and x̄, ȳ their means.
In the narrative corpus every adjective cell is negative, while the other four features all correlate positively with each other; the strongest pairs are third-person pronouns with public verbs (0.21), third-person pronouns with present tense (0.17), and public verbs with past tense (0.13). In the non-narrative corpus the picture flips for our running example: adjectives and third-person pronouns show the highest correlation in the matrix, 0.15, tied with third-person pronouns and public verbs.
Biber reported the same adjective pairs for English. In both languages, every pair is negative: in stories, texts rich in adjectives tend to have fewer third-person pronouns, public verbs, past-tense and present-tense forms.
Is the adjective pattern the same in English and Vietnamese?
Correlation of adjectives with each feature. Add the non-narrative corpus to see the contrast.
Source: Table VI; non-narrative values from Figure 2. The direction matches in every pair; the Vietnamese values are all closer to zero than the English ones.
The authors plan to extend the MD approach to corpora with both spoken and written Vietnamese, for comparative Vietnamese–English linguistics and practical language-processing tasks.
Read next
PACLIC 34 · 2020Naming the writer of a Vietnamese text from its word habits13 min readFAIR 2020Picking 100 million words that stand for Vietnamese14 min readAll publications · Get in touch about this research