English: A vs An

In English, there is an “indefinite” article a that can go before a word. For example, a raccoon. But for some words, we use an. For example, an apple. The rule we use is words that start with a vowel^[1] get an.

When procedurally generating text, I want a function a_or_an("apple") that tells me which article to use. That seems like it’d be easy. We check the first letter to see if it’s a vowel. But the rule is applied to the spoken word, not the written word. For example, we say a unicorn, not an unicorn, because the word <unicorn> starts with a consonant sound (“yoo-ni-korn”).

To implement the correct first-sound rule we need a pronunciation dictionary. I wondered how close the incorrect first-letter rule was, so I downloaded cmudict (https://github.com/cmusphinx/cmudict^[2]) and tried testing the first-sound vs first-letter rule. It produced tons of mismatches. I made a list of them, and decided to not include some of them in my analysis:

1. Proper names (3133). The trouble with names is that many of them are not English words and therefore don’t have to follow rules of English pronunciation. It’s not only names of people, but words like <nvidia> which starts with a vowel sound, but there are no normal English words that start with <nv>. Although these can show up in a sentence (an nvidia employee), I decided they’re not interesting to study. They can be put in an exception list.
2. Single letters (26). There’s a way to pronounce each letter of the alphabet, such as <h> sounding like “aitch”. Although these can show up in a sentence (an H was on his shirt), I decided they’re best handled as a hard-coded list.
3. Secondary pronunciations (9114). There are many words with multiple pronunciations, and cmudict lists each one. I looked at which ones might make a difference for a vs an, and the main one is that <herb> is pronounced differently in American and British English.
4. Punctuation (8574). These are primarily possessive (<arm's reach>) and contractions (<don't>), and although I could have handled them, I wanted to use a simple data structure indexed by 26 letters, so I skipped them.
5. Initialisms (10). In cmudict these are spelled out letter by letter. For example, <eu> is spelled out “ee yoo” (cmudict IY UW, ipa /i u/). There aren’t many of these, and I think they’re similar to proper names in some ways. Initialisms like <MIT> are different from acronyms like <NASA>.
6. Manual exclusions (2). <nth> and <urman>.
7. Other (81852). These are words in cmudict that weren’t in the BSD dictionary, for a mix of reasons. Some are names like <aesop>. Some are word variants like <lobsters>. Some are abbreviations like <jr>. I didn’t find an easy way to further separate these out so I excluded them all. However if I were going to update this project, I would take a closer look at these words.

I was left with 32455 words. I made a visualization using d3/ObservableHQ^[3] of the first two letters. If they are always a consonant sound, the line is black. If they are always a vowel sound, the line is blue. If they are sometimes vowel and sometimes consonant, the line is red:

There aren’t many exceptions:

eucalyptus euchre euchred eugenic eugenics eulogize eulogy eunuch euphemism euphemistic euphemistically euphony euphoria euphoric euphory eureka europium euthanasia ewe ewer heir heiress heirloom herb herbaceous herbal herbalist homage honest honestly honesty honor honorable honorably honoraria honorarium honorary honoree honorific hour hourglass hourly once one oneness oneself onetime ubiquitous ubiquity ukase ukulele ula unanimity unanimous unanimously unicellular unicorn unicycle unification unified uniform uniformed uniformity uniformly unify unilateral unilateralism unilaterally union unionism unionist unionization unionize unique uniquely uniqueness unison unit unitarian unitary unite united uniting unity universal universality universally universe university ural uranium ure urea uremia urethane urethra uric urinalysis urinary urinate urine urologist urology usable usage use used useful usefully usefulness useless user usual usually usurp usurpation usurping usury utensil uterine uterus utilitarian utility utilization utilize utopia utopian uva yttrium

I tried making a visualization showing where the exceptions occur:

However neither visualization was great. I was hoping to come up with a clever algorithm that reduces this down to a minimum decision tree. I think that’d be related to DFA minimization^[4]. However, I didn’t figure that out, so I wrote a set of rules manually:

VOWELS = "aeiou"

def starts_with_vowel(word):
    if word.startswith("eu"): return False
    if word.startswith("ew"): return False
    if word.startswith("heir"): return True
    if word.startswith("herb") and not word.startswith("herbiv") and word != "herbarium" and word != "herbicide": return True
    if word.startswith("homage"): return True
    if word.startswith("hones"): return True # see note
    if word.startswith("hono"): return True
    if word.startswith("hour"): return True
    if word.startswith("once"): return False
    if word.startswith("one") and not word.startswith("onerous"): return False
    if word.startswith("ula"): return False
    if word.startswith("unani"): return False
    if word.startswith("uni") and word[3:4] in "cfloqstv" and word != "unissued": return False
    if word.startswith("ub"): return False
    if word.startswith("uk"): return False
    if word.startswith("ur") and word[2:3] in VOWELS and word != "ur": return False
    if word.startswith("us") and word not in ("us", "usher"): return False
    if word.startswith("ut") and not (word.startswith("utm") or word.startswith("utt")): return False
    if word.startswith("uva"): return False
    if word.startswith("ytt"): return True
    return word[0] in VOWELS

# NOTE: the word "hones" itself should return False, but it's not in
# the /usr/share/dict/words file I was working with. There may be
# other words missing from my word list.

I didn’t try to deal with the old <an hospital> exception. My understanding is that long ago, some dialects of English didn’t pronounce the <h>, so they used an, and that got used in books and newspapers. When people speaking other dialects read those books and newspapers, they had to learn <an hospital> as an exception to the rule. Now that people pronounce the <h>, the exception rule is slowly going away.

Related: Antoine Amarilli (a3nm) worked on a similar problem, of identifying aspirated <h> in French^[5], and also built and visualized tries and then generated Python code.

Tangentially related: https://en.wikipedia.org/wiki/Rebracketing^[6] — before most people could read, people heard <a napron> and misinterpreted it as <an apron>, and people heard <an ewt> and misinterpreted it as <a newt>. We say <an orange> but in Spanish it’s <naranja> because the word originally had a leading <n>.

Future work: I’d like to figure out a better way to compress the trie. I was thinking of using an ordered decision tree instead of an unordered trie, similar to how PEGs are ordered and CFGs are not. Part of what makes this interesting is that the decision tree extends to words that aren’t in the set, and I want to study whether it’s better to use the shortest or longest pattern in the decision tree. I had given myself a time limit of 1 day for this project, and can try this in the future if I give myself another day.

Future work: it might be nice to take into account whether a word is commonly used or not, using the data from Peter Norvig^[7].

Source: preprocess-cmudict.py