English: A vs An

In English, there is an “indefinite” article a that can go before a word. For example, a raccoon. But for some words, we use an. For example, an apple. The rule we use is words that start with a vowel^[1] get an.

When procedurally generating text, I want a function a_or_an("apple") that tells me which article to use. That seems like it’d be easy. We check the first letter to see if it’s a vowel. But the rule is applied to the spoken word, not the written word. For example, we say a unicorn, not an unicorn, because the word unicorn starts with a Y sound.

To implement the correct first-sound rule we need a pronunciation dictionary. I wondered how close the incorrect first-letter rule was, so I downloaded cmudict (https://github.com/cmusphinx/cmudict^[2]) and tried testing the first-sound vs first-letter rule. It produced tons of mismatches. I made a list of them, and decided to not include some of them in my analysis:

1. Proper names (3133). The trouble with names is that many of them are not English words and therefore don’t have to follow rules of English pronunciation. It’s not only names of people, but words like nvidia which starts with a vowel sound, but there are no normal English words that start with nv. Although these can show up in a sentence (an nvidia employee), I decided they’re not interesting to study. They can be put in an exception list.
2. Single letters (26). There’s a way to pronounce each letter of the alphabet, such as h sounding like aitch. Although these can show up in a sentence (an H was on his shirt), I decided they’re best handled as a hard-coded list.
3. Secondary pronunciations (9114). There are many words with multiple pronunciations, and cmudict lists each one. I looked at which ones might make a difference for a vs an, and the main one is that herb is pronounced differently in American and British English.
4. Punctuation (8574). These are primarily possessive (arm's reach) and contractions (don't), and although I could have handled them, I wanted to use a simple data structure indexed by 26 letters, so I skipped them.
5. Initialisms (10). In cmudict these are spelled out letter by letter. For example, eu is ee yoo. There aren’t many of these, and I think they’re similar to proper names in some ways.
6. Manual exclusions (2). nth and urman.
7. Other (81852). These are words in cmudict that weren’t in the BSD dictionary, for a mix of reasons. Some are names like aesop. Some are word variants like lobsters. Some are abbreviations like jr. I didn’t find an easy way to further separate these out so I excluded them all. However if I were going to update this project, I would take a closer look at these words.

I was left with 32455 words. I made a visualization using d3/ObservableHQ^[3] of the first two letters. If they are always a consonant sound, the line is black. If they are always a vowel sound, the line is blue. If they are sometimes vowel and sometimes consonant, the line is red:

I tried making a visualization showing where the exceptions occur:

I was hoping to come up with a clever algorithm that reduces this down to a minimum decision tree. I think that’d be related to DFA minimization^[4]. However, I didn’t figure that out, so I wrote a set of rules manually:

VOWELS = "aeiou"

def starts_with_vowel(word):
    if word.startswith("eu"): return False
    if word.startswith("ew"): return False
    if word.startswith("heir"): return True
    if word.startswith("herb") and not word.startswith("herbiv") and word != "herbarium" and word != "herbicide": return True
    if word.startswith("homage"): return True
    if word.startswith("hones"): return True
    if word.startswith("hono"): return True
    if word.startswith("hour"): return True
    if word.startswith("once"): return False
    if word.startswith("one") and not word.startswith("onerous"): return False
    if word.startswith("ula"): return False
    if word.startswith("unani"): return False
    if word.startswith("uni") and word[3:4] in "cfloqstv" and word != "unissued": return False
    if word.startswith("ub"): return False
    if word.startswith("uk"): return False
    if word.startswith("ur") and word[2:3] in VOWELS and word != "ur": return False
    if word.startswith("us") and word not in ("us", "usher"): return False
    if word.startswith("ut") and not (word.startswith("utm") or word.startswith("utt")): return False
    if word.startswith("uva"): return False
    if word.startswith("ytt"): return True
    return word[0] in VOWELS

Tangentially related: https://en.wikipedia.org/wiki/Rebracketing^[5] — before most people could read, people heard a napron and misinterpreted it as an apron, and people heard an ewt and misinterpreted it as a newt. We say an orange but in Spanish it’s naranja because the word originally had a leading n.

Source: preprocess-cmudict.py