Long before the age of supercomputers and big data, codebreakers and linguists discovered that every language has a unique and predictable statistical profile. The frequency with which certain letters, pairs of letters, and words appear is not random. This statistical signature, a kind of linguistic DNA, provided a powerful tool for revealing hidden meaning, whether it was concealed by a cipher or by the passage of time.
The Signature in the Script
At its heart, the concept is stunningly simple. If you take any reasonably long piece of English text—a newspaper article, a chapter from a novel, or even this blog post—and count how many times each letter appears, you will find a remarkably consistent pattern. The letter ‘E’ will almost always be the star of the show, making up around 12% of all letters. It’s followed, in descending order, by a familiar cast of characters:
- E (~12.7%)
- T (~9.1%)
- A (~8.2%)
- O (~7.5%)
- I (~7.0%)
- N (~6.7%)
This sequence is so famous among typographers and cryptographers that its first twelve letters have a mnemonic: ETAOIN SHRDLU. This isn’t an arbitrary quirk; it’s a direct reflection of the building blocks of English. The high frequency of ‘E’ is due to its presence in the most common word (“the”), in common pronouns (“he”, “she”, “we”, “me”), the verb “be”, and countless prefixes and suffixes (-re, -de, -ed).
This fingerprint is unique to the language. In Spanish, the most frequent letters are ‘E’ and ‘A’, nearly tied for first place. In French, it’s ‘E’ followed by ‘A’ and ‘S’. In German, ‘E’ reigns supreme, but ‘N’ and ‘I’ are far more common than in English. Just by counting the letters, one can make a highly educated guess at the language of a text without understanding a single word.
The Codebreaker’s Secret Weapon
For spies and military commanders, this fingerprint was the ultimate skeleton key for cracking simple substitution ciphers, where each letter of the alphabet is consistently replaced by another letter or symbol. The first person known to have systematically documented this technique was the 9th-century Arab scholar, Al-Kindi, in his “Manuscript on Deciphering Cryptographic Messages”. He advised scribes to find a long sample of plaintext in the target language, count the letters, and then do the same for the coded message. The most common symbol in the ciphertext, he reasoned, almost certainly stood for the most common letter in the plaintext.
This method has been the protagonist in countless real and fictional tales of espionage. Edgar Allan Poe famously demonstrated it in his 1843 short story, “The Gold-Bug”. His protagonist, William Legrand, is confronted with a cipher text:
53‡‡†305))6*;4826)4‡.)4‡);806*;48†8¶60))85;1‡(;:‡*8†83(88)5*†;46(;88*96*?;8)*‡(;485);5*†2:*‡(;4956*2(5*—498¶8*;4069285);)6†8)4‡‡;1(‡9;48081;8:8‡1;48†85;4)485†528806*81(‡9;48;(88;4(‡?34;48)4‡;161;:188;‡?;
Instead of panicking, Legrand calmly tallies the symbols. He finds that ‘8’ appears most often. “Now, in English, the letter which most frequently occurs is e“, he explains. He assumes ‘8’ is ‘E’. The most frequent “word” in the code is ;48. By substituting ‘E’ for ‘8’, he gets ;4e. It’s a safe bet this is the word “the”. In a single move, he has uncovered ‘T’, ‘H’, and ‘E’—the three most useful letters—and the rest of the message quickly unravels like a ball of yarn.
This same fundamental logic was applied on a massive scale during the World Wars. While machines like the Enigma added layers of complexity, frequency analysis remained a foundational tool for cryptanalysts at Bletchley Park, helping them find entry points and verify their hypotheses about the machine’s settings.
The Linguist’s Window into Language
While codebreakers used frequency analysis to unmask language, linguists used it to understand it. The alphabet’s fingerprint isn’t just a statistical oddity; it’s a reflection of a language’s deep phonological (sound), morphological (word formation), and semantic (meaning) structures.
One of the most fascinating applications is in authorship attribution. By analyzing not just letter frequency but word frequency, punctuation habits, and sentence length, scholars can create a statistical profile of an author’s style. This technique was famously used to settle the debate over the anonymous Federalist Papers, assigning the disputed essays to either Alexander Hamilton or James Madison with a high degree of confidence.
Frequency analysis also led to the discovery of fascinating linguistic laws. Zipf’s Law, named after linguist George Kingsley Zipf, observes that in any corpus of natural language, the most frequent word will occur approximately twice as often as the second most frequent word, three times as often as the third, and so on. In English, this means “the” appears about twice as often as “of”, which appears about three times as often as “and”. This isn’t just a rule for English; it’s a near-universal property of human language, suggesting something fundamental about how we create and process information efficiently.
From Secret Messages to Smart Phones
The parallel histories of the codebreaker and the linguist, once running in separate channels, have now converged in our digital world. The simple act of counting letters has evolved into the complex field of Natural Language Processing (NLP), the science of teaching computers to understand human language.
The principles of frequency analysis are the bedrock of this modern magic:
- Predictive Text: When your phone suggests the next word as you type, it’s using a probabilistic model based on the frequency of word sequences (n-grams) from vast datasets.
- Spam Filters: Email services analyze the frequency of certain “spammy” words and phrases to decide whether a message is junk.
- Machine Translation: Services like Google Translate rely on statistical models that analyze the frequency of word and phrase alignments between two languages in massive bilingual texts.
From a 9th-century manuscript to the smartphone in your pocket, the alphabet’s fingerprint remains one of the most powerful concepts for understanding text. It’s a testament to the fact that language, for all its creative beauty and chaotic expression, is built upon a stable, predictable, and ultimately decipherable foundation. It’s a secret that was once the domain of spies and scholars, but now powers the very way we communicate.