From text to numbers

1.1 From text to numbers

A neural network is a pile of matrix multiplications. It cannot multiply the word "cat", it can only multiply numbers. So before any model sees your text, that text is chopped into tokens and each token is mapped to an integer ID (its position in a fixed vocabulary).

"Your cat is lovely"  →  ["Your", "cat", "is", "lovely"]  →  [105, 6587, 5475, 65]
        text                        tokens                        input IDs

A token is not always a word. Depending on the tokenizer it might be a word, a piece of a word, a single character, or even a byte. GPT-2, for example, works with a vocabulary of 50,257 tokens. A useful rule of thumb for English:

1 token ≈ ¾ of a word (≈ 4 characters).

This is the number that quietly governs your bills and your limits: API pricing is per token, and a model's context window is measured in tokens. How you split text is therefore not a detail, it decides cost, speed, and how much text fits in a prompt.

The process of doing the splitting is called tokenization.

---

1.2 How good is a tokenizer? Three metrics

Given the same sentence, two tokenizers can produce very different token sequences. We judge them with three numbers.

Metric	Definition	Direction	Why it matters
Vocabulary size	Number of unique tokens the tokenizer knows	N/A	Bigger vocab = more words fit in a single token, but more memory in the model's embedding table
Token fertility	Average tokens per word	Lower is better	Fewer tokens per word means shorter sequences (cheaper, faster)
Compression ratio	Average characters per token	Higher is better	Each token carries more text, so you spend fewer of them

Some reference points:

• Vocabulary size, GPT-2 ≈ 50K, Llama 2 ≈ 32K, Llama 3 ≈ 128K. A larger vocabulary can represent more words as single tokens, at the cost of a bigger embedding matrix.
• Token fertility, a fertility of 1.0 would mean every word is exactly one token. Typical English tokenizers land around 1.3–1.5. For under-represented languages it can be much higher, the same sentence in, say, Burmese may cost several times more tokens than in English, which is a real fairness-and-cost issue.
• Compression ratio, higher means each token encodes more characters, so you need fewer tokens overall. This directly improves cost and context-window utilization.

Worked example: computing the three metrics by hand

Take the sentence "unhappiness affects everyone", 3 words, 28 characters including spaces. Suppose a subword tokenizer splits it as:

["un", "happiness"]  ["affects"]  ["everyone"]

That's 4 tokens for 3 words and 28 characters:

• Token fertility = tokens / words = 4 / 3 ≈ 1.33, right in the typical English range quoted above.
• Compression ratio = characters / tokens = 28 / 4 = 7.0 characters per token.

Now compare a coarser tokenizer that refuses to split "unhappiness" and instead treats it as one unknown-word token: fertility drops to 3 / 3 = 1.0, which looks better on that one metric. But that single token now has to represent "un", "happy", and "ness" glued together, with no way for the model to recognize the piece "happiness" it already knows from other words. The lesson: fertility and compression ratio are cheap to compute and compare, but a "better" number on one metric can hide a worse ability to generalize to new words.

Why isn't "1 word = 1 token" the goal? It sounds tidy, but it breaks immediately. You would need a vocabulary entry for every word in every language, including names, typos, and words invented tomorrow. Anything unseen becomes an "unknown" token and its meaning is lost. Subword tokenizers (below) solve this: they keep a modest vocabulary of frequent pieces and assemble rare words from those pieces.

---

1.3 Naïve tokenizers (and why they fail)

The obvious approaches split on surface patterns:

• Whitespace, text.split(). Simple, but "$142.50" and "142.50" become different tokens, punctuation sticks to words, and languages without spaces (Chinese, Japanese) are hopeless.
• Regex, e.g. keep runs of letters with [A-Za-z]+. Now you've thrown away the numbers and the % and $ signs entirely.

Consider:

"The stock price rose 15.3% to $142.50 per share."

A whitespace split keeps 15.3% and $142.50 glued together as single odd tokens; a letters-only regex deletes the very numbers a financial model needs. Neither generalizes. The winning idea is to learn the split from data.

---