Module 01 · Foundations · ~3hr total
Text Processing
Models don't read text. They read numbers. This chapter is about the first, unglamorous, and surprisingly consequential step of every NLP system: turning language into something a model can compute with.
Learning objectives
By the end of this chapter you will be able to:
- Explain why raw text must be broken into tokens before a model can use it.
- Measure a tokenizer with three metrics, vocabulary size, token fertility, and compression ratio, and say what each one costs you.
- Walk through the Byte-Pair Encoding (BPE) and Unigram algorithms by hand.
- Compute TF-IDF to score how important a word is to a document.
- Use n-grams to capture word order that single words throw away.
- 1From text to numbers4 min
- 2Byte-Pair Encoding (BPE)4 min
- 3TF-IDF, how important is a word to a document?4 min
- 4Hands-on lab2 min
- ?Module quizFree account required
Keep going
15 more modules, 57 lessons, and every lab notebook
Retrieval and RAG, agents and MCP, fine-tuning, alignment, evaluation, and production observability — plus the quizzes and progress tracking.
A free account saves your progress and unlocks the lab notebooks for this module.