The GPT Tokenizer: A Practical Guide to Tokenization in LLMs - Let's Build It!

Ever asked ChatGPT a seemingly simple question and received a bizarre answer? Perhaps you asked it to count the letters in a word, and it failed spectacularly. Or maybe you noticed it struggles with spelling, reversing a string of text, or basic arithmetic. It feels strange, doesn't it? How can a model that writes beautiful poetry and complex code get tripped up by something a child could do?
The answer, in most cases, isn't a flaw in the AI's "thinking" but lies in a hidden, foundational process called tokenization.
Tokenization is the secret language of large language models (LLMs). It’s the essential, often-overlooked first step that translates our human-readable text into a format the AI can actually understand. Think of it as the bridge between our world of words and the model's world of numbers. Every single quirk, limitation, and unexpected behavi

1. The GPT Tokenizer: A Practical Guide to
Tokenization in LLMs - Let's Build It!

I. Introduction: The Hidden Force Behind AI's Language Understanding
Ever asked ChatGPT a seemingly simple question and received a bizarre answer? Perhaps
you asked it to count the letters in a word, and it failed spectacularly. Or maybe you noticed it
struggles with spelling, reversing a string of text, or basic arithmetic. It feels strange, doesn't
it? How can a model that writes beautiful poetry and complex code get tripped up by
something a child could do?
The answer, in most cases, isn't a flaw in the AI's "thinking" but lies in a hidden, foundational
process called tokenization.
Tokenization is the secret language of large language models (LLMs). It’s the essential,
often-overlooked first step that translates our human-readable text into a format the AI can
actually understand. Think of it as the bridge between our world of words and the model's
world of numbers. Every single quirk, limitation, and unexpected behavior you see in an LLM
can often be traced back to the way it "sees" text through its tokenizer.
In this guide, we're not just going to talk about tokenization. We're going to roll up our
sleeves and build a GPT-style tokenizer from the ground up. By the end, you'll understand
not just what it is, but why it works the way it does, and you’ll finally have the key to
deciphering AI’s most mysterious behaviors.
We’ll also look to the future. While tokenization has powered the AI revolution so far, the
industry is already taking its first steps beyond it. In late 2024, researchers at Meta
introduced a groundbreaking tokenizer-free model, a hint at a future where AI reads text

2. more like we do. But to understand where we're going, we first have to master the foundation
of where we are today. Let’s get building.
II. Understanding the Basics: From Text to Numbers
At its heart, a computer doesn't understand "A," "B," or "C." It understands numbers. To a
machine, all data is just a sequence of ones and zeros. So, how do we get from a rich,
nuanced sentence to a string of numbers an AI can process?
The journey starts with converting text into integers. This process is called tokenization, and
a "token" is simply a piece of text—a word, part of a word, or a single character—that has
been mapped to a number.
Let's take a simple example: "Hello World".
A naive, straightforward approach would be to create a vocabulary of every unique character
in our text. For "Hello World," the unique characters are H, e, l, o, (space), w, r, d. We can
assign a unique integer to each:
●​H -> 0
●​e -> 1
●​l -> 2
●​o -> 3
●​(space) -> 4
●​w -> 5
●​r -> 6
●​d -> 7
Using this vocabulary, "Hello world" becomes the sequence of tokens: [0, 1, 2, 2, 3,
4, 5, 3, 6, 2, 7]
. Now the computer has something to work with. This is called
character-level tokenization. It’s simple and it works, but it has a huge problem: the
sequences are very, very long. The book War and Peace would turn into millions of
individual character tokens, making it incredibly difficult for an AI to find patterns across long
stretches of text.
To handle the vast diversity of human language—from English to Korean (안녕하세요),
including emojis (??????)—we need a universal system. This is where Unicode comes in.
Unicode is like a giant dictionary that assigns a unique number, called a "code point," to
nearly every character imaginable. The letter 'A' is 65, while the Korean character '안' is
50,000.
But even with Unicode, storing every character as its full code point can be inefficient. This is
why we use an encoding scheme like UTF-8. UTF-8 is a clever, variable-width recipe for
representing Unicode characters as a sequence of bytes (a byte being a number from 0 to
255). Simple English characters that are also in ASCII take up just one byte, while more
complex ones might take up four bytes. This makes it backward compatible with ASCII, the
older English-centric standard.

3. UTF-8 is the standard for text on the internet, and it’s the raw material our GPT tokenizer will
work with. The challenge for a modern tokenizer is to take these raw bytes and group them
in an intelligent way—more meaningful than single characters, but more flexible than whole
words.
III. The Magic of Byte Pair Encoding (BPE)
If single characters are too small and whole words are too rigid, what’s the right balance?
The answer lies in a clever algorithm called Byte Pair Encoding (BPE). It’s the engine that
powers most modern tokenizers, including those used by GPT models.
In plain English, BPE is a data compression algorithm that finds the most common pair of
consecutive bytes in a text and replaces them with a new, unused byte. It repeats this
process over and over, building a vocabulary of a predetermined size.
Imagine BPE as creating a form of shorthand for a language it's never seen before. It starts
with a long piece of text and looks for letter combinations that appear all the time. If it sees
"t" and "h" together constantly, it might decide to create a new shorthand symbol, "th," and
replace every instance of "t-h" with it. Then it scans the text again. Maybe it notices "th" and
"e" appear together a lot. So, it creates a new symbol, "the," and replaces all instances of
"th-e." It keeps doing this until it has a nice, compact dictionary of shorthand symbols. Let's walk through a simple, visual example. Our starting text is:​
aaabdaaabac
Step 1: Find the most frequent pair.​
The pair
aa appears most often (twice). So, we'll merge aa into a new token. Let's call it Z.
Our vocabulary now includes our original characters plus
Z. We replace every aa in the text
with
Z:​
ZabdaZabac​
The text is now shorter, down from 11 tokens to 9.
Step 2: Repeat the process.​
Let's look at the new text. What's the most frequent pair now?
ab appears twice. Let's merge
it into a new token,
Y. We replace ab with Y:​
ZYdZYac​
The text is even shorter, now just 7 tokens long.
Step 3: One more time.​
The most frequent pair is now
ZY. Let's merge that into X. We replace ZY with X:​
XdXac​
In just three merge steps, we've compressed an 11-character string into a 5-token sequence.
We've also created a "merge rulebook" that tells us
Z means aa, Y means ab, and X means
ZY.
This is exactly how the GPT tokenizer works, but instead of starting with letters, it starts with
the 256 possible bytes from UTF-8. It takes a massive amount of text (like a huge chunk of

4. the internet), starts with a vocabulary of 256 bytes, and performs this merge operation tens
of thousands of times. For GPT-2, this was done about 50,000 times, creating a final
vocabulary of around 50,257 tokens (256 base bytes + 50,000 merges + 1 special token).
The result is a vocabulary that contains single-byte characters, but also multi-byte tokens for
common letter combinations (" ing", " the"), common words (" Hello", " world"), and even
parts of words that appear frequently across the training data. This BPE algorithm is the key
to creating a tokenizer that is both efficient and flexible enough to handle the complexity of
human language.
IV. GPT's Tokenization Evolution: From GPT-2 to GPT-4
As OpenAI developed more powerful models, their approach to tokenization also evolved. A
tokenizer isn't just about running the BPE algorithm; it involves clever engineering choices
that have a huge impact on model performance.
The first crucial refinement was something called pre-tokenization. Instead of just feeding
raw text into the BPE algorithm, the GPT-2 developers realized they needed to set some
ground rules. Imagine the word "dog." in a sentence. Without any rules, BPE might see
"dog." so often that it merges them into a single token. This isn't ideal, because you're mixing
the semantic meaning of "dog" with the grammatical function of a period. The model then
has to learn that the token "dog." is very similar to "dog!" and "dog?". To solve this, GPT-2 first splits the text into chunks using a regular expression (regex). This
complex-looking pattern is actually just a set of rules that breaks text apart based on
categories:
1.​Contractions: Keeps common contractions like 's, 're, and 've together.
2.​Words: Groups consecutive letters together.
3.​Numbers: Groups consecutive numbers together.
4.​Punctuation: Groups other characters, like !, ?, and . together.
5.​Whitespace: Handles spaces.
BPE is then performed within these chunks, but never across them. The 'e' at the end of
"apple" can never merge with the '!' in "apple!". This simple step helps the tokenizer create
more meaningful, semantically consistent tokens.
When GPT-4 was released, its tokenizer, known as cl100k_base, introduced further
improvements. The developers learned from the limitations of the GPT-2 tokenizer and made
several key changes:
●​Better Contraction Handling: The regex was made case-insensitive, so HOW'S is
tokenized consistently with
how's, fixing a major annoyance in GPT-2.
●​Smarter Number Handling: The GPT-4 tokenizer will only group numbers that have
two or more digits. This prevents single digits from being treated as standalone
numbers, which helps with mathematical reasoning.
●​Improved Whitespace Merging: GPT-4 is much more efficient at handling code and
structured text because it merges consecutive spaces, whereas GPT-2 would create

5. a separate token for every single space. This was a huge bottleneck for processing
Python code, where indentation is key.
●​Larger Vocabulary: The vocabulary size was increased from ~50,000 for GPT-2 to
over 100,000 for GPT-4. This allows the tokenizer to create more tokens for different
languages and common code sequences, making it more efficient overall.
Finally, both tokenizers include special tokens. These are tokens that exist outside the
normal BPE vocabulary and are used to send signals to the model. The most famous one is
<|endoftext|>. This token is inserted between different documents in the training data to
tell the model, "Okay, stop what you were thinking about. The next piece of text is completely
unrelated." Over time, the list of special tokens has grown to handle instructions for chat
models, code interpreters, and other advanced functions.
This evolution shows that building a great tokenizer is as much an art as it is a science,
involving a deep understanding of language, data, and the downstream effects on the
model's behavior.
V. Why Tokenization Causes Strange AI Behaviors
Now we get to the fun part. Armed with our knowledge of tokenization, we can finally solve
the mysteries of why LLMs act so strange sometimes.
The Spelling and Reversing Problem​
Let's revisit the spelling mystery. Why can't a powerful AI count the 'l's in a word like
.DefaultCellStyle? Because for the GPT-4 tokenizer, .DefaultCellStyle is not a
sequence of 18 characters. It is a single token (token ID 98518). The model doesn't see the
individual letters
D-e-f-a-u-l-t...; it just sees one indivisible unit. It has no more insight
into the letters inside that token than you have into the atoms that make up your chair. This
is also why it can't reverse the string—it has nothing to reverse.
The interesting trick is that if you ask the model to first "print out every single character
separated by spaces," it will succeed. By doing this, you force it to break the single token
apart into individual character tokens. Once the letters are visible to the model as separate
tokens, it can then easily reverse them. You're essentially doing the tokenization for the
model.
Why AI Struggles with Non-English Languages​
Most tokenizers for major models like GPT are trained on a dataset that is overwhelmingly
English. As a result, the BPE algorithm creates a rich, efficient vocabulary for English.
Common English words and word parts become single tokens.
For other languages, however, the tokenizer often has to fall back to shorter character
combinations or even individual bytes. For example, the English phrase "Hello how are
you?" might be 5 tokens. Its Korean equivalent, "안녕하세요 어떻게 지내세요?", is 15
tokens. That’s a three-fold increase! This makes non-English text more "bloated" and less
efficient for the model to process. It consumes more of the precious context window and
gives the model less semantic information per token, leading to poorer performance.

6. Arithmetic Challenges in LLMs​
Addition and other arithmetic tasks rely on consistent, place-based rules—you line up the
ones, the tens, the hundreds. But tokenizers don't see numbers this way. A number like
"1296" might be a single token, but "3457" might be split into two tokens: "34" and "57".
Because the representation is arbitrary and inconsistent, it's incredibly difficult for the model
to learn the stable rules of arithmetic. It's like trying to learn addition when the numbers keep
changing their shape. Newer models, like Llama 2, address this by forcing their tokenizer to
always split digits individually, which greatly improves their math skills.
The SolidGoldMagikarp Mystery Solved​
One of the most famous tokenization bugs is "SolidGoldMagikarp." Researchers discovered
that asking early GPT models about this seemingly random phrase would cause them to
break completely, producing evasive, insulting, or nonsensical responses.
The answer, once again, was tokenization. The tokenizer was trained on a dataset that
included a large amount of data from Reddit. On Reddit, there was a very active user named
u/SolidGoldMagikarp. Because this username appeared so frequently in the tokenizer's
training data, the BPE algorithm created a single, dedicated token for the string
"SolidGoldMagikarp".
However, the language model itself was trained on a different, more curated dataset that did
not include this Reddit data. This meant the "SolidGoldMagikarp" token existed in the
vocabulary, but the model never saw it during its training. Its embedding vector was never
updated; it was essentially uninitialized, random noise.
When a user prompted the model with this token, it was like feeding garbage into the
system. The model encountered a completely foreign, untrained concept, and its behavior
became undefined and erratic. It was the tokenization equivalent of a segmentation fault in
programming.
Trailing Whitespace and Partial Tokens​
Have you ever seen a warning in an AI playground that your prompt "ends in a trailing
space"? This happens because spaces are typically prepended to the start of a word to form
a token (e.g., the token is " world", not "world"). If your prompt is "An ice cream shop
tagline:", the model expects to predict a token that starts with a space, like " Scoops". But if
your prompt is "An ice cream shop tagline: ", you've already provided the space as a
separate token. This puts the model in an "out-of-distribution" state it rarely saw in training,
leading to worse performance. You've split a token in a way it isn't used to seeing.
VI. Alternative Approaches: SentencePiece and Beyond
The GPT series uses a tokenizer library called tiktoken, which works directly on UTF-8
bytes. But this isn't the only way to do things. Many other popular models, including Meta's
Llama and Mistral, use a different library called SentencePiece.
The fundamental difference lies in the raw material they work with:
●​tiktoken (GPT approach): Takes text, converts it to UTF-8 bytes, and then runs BPE
on the bytes. The base vocabulary is always the 256 possible byte values.

7. ●​SentencePiece (Llama approach): Takes text and runs BPE directly on the Unicode
code points. It works with characters first.
So what happens when SentencePiece encounters a rare character that's not in its
vocabulary? This is where its cleverest feature comes in: byte fallback. If it sees a rare
Chinese character, instead of mapping it to an "unknown" token, it encodes the character
into its UTF-8 bytes and represents it using special byte tokens (e.g.,
<0xE4>, <0xBD>,
<0xA0>).
Let's use an analogy. Imagine you're building a cookbook. The tiktoken approach is to
first translate every recipe in the world into a universal "ingredient code" (the bytes) and then
find common patterns in those codes. The
SentencePiece approach is to try and work with
the original recipe words ("flour," "sugar") and only when it finds a very rare ingredient does it
look up its chemical formula (the bytes) as a fallback.
This makes SentencePiece particularly efficient for multilingual applications. By working at
the character level, it can create much more compact and meaningful tokens for languages
with large character sets, like Chinese or Japanese, without having to resort to raw bytes for
everything.
The trade-off is complexity. SentencePiece has many more configuration options and
historical quirks. The
tiktoken approach is arguably cleaner and more universal, treating
all languages equally (even if less efficiently for some), while SentencePiece is more
optimized for a multilingual world. Choosing between them depends on the specific goals of
the model you're building.
VII. The Future: Tokenizer-Free Models
For all its cleverness, tokenization remains a bottleneck and a source of many of the
problems we've discussed. It's a "lossy" compression of text that forces the model to see the
world through a fixed, predefined vocabulary. What if an AI could just read the raw bytes of
text directly, just like a computer program does?
This is the promise of tokenizer-free models, and it's where the cutting edge of AI research
is heading.
In December 2024, researchers at Meta published work on the Byte Latent Transformer
(BLT), a model architecture that eliminates the need for a static tokenizer. Instead of
breaking text into a fixed set of tokens before it reaches the model, BLT learns to
dynamically group or "patch" sequences of raw bytes as part of its own architecture.
Think of it this way: a traditional tokenizer is like a factory worker who chops up a long string
of ingredients into predefined chunks before they go down the assembly line. The size and
type of chunks are fixed. A tokenizer-free model like BLT is like a master chef on the
assembly line who looks at the incoming stream of ingredients and decides, on the fly, the
best way to group them for the recipe at hand.
This approach offers several huge advantages:

8. 1.​No More "Out-of-Vocabulary" Issues: The model can, in theory, process any string
of bytes, no matter how rare or unusual. Problems like "SolidGoldMagikarp" would
disappear.
2.​Perfect Spelling and Character Awareness: By operating on the byte level, the
model retains full information about the text. It can "see" every individual character,
which could unlock true spelling and character-manipulation capabilities.
3.​True Multilingualism: All languages are just different sequences of bytes. A
tokenizer-free model would be inherently language-agnostic, eliminating the
performance gap caused by biased vocabularies.
4.​Efficiency: Meta's research suggests these models can be significantly more
efficient, potentially reducing the computational cost (FLOPs) at inference time by up
to 50%.
Tokenizer-free models represent a fundamental shift, moving complexity away from a
separate, brittle preprocessing step and integrating it directly into the neural network's
learning process. While still an active area of research, this is the direction the industry is
headed. Tokenization was the brilliant hack that got us here, but the future of AI may not
need it at all.
VIII. Building Your Own Tokenizer: A Practical Guide
The best way to truly understand tokenization is to build one yourself. Following Andrej
Karpathy's
minbpe (minimum BPE) exercise is a fantastic way to do this. You don't need to
be an expert programmer to follow along. Here’s the roadmap for your journey:
Step 1: Basic BPE Implementation​
Start with the fundamentals. Write a simple Python class that can:
●​train: Take a piece of text and a vocabulary size, and perform the BPE merge
algorithm to create a vocabulary.
●​encode: Take a string and convert it into a sequence of token IDs using the trained
vocabulary.
●​decode: Take a sequence of token IDs and convert it back into a string.​
At this stage, you'll be working directly on raw UTF-8 bytes without any fancy
preprocessing.
Step 2: Add Regex Preprocessing​
Now, level up to a GPT-2 style tokenizer. Instead of encoding the whole text at once, you'll
first use a regex pattern (like the one used in GPT-4) to split the text into chunks of words,
numbers, and punctuation. You'll then run your BPE encoding on each chunk individually
and concatenate the results. You will immediately see more meaningful tokens emerge.
Step 3: Load GPT-4 Merges​
To perfectly replicate the official GPT-4 tokenizer, you don't need to retrain it. OpenAI has
published the "merge rulebook" for
cl100k_base. Your task in this step is to load these
predefined merge rules and use them in your
encode function instead of training your own.
This is the most complex step, as it involves handling some specific byte permutations used
by OpenAI.

9. Step 4: Handle Special Tokens​
The final step is to add support for special tokens like
<|endoftext|>. This involves
modifying your
encode function to first look for any special tokens in the text, replace them
with their corresponding IDs, and only then run the BPE process on the remaining parts of
the text.
By completing these four steps, you will have built a fully functional tokenizer from scratch
that behaves identically to OpenAI's official
tiktoken library. You will have demystified the
entire process from start to finish.
IX. Practical Implications and Best Practices
Understanding tokenization isn't just an academic exercise. It has direct, practical
consequences for anyone building with or using LLMs.
Token Efficiency Matters (YAML vs. JSON)​
You pay for AI services per token. The fewer tokens you use to represent the same
information, the faster and cheaper your application will be. For example, while
human-readable YAML is often more token-efficient than pretty-printed JSON, a minified,
whitespace-removed JSON object is typically the most token-efficient format of all. Simply
changing your data format can lead to significant cost and performance improvements. You Pay Per Token​
Always remember that context windows are limited, and API calls are priced by the number
of tokens processed (both input and output). A deep understanding of how your chosen
model tokenizes text allows you to optimize your prompts to be as compact and efficient as
possible, saving you money and getting more out of the model's limited context.
Optimizing Prompts​
Be mindful of how small changes can affect token count. Adding unnecessary whitespace,
using inefficient data formats, or phrasing a question in a verbose way can all increase your
token usage. Tools like OpenAI's
tiktokenizer are invaluable for experimenting with your
prompts and seeing exactly how they are viewed by the model.
Security Concerns​
Finally, special tokens can be a security risk. If an application allows user input to contain
special tokens that are then processed by the model, it can open the door to "prompt
injection" attacks. An attacker could potentially confuse the model, make it bypass its safety
guidelines, or even hijack its functions. Always sanitize user input and control which special
tokens, if any, are allowed. X. Conclusion: Mastering the Foundation
We've traveled from the simplest concept of turning letters into numbers to building a
complete GPT-4 tokenizer, demystifying bizarre AI behaviors, and even peering into the
tokenizer-free future.
Tokenization is the foundational layer upon which all of modern AI is built. It’s the invisible
bridge between human language and the neural network, and every little quirk in its design

10. cascades up to affect the model's behavior in profound ways. Understanding it is a
superpower. It allows you to debug AI systems more effectively, build more efficient and
cost-effective applications, and stay ahead of the curve as the technology evolves.
The dream of tokenizer-free models is on the horizon, promising a future where AIs can read
text with the same nuance and completeness that we do. But for now, and for the
foreseeable future, tokenization is the name of the game. The next time your favorite LLM
does something strange, you won't just be confused. You'll smile, because you'll know the
culprit is often the tokenizer.
More Posts:
●​Get This New App Freezur That Creates Brainrot Viral Videos That
Make Your Viewers Freeze In Their Tracks & Watch As If Compelled
●​$37 Investment, $2,000+ Monthly Returns: The StudioBook AI
Income Breakdown
●​InfluAIcer Review: Creates Talking Digital Influencers That Sell
Products, Build Followers & Go Viral For You – 24/7
●​Stop Losing Sales: 8 WebinarKit Tools That Convert While You Sleep
●​Coaching Profit System Review: 3 Proven Ways This Revolutionary
System Solves Your Income Problems