What is Tokenizer?

1. Definition and basic problem: Why not direct words?

Large language models (GPT-4, Claude, Llama, etc.) do not read text letter by letter or word by word like a human. Neural networks can only handle matrices, tensors and numbers. Therefore, the text must first be translated into numbers.

Historically, three different approaches have been tried in natural language processing (NLP):

1. Character-Level Processing: The text is split into individual letters (b, o, o, k). The vocabulary size is very small (a few hundred characters), but the sentences become very long. Since the computational complexity of the attention mechanism (Self-Attention) in the Transformer architecture increases with the square of the sequence length (O(N²)), the model's memory is rapidly depleted.
2. Word-Based Processing: Each word is considered a separate unit. However, in this case, for every inflectional suffix, typo, and new word in the language, the dictionary skyrockets to millions of entries; every word not found in the dictionary falls into the "unknown" (<unk> - Out of Vocabulary) tag, and the model loses meaning.
3. Subword Solution: It is today's modern standard. Frequently used words are kept as a single piece ("book"), while rare or derived words are split into meaningful subroots and suffixes ("book" + "ish" + "ness"). Thus, an infinite number of words can be represented with a fixed vocabulary size of 32,000 to 128,000.

2. Tokenizer algorithms and mathematical logic

The major tokenizer algorithms at the heart of modern language models are:

• Byte Pair Encoding (BPE): Originally a data compression algorithm, BPE is today the foundation of the GPT series and Llama models. It starts with all the basic characters in the text and iteratively merges the most frequently occurring consecutive character pairs in the corpus, adding them to the vocabulary.
• WordPiece: Popularized by Google in the BERT model, this method is based on probability rather than frequency. When merging pairs, it selects the subword pieces that most increase the language model's likelihood score on the training data.
• SentencePiece and Byte-Fallback: It treats spaces as a special underscore character and handles text as a raw byte stream. Whenever any rare Unicode character not found in the vocabulary is encountered, it falls back directly to the UTF-8 byte (Byte-Fallback), reducing the <unk> error to zero.

3. "The Tokenizer Tax" in Turkish

More than 85% of the training data of large language models is in English. This causes the tokenizer dictionary to be filled with predominantly English roots and words.

In rich agglutinative and morphological languages ​​such as Turkish, this creates a serious cost and context inequality:

• The sentence "Artificial intelligence is transforming software engineering." is approximately 7 tokens.
• The sentence "Yapay zekâ yazılım mühendisliğini dönüştürüyor." can consume 14-16 tokens due to the splitting of suffixes.

For this reason, Turkish-speaking users can fit fewer documents in the same context window and pay twice as much for API services. Increasing the dictionary size to over 128k with Llama 3 and GPT-4o has significantly increased the Turkish token efficiency.

4. Security and Edge Cases: Glitch Tokens

Special tokens that appear in the Tokenizer dictionary but appear rarely or in meaningless contexts in the text body during pre-training of the model are called "Glitch Tokens".

For example, when tokens like SolidGoldMagikarp, derived from Reddit usernames or e-commerce site codes, are queried, the AI begins to hallucinate because it cannot correctly position the token's vector in the embedding space; it may spout nonsensical profanities or lock up.

Questions and answers

Related terms

Related tools