dictionary

Tokens & Tokenization

Before an LLM can read or write text, the text must be broken down into tokens. In English, a token is generally about 4 characters or 0.75 words. Common words might be a single token, while complex words are broken into syllables. Models charge by the token and have strict token limits in their context windows.