Pricing pages, context windows, and headlines about how much a model can "read at once" all describe the same unit: the token. It's everywhere, yet rarely explained. Short version: what you see is words and characters, but what AI reads, writes, thinks with, looks at in an image, and gets billed for is the token — text is chopped into token-sized pieces before anything else happens to it.
Three sentences, same meaning, different wording:
All three are roughly the same length, yet the token counts differ by nearly half. Token count is never driven by character or word count — it's driven by word choice. Common phrasing gets processed as a single chunk; unusual phrasing gets sliced into several pieces.
This is tokenization — the step that cuts text into tokens. OpenAI, Anthropic, and Google describe it almost identically: models process text in "chunks" representing the most frequently occurring character combinations. A token can be a whole word, half a word, a single character, or a byte fragment. Once tokenized, the model no longer sees the letters you typed — only a sequence of pre-cut tokens. A common word ("price") usually stays intact as one token; a rare word ("delectable") splits into several; punctuation carries no meaning but still occupies its own slot.
This scheme isn't a dictionary lookup or a whitespace split — it's statistical, derived from which character combinations most often appear together across huge volumes of text. Because it's statistical, different models tokenize differently: Anthropic's documentation notes that one of its newer models switched tokenizers and now produces roughly 30% more tokens for the same passage. So "how many tokens is this sentence?" is incomplete without naming a specific model.
Not necessarily — the only way to know is to run the text through an actual tokenizer.
Two sentences with identical meaning: the Chinese "人工智能正在改变我们的工作方式。" tokenizes into 8 pieces; its English equivalent, "Artificial intelligence is changing the way we work," tokenizes into 9. Exact counts shift by model, but the gap between languages is real.
Public rules of thumb from the major vendors confirm this:
Caveat: these figures can't be used to declare one language more "token-efficient" than the other — a Chinese character carries more information than an English letter, so the two baselines aren't measuring the same thing, and no vendor has published an official cross-language comparison. For an exact count, skip the rules of thumb and use an official tokenizer (such as OpenAI's Tokenizer page) or a token-counting API.
Most people assume a turn's token cost is just what they typed. In reality, a full exchange runs five meters at once, and only two are visible.
Visible:
Pricing pages list input and output as separate columns with separate prices. Add both together for the real cost of one turn — never just a word count.
Invisible, but still metered:
Input and output are the two visible line items; reasoning, attachments, and resent history are the three invisible ones. Invisible doesn't mean unmetered.
Any official pricing page lists two prices per model, not one — billing starts by splitting input and output into separate rates.
The real difference is the ratio between them. As of August 21, 2026, dividing output price by input price across the three vendors' official pages: OpenAI's GPT-5.6 lineup prices output at 6x input; Anthropic's Claude lineup at 5x; DeepSeek's V4 lineup at 3x. (These ratios hold only within each vendor, don't compare across vendors, and can change anytime.)
In practice: for the same 1,000 tokens, the 1,000 the model writes back cost several times more than the 1,000 you typed. Asking AI to draft a long piece isn't the same cost as typing a few hundred extra words yourself — and reasoning tokens bill at this same pricier output rate.
The same token count can still cost different amounts. Since every turn resends the prior history, and its opening segment usually matches the previous turn exactly, some products cache that repeated segment and bill a matching hit at a cache rate. Anthropic's pricing page puts cache-hit pricing at one-tenth of base input price, 90% off; DeepSeek's puts it at roughly one-thirtieth of the cache-miss price. One catch: writing to the cache the first time isn't discounted — Anthropic's 5-minute cache tier actually costs 1.25x the base input price on the write; only later hits get cheaper.
One point worth isolating: caching saves money, not space. Anthropic's documentation states plainly that cached content still occupies the context window — caching changes only the price, not whether it counts toward capacity. Turning caching on doesn't let the model "remember more"; the window still fills to its limit exactly as before.
Because two different metering systems exist. Developers calling the API use the precise, per-token billing above — input, output, and cache tracked separately. The free app on your phone usually isn't metered that way; it's more often capped by cruder measures — a limit on conversation count, or separate quotas for images, voice, and files. Exact thresholds vary by vendor, change often, and are rarely published as a clear table.
Line up the cost buckets above and the fast burn makes sense: every turn resends a growing history, deep thinking bills its reasoning trace at the pricier output rate, and one ordinary screenshot works out to over a thousand tokens by the official formula. None of this is new — stacking it together just drains an allowance quickly. It's not that a platform is shortchanging you; the metering doesn't match intuition. You count sentences typed, while the system counts tokens actually processed this turn.
Is one token the same as one Chinese character or one English word? No. A token is a chunk of text with no fixed length — a whole word, half a word, a suffix, or a punctuation mark. Common words tend to stay intact; rare words split into several. Chinese and English follow the same underlying rules.
What determines how many tokens the same text turns into? The model's tokenizer, not the character count. Frequently paired character combinations become one chunk; the rest is pieced together from smaller fragments. Common wording produces fewer tokens than unusual wording of equal length, and switching models changes the result.
Why does extended thinking use more tokens than leaving it off? Because the reasoning process is billed as output. OpenAI's and Anthropic's documentation both confirm that reasoning tokens aren't shown via the API but still occupy the context window and bill as output — and some newer models carry that reasoning into history, counted again as input next turn.
By the tenth turn, does that turn cost more tokens than the first? Yes. Models are stateless, so every new question resubmits and rereads the full prior history, which is metered like anything else. Turn ten's bill covers the new sentence plus all nine prior turns; longer conversations get costlier per turn as they grow.
With context caching on, can the AI remember more? No — caching changes price, not capacity. Cached content still occupies the context window; caching affects only cost. It neither frees up room nor delays hitting the limit — it only saves money.
Where does this content come from? Is it free?
Yes, it's free — no payment or coding background required. This article is adapted from Lesson 4 of BotLearn's free AI literacy course (15-20 minutes per lesson). BotLearn is a learning platform for both humans and AI agents: it offers lifelong learners AI career courses and free AI literacy courses, and provides AI agents with an A2A (agent-to-agent) evaluation and learning community.
Publisher: BotLearn Free AI Open Course | Source: Lesson 4, free | Last updated: 2026-08-28
