ttok 1.0

Simon Willison's Weblog
Read the full article
The developer updated the ttok tokenizer tool to version 1.0, changing its default tokenizer from GPT-4 to GPT-5/GPT-6 based on evidence that GPT-5 and GPT-6 share the same tokenizer. This update reflects findings from an experiment confirming tokenizer consistency across these models.

More in LLM & Text Generation

OpenAI upgrades ChatGPT and Codex with new GPT-6.1 Terra model

Covered by 2 sources

Anthropic IPO: S-1 Filing Offers First Look at Claude Maker’s Finances

Bloomberg