ttok 1.0
Simon Willison's Weblog
Read the full articleThe developer updated the ttok tokenizer tool to version 1.0, changing its default tokenizer from GPT-4 to GPT-5/GPT-6 based on evidence that GPT-5 and GPT-6 share the same tokenizer. This update reflects findings from an experiment confirming tokenizer consistency across these models.

