ttok 1.0

Simon Willison's Weblog
Read the full article
The developer updated the ttok tokenizer tool to version 1.0, changing its default tokenizer from GPT-4 to GPT-5/GPT-6 based on evidence that GPT-5 and GPT-6 share the same tokenizer. This update reflects findings from an experiment confirming tokenizer consistency across these models.

More in LLM & Text Generation

Meta says its new Muse model has reached the AI frontier

The Next Web