Get the app
LLM & Text GenerationMachine Learning1 min reading time

ttok 1.0

Simon Willison's Weblog
Read the full article
The developer updated the ttok tokenizer tool to version 1.0, changing its default tokenizer from GPT-4 to GPT-5/GPT-6 based on evidence that GPT-5 and GPT-6 share the same tokenizer. This update reflects findings from an experiment confirming tokenizer consistency across these models.
#gpt-4#gpt-5#gpt-6#tokenization#ttok#william-liu

More in LLM & Text Generation

LLM & Text Generation5 min read

Researchers from Stanford and UC Berkeley Introduce TokenGate: Token-Gated Attention Makes Long-Context Prefill Up to 11.3x Faster

MarkTechPost
LLM & Text Generation2 min read

Moonshot releases Kimi K3.1 Pro and says it cut its hacking skills

Covered by 4 sources