Pretraining data, not verifiability, is why LLMs are especially good at math (and coding)

A new analysis argues that large language models' strong performance in math and coding tasks stems primarily from their extensive pretraining data rather than from the verifiability of these domains. This challenges the idea that the ability to verify outputs is the main reason for their proficiency in these areas.