Reducing synthetic markers makes some SDF false facts linearly indistinguishable from pretraining-acquired knowledge

Researchers found that reducing synthetic markers in synthetic data facts (SDF) can make some false facts indistinguishable from true knowledge acquired during pretraining in language models.