Grokking

To Grok Grokking: Why Neural Networks Sometimes Understand Late

Grokking happens even in ridge regression: fast memorization, a long plateau of orthogonal noise, then late generalization with a delay that scales as 1/λ.