Ok, here's one you may have missed.

A thread in Lobsters' AI section has picked up Gwern's essay "Human-like Neural Nets by Catapulting" — and the comment section is actually worth reading. Gwern is the person who anticipated the scaling hypothesis long before it became mainstream conversation, so when he puts forward a theory, there's usually reason to pay attention.

The core idea is this: what if we train a model on a relatively limited dataset, but continue the training process long after the loss curve appears to have flattened? The theory builds on the "grokking" phenomenon — the observed effect where a model seemingly stagnates, then suddenly generalizes in an entirely different way. Gwern speculates that models large enough, subjected to heavy weight decay to prevent memorization, can be forced to internalize patterns in a way that more closely resembles human generalization than ordinary statistical interpolation.

The analogy he uses is sleep: our brains "forget" during REM sleep, and this appears to strengthen — not weaken — the ability to generalize. Could overtraining with aggressive regularization do the same for neural networks?

No one has yet published independent benchmarks confirming the effect at scale — but the argument is clean enough that people are taking it seriously.
Gwern's 'catapult' theory explodes on Lobsters: Grokking as a path to AGI? - Bilde 1

What's particularly interesting right now is the timing. We're in a period where the conventional certainty that "just scale up data and parameters" is starting to crack. Multiple players are reporting that pure scaling gains are plateauing. In that context, alternative training strategies — including this kind of intensive overtraining — feel more relevant than they did two years ago.

Be aware of the limitations here: this is community speculation around an essay, not a peer-reviewed study. No one has presented reproducible results from large models. And "grokking" has primarily been demonstrated on relatively simple tasks — whether it scales to frontier models is an open question.

But that's precisely the kind of question that moves from the underground into the labs. Watch for whether anyone starts publishing ablations around overtraining regimes in the coming months — that could be an early signal that someone is testing this more systematically.

The source is Lobsters and the associated HN discussion. Early signal, not conclusion.