The genome hypothesis: a thought experiment about what pretraining actually finds

This is a living document about what pretrained neural networks are, what they're not, and what might be missing from how we build AI. No experimental results here. Just questions, a few hypotheses, and where they come from. I'll update it as the thinking changes.

Current raw factors/questions influencing this exploration:

1. Take a model trying to learn m * a, what is it trying to fit. At 1000 examples with seed A and the code as present in the example, why does it need 64 neurons/dimensions to fit the data. And why does it not generalize to the rule?
2. How does the number of neurons matter for an arbitrary dataset? What's the dependency? Does more neurons mean better model? If so, in what configuration of width/depth? If more neurons generally mean better model, why are elephants not more general purpose than humans?
3. What really is reality perceived by an intelligent brain? Is it the environment? Every biological orgnaism lives on the same planet but perceive its own world differently? Why? A cockroach's reality about the world is not the same as a human's.
4. In an abstracted representation, why is a pre trained model so bad at sample efficiency? Almost like its starting from scratch and doing the analogous of evolution. In that same abstract space, can we say gradient descent is credit assignment algorithm much akin to evolution. Both are not sample efficient and searching randomly in high dimensional space to find the intelligence that fits the reality/dataset. In this abstract world, the specifics like evolution has survival as direct result of the process may not matter. Or maybe every single pass of the learning algoritm is the analogous to survival. How do we map the abstract high level processes at a sufficiently zoomed out lens so that specifcs matter exactly where it should and if we can, what does it tell us about what's missing?
5. What could be the key for an biolgoical intelligence to be so much more sample efficient and energy efficient? Embodiment, but that's a physical constraint of the system itself, not of the intelligence. Or is it? What if inteliigence was the only system evolution ever tried to optimize for without phsyical embodiment constraints? What would be the necessary conditions for such a process? What does such a world look like? Does a pre trained model with no embodiment match the output of such a world? And if yes, then the way evolution built intelligence could be the answer. Evoution didn't create a brain, it created encoding/map for an intelligence. RNA -> DNA -> Genome -> Development -> Brain. A brain of a human child at birth is not blank, its already good at so many things.
6. At sufficient scale of data, can we presume that we are approaching the reality of humans itself? How do we answer how much data is enough? Pure text based self supervised learning can model the world vividly enough to make comprehensiion possible in modern llms but the causal chains break down, why? Is it the lack of a physical world model? If so, can forcing it to learn the physical properties of the world improve the causal understanding? Forget the work everyone is doing around JEPA/world models and focus on it from first principles. When you ask a model to draw a tank 10 metres above the ground, what does it need to understand to actually draw it such that itse physically probable? And how does it affect the weird jaggedness of intelligence.
7. Abstract representation irresepective of modality: Why does a human having no eye sight or no auditory input or no abilty to talk be able to perceive the relaity the same as people who can't. Cleaner test is to look at people lacking these inputs from birth. And when a person is chronically blind and deaf together, how does it make them so much harder to train. There's only one dicymented case of a nun who was wild in her child https://en.wikipedia.org/wiki/Marie_Heurtin who was finally able to learn to navigate the world. How was she impaired structurally and how did she overcomoe the abstract representation of the world? Modality even in modern massive llms with multi modal capabilites converge to the same represenations of the world internally even though the modality of input/output are different. What is it a property of? What happens when you train identical nns to learn the same data but from different modlaities and even alter the statistical relationship of data? What remains the same and what changes if the architecture can be made identical? What does it tell us about the represenation itself? Why can't we segregate the input modality from the representation itself like humans can? Is it the lack of modularity or the lack of understanding of what representations even are in high dimensional space?
8. Infant amnesia: At birth, a human child or any new born of an organism is remarkable at things it can do. It surely can't be a blank slate model in the brain. It needs some very veyr strong priors encoded. What's the trace and origin of each such prior for each organism? Why do different offsprings of different animals take different amount to learn to a skill? Why does a human baby take the most time to walk? Why is the development so slow? And during this development, the neurons are constantly rewiring itself such that babies hardly remember the first 5 years of their lives (in humans), why? Why not let them remember? What defines how a baby learns? (Podcast reference on learning in human babies is good resource to understand it)
9. Octopus intelligence and convergent evolution. Tracing the nature of intelligence across organisms that evolved it independently. Octopuses evolved complex cognition on a completely separate branch from vertebrates. What's shared and what's different tells you what's necessary versus contingent in the architecture of intelligence.
10. Emergence of language, writing, speech. What's the mandatory condition for tribal knowledge to compound rapidly? Is language causal to the acceleration of human intelligence or is it a consequence? Tracing the sequence — speech, writing, printing, internet — as successive compression and transmission layers for collective knowledge. And whether compounding is even the right frame.
11. CNN→ViT→genome compression pipeline. Evolution as brute force search. Genome as compressed extracted inductive biases, not a replay of the search. The missing mechanism for extracting and compressing learned biases from large-scale trained models into priors for sample-efficient next-layer systems. Organism-specific reality as the selection criterion for which biases get encoded. Your failed compression experiments as a data point that direct extraction doesn't work cleanly.
12. The human genome has been around for ~100,000 years. Back then we were still being hunted by animals. The exact same machinery, the exact same process, the exact same learning rate runs in a baby born today, except now it learns maths, learns to code, grows up and goes to space. A baby born today vs a baby born 100,000 years ago has had the same machinery, the same process, the same learning rate, and yet it has generalized to a world it was never built to see. The genome never got an example of any of this, was never selected against it, and still the thing it builds handles it trivially. Where does that generalization actually live? In the genome, in the development, or in the world being far more learnable than it looks?
13. And how do we even think about generalization in the first place. Take a person from the 1800s and tell them that one day they'll ride a machine at 60-80 miles an hour, controlling themselves how fast to go and how to steer, while 100s of other humans on similar machines share the same stretch of road, all of them deciding when to stop, when to move, how fast to go so nobody gets killed, and that they'll do all of it trivially, without even thinking about it. What would that person actually picture? Could they believe it's possible, or would they call you a lunatic? What does that pose about capacity, generalization and capability, when the person had the full machinery to do it and never knew the capability was even there? (Side note: in the 1800s, what's the fastest a human had ever moved? If the timeline isn't clean enough for it to feel like lunacy because trains already existed, push back a century or two and the point still holds.) And if the capability was latent in the architecture the whole time, just waiting for the world to hand it the problem, what's latent in us, and in our models, that simply hasn't been shown to it yet?

If you want the experiments, they're in a separate post: What if pretrained weights are a genome, not a brain?

A lot of the findings in the blog are actually known empirical/already proposed findings. A lack of citation should not be treated as the findings being novel. Infact, counterintuitively, almost nothing in this document at this point is novel unless explicity said so.

What follows turns the questions above into explicit hypotheses. An earlier version of this section was a long, confident write-up of a single framework. It was drafted by a language model, and it did two things I want to undo here: it stated other people's ideas as if they were mine, and it argued for a conclusion instead of generating hypotheses. So this is shorter. Each hypothesis names its source where I know it, and none of it is offered as settled or novel.

Hypothesis: pretraining is a search of the same kind as evolution

From question 4. Gradient descent and natural selection are both slow, sample-inefficient searches through a high-dimensional space for a structure that happens to fit some reality. The analogy is old, and I'm keeping it only to find where it breaks. If one training step is the analogue of one organism living or dying, then a pretrained model is the output of the search, not the thing the search was running in order to build. The question I actually care about is the exact point where that mapping stops holding, because that boundary is where "what's missing" should become visible.

Hypothesis: evolution encoded a genome, not a brain

Zador [1] argued that the genome doesn't store a finished brain. It stores wiring rules and a developmental program, and most of what a newborn animal can already do comes from that, not from learning within its lifetime. What I'd add is a guess about where pretrained weights sit in that picture: closer to the genome layer (a compressed set of priors) than to the brain. If that's right, deploying a pretrained model directly is like trying to run the genome as if it were the animal, and the developmental step that turns priors into an efficient learner is simply absent from how we build these systems.

Hypothesis: the useful artifact is compressed biases, not the full weights

From question 11. Evolution's search space dwarfs the genome it leaves behind, and the genome is not a recording of the search. It's a compression of what the search found. If pretraining is the search, the analogous object would be the extracted inductive biases, not the whole weight matrix. This is adjacent to distillation [2] and to the genomic-bottleneck framing above. The honest data point: my attempts to pull that compressed object directly out of trained weights (as a mask, as a low-rank factor) did not work cleanly. I'm treating that as evidence about the mechanism, not just a failed run. Whatever the genome is, it isn't recoverable by the naive extraction I tried.

Hypothesis: representation is a property of the reality, not the modality

From questions 3 and 7. Multimodal models tend to converge on similar internal representations regardless of input channel; Huh et al. [3] call this the Platonic Representation Hypothesis. It rhymes with the fact that a person blind and deaf from birth can still build a workable model of the same world. The hypothesis is that the representation is fixed by the structure of the reality being modeled, not by the channel it arrives through. If so, the open problem isn't modality at all; it's why we can't yet separate the representation from the input channel the way a brain apparently can. The clean test is identical architectures trained on the same underlying reality through different modalities, with the data statistics deliberately distorted, asking what stays invariant.

Hypothesis: the missing piece is development

From questions 5 and 8. The same gap shows up under most of the questions above. We have a search (pretraining) and we have a deployed artifact, and nothing in between that corresponds to development: the slow, rewiring process that builds a brain out of a genome and, in humans, overlaps with the window we don't remember. A newborn is not a blank slate, and neither, probably, are pretrained weights; but there's no stage in our pipeline that does what development does. I don't have a mechanism for it. I'm naming it as the load-bearing unknown, not claiming to have found it.

Status

These are hypotheses, not findings. Where an idea has a known origin I've named it: Zador [1] for the genome-versus-brain split, Huh et al. [3] for representational convergence, Hinton et al. [2] for distillation. The rest is speculation generated by the questions, which I expect to revise or drop. The experiments that would actually separate these are in the companion post, and so far they constrain the picture more than they support it.

---

References

1. Zador, A. M. (2019). A critique of pure learning and what artificial neural networks can learn from animal brains. Nature Communications, 10, 3770. doi:10.1038/s41467-019-11786-6

2. Hinton, G., Vinyals, O., & Dean, J. (2015). Distilling the knowledge in a neural network. arXiv:1503.02531

3. Huh, M., Cheung, B., Wang, T., & Isola, P. (2024). The Platonic Representation Hypothesis. arXiv:2405.07987