A simple physics-inspired model sheds light on how AI learns

Artificial intelligence systems based on neural networks—such as ChatGPT, Claude, DeepSeek or Gemini—are extraordinarily powerful, yet their internal workings remain largely a "black box." To better understand how these systems produce their responses, a group of physicists at Harvard University has developed a simplified mathematical model of learning in neural networks that can be analyzed mathematically using the tools of statistical physics.

"Toy models," like the one presented in a study published in the Journal of Statistical Mechanics: Theory and Experiment, provide researchers with a controlled theoretical laboratory for investigating the fundamental mechanisms of neural networks.

A deeper understanding of how these systems work could help design artificial intelligence systems that are more efficient and reliable, while also addressing some of the current challenges.

The laws of AI

It's a bit like when Kepler described the laws governing the motion of the planets. "The way Newton's laws of gravity were discovered was first by identifying scaling laws between the orbital periods of planets and their radii," explains Alexander Atanasov, a Ph.D. student in theoretical physics at Harvard University and first author of the new study.

Kepler formulated his laws by observing planetary motion, without fully understanding the mechanisms behind it. Yet that work proved crucial: it later enabled Newton to uncover gravity, leading to a much deeper understanding of the universe.

In studies of deep learning—the branch of artificial intelligence based on neural networks—we may still be in a similar Keplerian phase. Today, researchers have identified several empirical laws that describe how neural networks behave, but we still lack a kind of "theory of gravity" explaining why they behave that way.

Scientists, for example, know about the scaling laws. "We know that if we take a model and make it bigger, or give it more data, its performance increases," explains Cengiz Pehlevan, Associate Professor of Applied Mathematics at Harvard University and senior author of the study.

These laws make performance predictable, but they do not yet reveal the deeper mechanisms behind it. This approach is not only inefficient—today's AI systems consume enormous amounts of energy—but also does little to advance our understanding of how these systems actually work.

Neural networks as biological organisms

"Deep learning models are not algorithms written by hand as a set of rules. They're not engineered manually," explains Atanasov. "It's much more similar to an organism being grown in a lab."

Generative AI chatbots rely on neural networks, a technology that—in a very distant way—resembles the functioning of a biological brain. They consist of many small processing units, called artificial neurons, each performing simple operations but connected together in a complex network.

It is this networked structure that allows "intelligent" behavior to emerge. Although we know the mathematical operations performed by each individual component, predicting and mechanistically explaining the behavior of the system as a whole remains extremely difficult: as the number of components grows, the complexity increases rapidly.

A toy model

Since it is currently impossible to analyze a full-scale neural network with exact mathematical methods, Atanasov and his colleagues chose to work with a simplified model that still captures many key features of more complex systems.

"The model we're studying is simple enough to be solved mathematically," explains Jacob Zavatone-Veth, Junior Fellow at the Harvard Society of Fellows and co-author of the study. "At the same time, it reproduces several of the key phenomena seen in large neural networks."

The toy model used in the study is ridge regression, a variant of linear regression.

Linear regression is a statistical method used to estimate relationships between variables. For example, if we know the height and weight of 100 people, we can use linear regression to identify a mathematical relationship between the two and estimate the height of a new person based only on their weight.

The mystery of overfitting—and why it often doesn't happen

Ridge regression is a type of regression that helps reduce the phenomenon known as overfitting. When models are trained on large datasets, a neural network—a bit like a very diligent but perhaps not particularly insightful student—may end up simply memorizing the training data instead of learning patterns that allow it to generalize and make reliable predictions on new data.

Yet deep learning models often behave in a surprising way. "Despite being extremely large, these models can learn from the data without overfitting," explains Atanasov, calling it "one of the great mysteries of deep learning."

At first glance this seems counterintuitive. In theory, larger models should be more prone to overfitting. Instead, the scaling laws show that performance often improves as more data are used during training.

New insights

The new study offers one possible piece of that explanation. According to the researchers, the ability of neural networks to learn without overfitting may arise from principles related to renormalization theory, a framework widely used in statistical physics.

To see why, it helps to consider the dimensionality of the data processed by modern AI systems. In the earlier example of linear regression, we considered only two variables—height and weight.

Real systems such as ChatGPT, however, operate in spaces with thousands or even millions of variables, making an exact mathematical analysis extremely difficult.

Here ideas from statistical physics become useful. In very high-dimensional data, small random variations—known as statistical fluctuations—naturally appear. Renormalization theory shows that many microscopic details can be effectively absorbed into a small number of parameters, meaning that even very complex systems can display relatively simple large-scale behavior.

Using this framework and their simplified toy model, the researchers show how these high-dimensional fluctuations can actually stabilize learning rather than destabilize it.

"This is something we can understand by analyzing simpler linear models," explains Pehlevan, suggesting that the same mechanism may explain why current neural networks avoid overfitting even when they are highly over-parameterized.

The simplified model may also serve another purpose. As Zavatone-Veth notes, it could be a kind of baseline for understanding how learning might behave in very high-dimensional systems.

By studying a model that is simple enough to analyze mathematically, researchers can identify which aspects of learning are likely to be generic—that is, expected to appear across many different neural networks—and which instead depend on the details of a specific model. In this sense, studies like this may help clarify some of the more fundamental principles underlying learning in complex systems.

Provided by SISSA Medialab