Modern machine learning models are often vastly overparameterized, meaning that many different parameter configurations can solve the same learning problem equally well. Yet, in practice, training algorithms consistently converge toward particular solutions, a phenomenon called implicit bias. What drives this selection?
The key idea I will present is that many learning systems possess hidden symmetries and redundancies whose proper accounting leads to an explicitly computable correction of the loss that governs learning. This term acts as a hidden selection principle, explaining why learning algorithms consistently favor particular solutions among the many that achieve the same performance. Through examples ranging from matrix factorization to attention mechanisms, I will show how the same framework can be used to explain known biases and engineer desired ones.
Finally, I will briefly discuss how the same principles naturally extend beyond machine learning and suggest a new perspective on emergent behavior in physical and complex systems.