Neural networks transform high-dimensional data into compact, structured representations that are often described as elements of a lower-dimensional vector space. In this talk, I will introduce an alternative perspective, interpreting neural models as dynamical systems acting on the representation manifold.
In particular, autoencoder models implicitly induce a vector field in the latent space through the iterative application of the encoding–decoding map, without requiring any additional training procedure. Common inductive biases used during training give rise to fixed points and attractors within this vector field, enabling a systematic analysis of the resulting dynamics. Viewing neural networks through this dynamical lens provides a novel framework for understanding both model behavior and data structure: the latent dynamics connects to key phenomena such as generalization and memorization, allows the extraction of prior knowledge encoded in model parameters directly from attractors without relying on input data, and offers a way to characterize out-of-distribution samples via their trajectories in latent space. Finally, I will discuss how this perspective might be extended beyond autoencoders to broader classes of models, and how it can inform the design of improved sparse autoencoder models. Taken together, this direction suggests new ways to understanding how neural networks process, organize, and generalize information.