Transformer language models are often assumed to behave as lossy compressors, with components such as nonlinearities and normalization layers believed to discard information. In this talk, we challenge this view by approaching the question from both theoretical and practical perspectives, showing that, as mappings from discrete token sequences to continuous hidden states, standard transformer-based LLMs are in fact injective and therefore lossless. We present a mathematical argument demonstrating that injectivity arises generically at initialization and is maintained throughout training, and pair this with extensive empirical evidence: across billions of collision tests on six state-of-the-art models, we find no instances of distinct inputs yielding identical internal representations. Building on these results, we introduce SIPIT, an algorithm that efficiently and provably reconstructs exact input text from hidden activations in linear time.
This is an online-only event. To participate, you are kindly invited to subscribe to our newsletter via form.