AI/Cognitive/Science/Report/25/11/Understanding as Compression
Understanding as Compression: Academic and Philosophical Foundations in Human Cognition and Artificial Intelligence
Introduction
- The proposition that "understanding is compression and compression is understanding" is both a philosophical and computational thesis with deep roots in cognitive science, philosophy of mind, and artificial intelligence research. This report surveys foundational and contemporary academic literature that develops and supports the idea that understanding—whether in humans or machines—involves compressing complex information into simpler, more generalizable representations. Key theories and frameworks, such as Kolmogorov complexity, Minimum Description Length (MDL), the Information Bottleneck method, and the role of generative models in cognition and AI, are examined alongside philosophical analyses. Foundational and recent contributions from both theoretical and empirical perspectives are highlighted.
Foundational Theories Linking Understanding and Compression
Kolmogorov Complexity and Algorithmic Information Theory
- Andrey Kolmogorov, Ray Solomonoff, and Gregory Chaitin formalized the concept that the complexity of an object (such as a string or dataset) can be measured by the length of the shortest computer program that generates it. This framework, known as Kolmogorov complexity, equates understanding with the ability to compress data into the simplest possible form—intuiting the "program" behind the information. In cognitive terms, learning or generalization can be seen as discovering the shortest (most efficient) models that explain sensory data[1].
- Ray Solomonoff extended these ideas in his theory of inductive inference, arguing that prediction and understanding are achieved by inferring the simplest models consistent with observed evidence, thus formalizing the connection between understanding, prediction, and compression[2].
Minimum Description Length (MDL) Principle
- Jorma Rissanen advanced the Minimum Description Length principle, which treats inductive inference as the search for the model that enables the most concise encoding of observed data. MDL is both a statistical method and a theory of understanding: the best explanations are those that compress data most efficiently, balancing simplicity with the ability to account for observed complexity[3]. MDL has influenced both philosophical theories of science and practical approaches in machine learning.
Information Bottleneck Theory
- Naftali Tishby, Fernando C. Pereira, and William Bialek introduced the Information Bottleneck method, which formalizes the process of compressing observed data while preserving relevant information for prediction or task performance. This framework has been interpreted as capturing an essential aspect of "understanding": extracting salient, generalizable features while discarding extraneous detail—compression with a purpose[4]. Recent work has applied the Information Bottleneck to deep learning, connecting model compression to the emergence of generalizable representations[5].
Cognitive Science and Philosophy: Compression in Human Understanding
Predictive Processing and Generative Models
- Karl Friston's Free Energy Principle and related "predictive processing" models in cognitive neuroscience propose that brains are inference machines seeking to minimize the discrepancy between sensory input and internal generative models. These models allow the brain to compress complex sensory data into succinct, high-level representations (beliefs, concepts), providing a compelling neurocomputational account of human understanding as compression[6].
Philosophical Analyses: Abstraction, Parsimony, and Meaning
- Philosophers from Occam (Ockham) to modern thinkers have championed simplicity and parsimony as marks of true understanding. Occam's razor, for instance, prescribes favoring explanations that minimize unnecessary complexity, resonating with the computational imperative of compression[7].
- More recently, David Marr's levels of analysis—computational, algorithmic, implementational—emphasize that at the computational level, cognition is about representing and transforming information in the most efficient form[8].
- Daniel Dennett further argues that understanding involves "finding the pattern," which is fundamentally about compression—turning disparate data into a coherent, simplified structure capable of explanation or prediction[9].
Artificial Intelligence: Compression as a Route to Machine Understanding
Deep Learning, Neural Nets, and Model Compression
- In AI research, the success of deep learning has renewed interest in compression as a key to both efficiency and understanding. Geoffrey Hinton and colleagues have explored how deep networks learn compressed internal representations that support generalization and abstraction—hallmarks of understanding in machines[10].
- Recent theoretical work connects the Information Bottleneck principle with the learning dynamics of deep neural networks, suggesting that compression of intermediate representations is linked to better generalization and more "understanding-like" behavior[5].
AI and Universal Induction
- Solomonoff's theory of universal induction foreshadowed the idea that the ultimate intelligent system is the one that discovers the simplest generative model consistent with all observed data—a system that understands by compressing experience into programs or rules[2]. This idea has influenced both philosophical debates on machine understanding and technical approaches in AI.
Modern Integrative Perspectives
- Jeff Hawkins's "A Thousand Brains" theory of intelligence argues that understanding in both biological and artificial systems depends on building and sharing compressed models of the world across networks—representing knowledge as reusable patterns[11]. This meshes with the algorithmic perspectives above and provides a bridge between cognitive neuroscience and machine learning.
Dissent and Nuance: Limits and Critiques
- While the thesis that "understanding is compression" has robust theoretical and empirical support, some philosophers and scientists caution against full equivalence. Critics argue that not all forms of understanding reduce solely to compression; contextual, pragmatic, and normative elements (e.g., social meaning, embodied practice) may exceed information-theoretic accounts[12]. Nonetheless, compression remains widely acknowledged as a central (albeit not exclusive) component.
Conclusion
- Across philosophy, cognitive science, and artificial intelligence, understanding as compression has strong conceptual and empirical foundations. From Kolmogorov complexity and the Minimum Description Length principle to information bottlenecks in brains and neural networks, the act of understanding consistently appears as a process of compressing complexity—discovering simple, generalizable structures hidden in data. While debates persist about the full adequacy of information-theoretic models, their explanatory power and unifying elegance have established "understanding is compression" as a central and generative concept in both human and machine cognition.
Sources