Effective theories play a fundamental role in how we reason about the world. Although real physical processes are very complicated, useful models abstract away the irrelevant degrees of freedom to give parsimonious representations. In contrast, overly complex models can be difficult to evaluate, suffer from numerical instabilities, and may overfit data. They also obscure useful insights into the relationship among different physical systems. I use information geometry to explore the role of simplicity in scientific explanation. I interpret a multi-parameter model as a manifold embedded in the space of all possible data, with a metric induced by statistical distance. These manifolds are often bounded and very thin, so they are well-approximated by a low-dimensional, simple model. For many types of models, there is a hierarchy of natural approximations that reside on the manifold’s boundary. These approximations are not black-boxes. They remain expressed in terms of the relevant combinations of mechanistic parameters and reflect the physical principles on which the complicated model was built. They can also be constructed systematically using computational differential geometry, as I illustrate with examples from physics and systems biology