Over the last 5-10 years, deep learning has transformed biomedical imaging, from enhancing acquisition to maximizing downstream utility of scans. My research group has been at the forefront of this revolution, developing novel methods that have laid the foundation for next-generation tools. As I will describe in my talk, much of this progress relies on predictive models and thus can be viewed as “curve-fitting” with general-purpose models. I will then show a few examples of recent work from my group, where we move beyond the curve-fitting paradigm and custom build models in ways that allow us to gain novel insights, understand how the output was computed, or empower the end-user to choose the solution best suited for their needs. These examples will be from a range of applications, including MRI reconstruction, image registration, and neural encoding with fMRI data.