Using a case study of invariant prediction, we first highlight the importance of formally specifying the space of adverse events we’d like to handle at deployment time. This provides a mathematical framework for analyzing, comparing, and improving the robustness of ML algorithms. Then, we explore how careful experimental probing of these methods’ failures leads to a deeper understanding of the underlying causes, and how these insights can inform the design of new methods with more reliable real-world behavior. We conclude with a brief summary of other past and ongoing works towards provably secure ML, including a scalable framework which enables certified robustness to adversarial train- and test-time attacks.