Artificial intelligence systems powered by deep neural networks have achieved remarkable success across a broad range of applications. However, perturbations such as natural image corruptions or crafted malicious queries, can cause significant performance degradation. This poses severe risks in safety-critical applications, such as autonomous driving and clinical decision-making. A key vulnerability of machine learning models is their inability to handle data outside the training distribution or knowledge. When facing unseen or otherwise challenging inputs, models often make incorrect decisions without warning users.

This thesis improves the safety of machine learning systems by building three stages for handling challenging inputs: (1) rejecting unexpected inputs with an explanation, (2) providing statistical guarantees on rejection, and (3) enabling models to adapt to challenging inputs. We consider two distinct scenarios: models with known training distributions (e.g., in cyber-physical systems) where challenges are out-of-distribution data, and models with unknown training distributions (e.g., large language models in a multilingual context) where challenges are defined by standards like harmful content or deficits in knowledge across languages. We further investigate how to address challenging inputs for two clinical applications, autism diagnosis and acne classification.