Despite impressive success in domains such as vision and language, machine learning is still far from reliable integration into many challenging real-world scenarios, such as healthcare, where the coverage of existing data and the ability to collect new, diverse data are limited. This talk focuses on mathematically formulating and addressing some of the challenges in data-driven decision-making systems, studied in the reinforcement learning (RL) framework. I will discuss decision-making based on two sources of data: historical (offline) data and actively-collected data. In learning from offline data, I first mathematically formulate the challenge of partial data coverage. I show that this formulation combined with pessimistic offline RL unifies the major offline learning paradigms: imitation learning and conventional offline RL. I then present statistically-optimal and practical offline RL algorithms that simultaneously exploit expressive models, such as deep neural networks, and historical datasets with any coverage, to learn good decision-making policies. In learning from interactive data, I present general formulations and theoretically-guaranteed algorithms that exploit problem structure and expressive models to collect data for learning good policies, with efficacy demonstrated in a variety of navigation and locomotion tasks.