Systems that support expressive, situated natural language interactions are essential for expanding access to complex computing systems, such as robots and databases, to non-experts. Reasoning and learning in such natural language interactions is a challenging open problem. For example, resolving sentence meaning requires reasoning not only about word meaning, but also about the interaction context, including the history of the interaction and the situated environment. In addition, the sequential dynamics that arise between user and system in and across interactions make learning from static data, i.e., supervised data, both challenging and ineffective. However, these same interaction dynamics result in ample opportunities for learning from implicit and explicit feedback that arises naturally in the interaction. This lays the foundation for systems that continually learn, improve, and adapt their language use through interaction, without additional annotation effort. In this talk, I will focus on these challenges and opportunities. First, I will describe our work on modeling dependencies between language meaning and interaction context when mapping natural language in interaction to executable code. In the second part of the talk, I will describe our work on language understanding and generation in collaborative environments, focusing on learning to recover from errors and on continual learning from explicit and implicit user feedback.