Social learning helps humans and animals rapidly adapt to new circumstances, coordinate with others, and drives the emergence of complex learned behaviors. What if it could do the same for AI? This talk describes how Social Reinforcement Learning in multi-agent and human-AI interactions can improve coordination, learning, generalization, and lead to the development of agents better able to anticipate and serve human needs. I propose a unified method for improving coordination and communication based on causal social influence. Beyond coordination, I demonstrate how multi-agent training can be a useful tool for improving learning and generalization even in the single-agent setting. I present PAIRED, in which an adversary learns to construct training environments to maximize regret between a pair of learners, leading to the generation of a complex curriculum of environments that improve both learning and zero-shot generalization. Ultimately, the goal of my research is to create intelligent agents that can assist humans with everyday tasks; this means interacting effectively with humans, and learning from human-AI interactions. I show that learning from human social and affective cues scales more effectively than learning from manual feedback. However, it depends on accurate recognition of such cues. Therefore I will discuss how to dramatically enhance the accuracy of affect detection models using personalized multi-task learning to account for inter-individual variability. Together, this work argues that Social RL is a valuable approach for developing more general, sophisticated, and cooperative AI, which is ultimately better able to serve human needs.