The future of the Internet of Things (IoT) envisions very large networks of spatially distributed devices cooperating to solve common tasks in both industrial and urban environments. That vision fundamentally relies on the use of wireless communication across such networks to enable flexibility, mobility, and dynamic configurations. To help realize that vision, we discuss three related challenges in this dissertation. First, we present a constrained reinforcement learning framework for the joint design of control and communication policies in wireless control systems. As the scale of deployment of spatially distributed control systems grows, however, the performance of learned policies deteriorate. In the second part of this talk we then investigate the design of scalable and transferable resource allocation policies to govern communication in wireless control systems. We conclude the talk by generalizing that approach to large-scale distributed optimization problems and show that one can learn distributed optimization algorithms that match or exceed the performance of hand-crafted ones.