In recent years, system designers have increasingly been turning to heterogeneous systems to improve performance and energy efficiency.  Specialized accelerators are frequently used to improve the efficiency of computations that run inefficiently on conventional, general-purpose processors. As a result, systems ranging from smartphones to datacenters, hyperscalers, and supercomputers are increasingly using large numbers of accelerators (including GPUs) while providing better efficiency than CPU-based solutions.  In particular, GPUs are widely used in these systems due to their combination of programmability and efficiency.  Traditionally, GPUs are throughput-oriented, focused on data parallelism, and assume synchronization happens at a coarse granularity.  However, programmers have begun using these systems for a wider variety of applications which exhibit different characteristics, including latency-sensitivity, mixes of both task and data parallelism, and fine-grained synchronization.  Thus, future heterogeneous systems must evolve and make deadline-aware scheduling, more intelligent data movement, efficient fine-grained synchronization, and effective power management first-order design constraints.  In the first part of this talk, I will discuss our efforts to apply hardware-software co-design to help future heterogeneous systems overcome these challenges and improve performance, energy efficiency, and scalability.  Then, in the second part I will discuss how the on-going transition to chiplet-based heterogeneous systems exacerbates these challenges and my vision for extending our work to address these challenges in chiplet-based heterogeneous systems by rethinking the control plane.