Fall 2025 GRASP on Robotics: Alan Yuille, Johns Hopkins University, "3D Vision Language Models and Interactive World Models"
2025-10-24 10:30 - 11:45 (America/New_York) | Wu and Chen Auditorium (Room 101), Levine Hall, 3330 Walnut Street, Philadelphia, PA, 19104, United States
This event will be in-person ONLY in Wu and Chen Auditorium.
ABSTRACT
Vision Language Models (VLMs) are extremely successful, but their performance degrades when asked questions involving spatial relations and 3D world knowledge. Inspired by Cognitive Science, we develop 3D VLMs which are 3D-aware and 3D-explicit to help us to diagnose their failure nodes. We present two approaches which involve developing datasets with 3D annotations for training the 3D VLMs. The first works was developed on realistic-synthetic datasets and the 3D VLM is built on a 3D Image Parser. This 3D VLMs significantly outperform conventional VLMs for questions involving 3D/6D (Xingrui Wang et al. CVPR 2025 highlight) and physical reasoning (Xingrui Wang et al., ICLR 2025). This work is extended to complex images taking VLMs as base models and evaluated on a 3D comprehensive reasoning benchmark (W. Ma et al. ICCV 2026). We develop a 3D-VLM which significantly outperforms conventional VLMs when asked questions requiring 3D knowledge (Wufei Ma et al. CVPR 2025 highlight). We further extend this approach to develop a 3D-VLM which performs even better and is also 3D-explicit (Wufei Ma et al. NeurIPS. 2025). We discuss the bigger picture which involves the need for world models as illustrated by (J. Chen et al. ICLR 2025), analysis by synthesis (T. Zheng et al. NeurIPS 2025), and early detection of cancer using radiology reports (P. Bassi et al. MICCAI 2025).

PICS Colloquium: Powering decarbonization with modeling and optimization of renewables in the multi-scale atmosphere with Michael F. Howland
2025-10-24 14:00 - 15:00 | PICS Conference Room 534 – A Wing , 5th Floor, 3401 Walnut Street, Philadelphia, PA, 19104, United States
To meet net-zero carbon emissions targets by mid-century, up to a ~30-fold increase in wind power capacity is required. Acceleration to this rate requires urgent improvements to efficiency and reliability of installed wind farms, as well as cost reductions for future offshore farms. To expand energy production, wind turbines are rapidly increasing in size, wind farms are proliferating to new locations and are increasing in size and siting density, and novel wind farm design and control methods are increasingly deployed. But current engineering models driving wind power design and control rely on idealized theory that neglects key aspects of the rotor aerodynamics and the turbulent atmospheric boundary layer, which are increasingly important for larger turbines and farms. We revisit the first-principles of mass, momentum, and energy conservation to develop a Unified Momentum theory for rotors across operating regimes, accounting for arbitrary misalignments between rotor and inflow and thrust coefficients. The model is validated against large eddy simulations and generalizes and replaces both classical momentum theory and the Betz limit. Going from the scale of a turbine to a farm, wake losses can reduce farm energy by 30%, a significant loss that negatively impacts economics and is increasing given wind power expansion. Using large eddy simulations of wind turbines operating in a range of atmospheric conditions, we systematically uncover the significant roles of Coriolis effects and stability on wake recovery, trajectory, and morphology. A new fast-running wind farm model that accounts for the coupled rotor operational and atmospheric effects on wakes is developed. The wind farm model is leveraged for applications including collective control and for control co-design, applied in both simulations and utility-scale field experiments. Collective control can increase the energy generation of wind farms through software modifications, without additional turbines or hardware. Going from the scale of a wind farm to the energy system, we leverage an integrated climate and energy system modeling framework to design minimum-cost decarbonized energy systems. Energy system optimization with high-resolution atmospheric predictions reveals opportunities for complementarity between spatiotemporal variations in wind and solar supply to align with energy demand and to lower the cost of decarbonized energy systems.

MEAM Seminar: “Manipulating Mechanical Wave Propagation with Phononic Materials”
2025-10-28 10:15 - 11:15 | Wu and Chen Auditorium (Room 101), Levine Hall, 3330 Walnut Street, Philadelphia, PA, 19104, United States
One grand challenge for materials and structures design is to satisfy multiple conflicting requirements. For example, energy infrastructure, especially those in remote and extreme environments such as offshore wind turbines and nuclear reactors, requires components to operate effectively over long time periods and avoid catastrophic failures. Structural materials in aviation must be lightweight but high in strength, stiff while dampening out harmful vibrations, survive damaging impact events, and interact with complex flows in non-detrimental ways. On smaller length scales, acoustic and ultrasonic sensors require specific frequency and dissipative responses, and need to detect wavelengths that are much smaller than their physical size. This talk focuses on a common theme to these critical engineering problems: understanding how mechanical waves interact with engineered materials across different length and time scales. In particular, the field of phononic materials studies how engineering micro- and meso-scale features in materials and structures can prescribe the frequency and spatial properties of acoustic waves. Features such as spatial periodicity of the material or geometry, resonant inclusions, and nonlinearities can lead to wave propagation and modal properties not found in natural materials. New wave propagation phenomena have been discovered in these material platforms, which has been a direct result of an interdisciplinary research approach, integrating additive manufacturing, acoustics, mechanics, materials science, and design. This presentation will discuss our group’s recent research in phononic materials, focusing on (1) effects of nonlinearity on wave propagation in phononic materials, and (2) applications of phononic materials to passive flow control, using reduced order models, finite element simulations, and experiments.

CIS Seminar: "Sparse View Synthesis"
2025-10-28 15:30 - 16:30 | Wu & Chen Auditorium
We seek the ability to take a few images of a scene of interest, and turn it into an immersive visual experience, where one can explore it from different viewpoints, in effect visualizing a 3D representation of an object, scene or photograph, and providing numerous applications in augmented reality, e-commerce and 3D photography. This problem, known as view synthesis or image-based rendering in computer vision and graphics, has a three-decade plus history, and is currently undergoing a renaissance with new representations of 3D geometry enabling unparalleled realism. We discuss some of the history in terms of capturing the light field (the space of light rays for any spatial position and viewing direction), and our own work on a sampling theory for view synthesis based on light fields, leading to the development of volumetric radiance fields as a fundamentally new approach to representing 3D geometry for view synthesis. We will also discuss parallels to Monte Carlo and volumetric rendering and simulation problems in computer graphics. We then ask the question of how far we can push the required number of images, in order to achieve sparse view synthesis with very few images, in the limit only one photograph. In this context, we also discuss our recent results on a number of applications including real-time live portraits, generative AI for 3D scenes, and differentiable light transport for inverse rendering.

ASSET Seminar: “​When Is a Conformal Set, a Conformal Set?”
2025-10-29 12:00 - 13:15 | Amy Gutmann Hall, Room 414, 3333 Chestnut Street, Philadelphia, 19104, United States
The two most popular vehicles for communicating uncertainty in the estimates of an unknown quantity are confidence sets and conformal sets. The set produced and its corresponding probability guarantee (conditional on the feature vector) depend upon assumptions the analyst has made about the underlying data generating process. For example, are the residuals independent and normally distributed? Is the data generating process exchangeable? Knowing only the underlying data upon which a model was built, how should one assess the validity of a proffered confidence or conformal set? Second, how should one use these sets to inform decision making?

Zoom: https://upenn.zoom.us/j/95189835192

Fall 2025 GRASP SFI: Roberto Martín-Martín, University of Texas at Austin, “Making Mobile Manipulation Real: New Learning Paradigms for Robots”
2025-10-29 15:00 - 16:00 | Levine 307, 3330 Walnut Street, Philadelphia, PA, 19104, United States
This is a hybrid event with in-person attendance in Levine 307 and virtual attendance via Zoom.
ABSTRACT
Most tasks people wish robots could do (fetching objects across rooms, assisting in the kitchen, tidying) require mobile manipulation, the integration of navigation and manipulation. While robots have made remarkable progress in each skill independently, bringing them together sequentially (navigate>manipulate>navigate...) or simultaneously (coordinating base and arm motion to open a fridge or wipe a table) remains one of the hardest challenges in robotics. The difficulty lies not only in mastering two complex capabilities, but in coupling them safely and efficiently, over long horizons, under uncertainty, and in contact‑rich settings. These conditions often break the assumptions of standard imitation and reinforcement learning, which tend to struggle to generalize, train safely, and anticipate, learn from, and recover from errors in unstructured environments. I’ll present three learning algorithms from my lab designed specifically for mobile manipulation: methods that extract skills from in-the-wild human video (SafeMimic), learn structured action spaces that make RL sample-efficient on real robots (SLAC), and integrate memory mechanisms with foundation models to reason over extended tasks (Bumble). Our latest results demonstrate multi-step single-video imitation, surface-wiping RL on wheeled mobile manipulators trained in real world under one hour, and broad task generalization to novel objects in building-wide scale with improved trial efficiency. I’ll close with an analysis of failures and limitations and a roadmap for scaling: toward robots with the adaptability, safety, and fluency needed to make learning mobile manipulation an easy and reliable part of everyday life.

CBE Seminar: "Computational Approaches for Understanding and Engineering Biomolecular Condensates” (Jerelle Joseph, Princeton University)
2025-10-29 15:30 - 16:30 | Wu & Chen Auditorium
Abstract:
Biomolecular condensates are membraneless compartments inside living cells that play critical roles in health and disease. Over the past two decades, a wide body of work has established that these compartments form through phase separation of molecules such as proteins and RNA. This discovery has sparked significant interest in uncovering the molecular factors that drive intracellular phase separation and in understanding how condensate material properties are encoded. In parallel, condensates offer a versatile platform for engineering novel functions and preventing aberrant behaviors. To advance our ability to understand and engineer condensates, our group develops computational models and molecular simulation approaches that enable us to zoom in on condensates and uncover how molecular interactions give rise to their complex behaviors, while also supporting simulation-driven condensate design. Specifically, by combining atomistic simulations with experimental data, we design residue-resolution force fields that balance the speed and accuracy needed to model the dynamics of these multicomponent structures. In this talk, I will discuss recent developments on this front.

MSE Seminar: "Hybrid Quantum-Classical Computing - Useful Quantum Devices in the NISQ Era" - Burns Healy - Dell Technologies
2025-10-30 10:30 - 12:00 | Wu and Chen Auditorium (Room 101), Levine Hall, 3330 Walnut Street, Philadelphia, PA, 19104, United States
After an introduction to quantum computing, we discuss the currently practiced framework of “hybrid” computing, which is a method of accelerating traditional High Performance Compute (HPC) with Quantum Processing Units. I will talk about some of the ways in which Dell’s research office has implemented this idea and the value it can bring to modern compute workloads, including specific applications to materials science. Then I will do a deeper dive into a particular algorithm I wrote using HQCC to improve results for Dell’s supply chain team.

ESE Fall Seminar - "From Circuits to Cognition: Silicon for Embodied Intelligence"
2025-10-30 11:00 - 12:00 | Raisler Lounge (Room 225), Towne Building, 220 South 33rd Street, Philadelphia, PA, 19104, United States
The next generation of intelligent and autonomous systems requires not only novel devices but also new silicon architectures and design workflows that transcend conventional approaches to deliver real-time learning, perception, and decision-making under severe power and resource constraints. In this talk, I will outline a cross-layer methodology for architecting silicon for embodied AI, from workload characterization and benchmarking to architecture exploration, compiler integration, and system prototyping. Central to this effort are compute-in-memory accelerators, mixed-signal neuromorphic architectures, and memory-centric SoCs that integrate hybrid RRAM/SRAM arrays, all designed within workflow frameworks that couple algorithmic needs with hardware capabilities. Case studies will highlight reconfigurable streaming-dataflow architectures for reasoning and decision-making, heterogeneous SoCs optimized for autonomous workloads, and bio-mimetic silicon platforms for navigation and planning. By unifying design flows, benchmarking, and circuit innovation, this work illustrates how silicon architectures can be systematically engineered to achieve the transparency, energy efficiency, and adaptability demanded by embodied intelligence and autonomy.

FOLDS seminar: Weak to Strong Generalization in Random Feature Models
2025-10-30 12:00 - 13:00 | Amy Gutmann Hall, Room 414, 3333 Chestnut Street, Philadelphia, 19104, United States
Zoom link: https://upenn.zoom.us/j/98220304722

Weak-to-Strong Generalization (Burns et al., 2023) is the phenomenon whereby a strong student, say GPT-4, learns a task from a weak teacher, say GPT-2, and ends up significantly outperforming the teacher. We show that this phenomenon does not require a strong and complex learner like GPT-4, nor pre-training. We consider students and teachers that are random feature models, described by two-layer networks with a random and fixed bottom layer and trained top layer. A ‘weak’ teacher, with a small number of units (i.e. random features), is trained on the population, and a ‘strong’ student, with a much larger number of units (i.e. random features), is trained only on labels generated by the weak teacher. We demonstrate, prove and understand, how the student can outperform the teacher, even though trained only on data labeled by the teacher, with no pretraining or other knowledge or data advantage over the teacher. We explain how such weak-to-strong generalization is enabled by early stopping. Importantly, we also show the quantitative limits of weak-to-strong generalization in this model.
Joint work with Marko Medvedev, Kaifeng Lyu, Dingli Yu, Sanjeev Arora and Zhiyuan Li.

BE Seminar - Celeste M. Nelson "Mitochondria, mechanics, and tissue morphogenesis"
2025-10-30 15:30 - 16:30 | 216 Moore Building

CIS Seminar: "Inverse Problems using Generative Priors"
2025-10-30 15:30 - 16:30 | Wu and Chen Auditorium (Room 101), Levine Hall, 3330 Walnut Street, Philadelphia, PA, 19104, United States
Inverse problems seek to recover an unknown source signal X for which we have indirect, partial, or noisy measurements Y. Most real-world inverse problems are ill-posed and the conventional line of attack has been to assume some structure (or prior) on X. Unfortunately, priors are not always available and often challenging to model mathematically. Generative models are powerful tools that learn patterns from data, hence a new opportunity to obtain samples from the priors of X. Given such a prior sample, it may be possible to compute how well the sample explains the measurement Y, and iteratively guide the denoising process to generate samples from the posterior p(X|Y). This paradigm of posterior sampling is unlocking a wide range of applications that—before the deep learning era—were facing performance walls. This talk will introduce the core ideas in this paradigm, generalize the framework, and show how this framework can be applied to multiple applications, including unsupervised speech separation, zero-shot human pose tracking, and inverse path planning.

Fall 2025 GRASP on Robotics: Kris Hauser, University of Illinois at Urbana-Champaign & Samsung Research America, “Modeling and Reasoning About 'Stuff'”
2025-10-31 10:30 - 11:45 | Wu and Chen Auditorium (Room 101), Levine Hall, 3330 Walnut Street, Philadelphia, PA, 19104, United States
This event will be in-person ONLY in Wu and Chen Auditorium.
ABSTRACT
Prevailing models in robotics reason about the world either as images (end-to-end learning approaches) or as a collection of rigid objects (classical approaches), but neither have proven to be suitable abstractions for manipulating cloth, ropes, piles of objects, plants, and natural terrain. My lab is investigating novel representations of “stuff” that are built de novo from visual and tactile perception data, whose properties are learned continuously through interaction. Volumetric Stiffness Fields, Graph Neural Networks, Neural Dynamics, and 3D metric-semantic maps are examples of models that allow robots to learn about their environment without having preconceived notions of individual objects, their physical properties, or how they interact. For a variety of domains and materials, these techniques are able to model complex interactions, uncertainty, and multi-modal correlations between appearance and physical properties. Applications will be shown in agriculture, construction, and household object manipulation.
(This talk solely represents the research and opinions of Dr. Hauser under his UIUC affiliation, and does not communicate any results, statements, or opinions on behalf of Samsung Research America, Samsung Electronics, or any of its affiliates.)

Fall 2025 GRASP Seminar: Aljoša Ošep, NVIDIA, "Segmenting More Than Meets the Eye: Towards Amodal 4D Segmentation"
2025-11-03 13:00 - 14:00 | Amy Gutmann Hall, Room 306, 3317 Chestnut Street, Philadelphia, PA, 19104, United States
This is an in-person event ONLY in AGH 306.
ABSTRACT
The future of AI is embodied — imagine intelligent agents that can navigate and manipulate the world, from robot assistants helping around the home to autonomous vehicles taking you anywhere safely. To act in the physical world, these agents must do more than process raw sensory inputs: they must reason about the underlying, dynamic world that gives rise to their observations. This requires a 4D scene model — 3D geometry evolving over time — that is object-centric, predictive, and grounded in language. Such representations enable agents to answer questions like “Where am I?”, “What is around me?”, and “How can I interact with this object?”
In this talk, I will advocate for an explicit, amodal representation of world geometry and objects learned from unlabeled sequences. Such a model supports robust perception in dynamic environments and enables language-driven interaction with the world. I will outline a blueprint for building such systems, centered around two complementary components: a slow video object mining method that discovers and tracks arbitrary objects in unlabeled videos, and a fast feed-forward network that learns from these tracks to detect, segment, complete, and forecast object trajectories.
I will trace the progression from early methods for self-supervised object discovery and detection, to recent models capable of promptable, text-conditioned 4D segmentation and amodal scene completion. Taken together, these components form a scalable recipe for learning object-centric 4D representations directly from raw video — a step toward grounded, general-purpose world understanding.

Tedori-Callinan Distinguished Lecture: "Robotic Predictions are Hard, Especially About the Future"
2025-11-04 10:15 - 11:15 | Wu and Chen Auditorium (Room 101), Levine Hall, 3330 Walnut Street, Philadelphia, PA, 19104, United States
Many autonomous systems (e.g, driverless cars and drones) must make decisions based on predictions of the future actions of other nearby agents, whose dynamics and intentions are unknown. E.g., autonomous cars must predict the motions of surrounding vehicles, pedestrians and bicycles. Autonomous racing drones must avoid crashing into other drones on the race course. Unfortunately, only partial and noisy data on the motions of these potential hazards are available. This talk will introduce a novel method to approximate, in real-time, a predictive Koopman operator for each potential hazard from noisy data, quantify the uncertainty of the future predictions, and use the quantified predictions to provide probabilistic collision avoidance guarantees within a real-time model predictive control framework. Experiments with ground robots, a drone, and a semi-autonomous crane on an ocean going vessel will illustrate the ideas.

ESE Fall Seminar - "Diamond and GaN: Wide-Bandgap Allies for Thermal and Power Management from Devices to 3D-Stacked Chips"
2025-11-04 11:00 - 12:00 | Raisler Lounge (Room 225), Towne Building, 220 South 33rd Street, Philadelphia, PA, 19104, United States
Once considered exotic, diamond and gallium nitride (GaN) have become practical enablers for next-generation electronic systems. Their convergence—diamond providing exceptional thermal conductivity and GaN delivering high-efficiency power conversion—lays the groundwork for integrated thermal–power co-design. As computing, RF, and high-performance systems push toward higher power densities, conventional packaging and cooling approaches struggle to manage buried hotspots and multilayer bottlenecks. In this talk, I’ll share our journey that began in 2015 with an unconventional idea—integrating thin-film polycrystalline diamond directly onto GaN high-electron-mobility transistors (HEMTs) while preserving their functionality. This effort led to some of our most significant findings, including the development of a low-temperature (400–500 °C), back-end-of-line (BEOL)–compatible diamond growth platform, now extended to silicon, oxides, and nitrides. Our “all-around” diamond-integrated GaN HEMTs achieved an average channel-temperature reduction of ~70 °C at 25 W/mm (DC) (IEDM ’22, ’23), while workload-representative, heater-based experiments demonstrated nearly a tenfold reduction in temperature rise within 3D architectures (IEDM ’24). In collaboration with Prof. Mitra’s team, we are advancing the thermal scaffolding paradigm for 3D chips—a concept that merges materials innovation with architectural design. It is exciting to build upon nearly two decades of GaN and diamond research—dating back to my Ph.D. work on vertical GaN transistors—and to see it evolving toward compact, energy-efficient, and thermally optimized electronics for the AI datacenter era. Much of our research has been carried out in close collaboration with industry partners, and some of our GaN efforts have already transitioned into industrial applications. I will also share some of the key lessons learned along the way, as well as the challenges that continue to shape this evolving field.

Penn AI Seminar Featuring Li Shen: Harnessing Trustworthy AI and Informatics for Dementia and Aging Research
2025-11-04 12:00 - 14:00 | Amy Gutmann Hall, Room 414, 3333 Chestnut Street, Philadelphia, 19104, United States
Alzheimer’s disease and related dementias (ADRD) remains a major health crisis with profound social and economic burdens. Innovative strategies are needed to identify genetic risk and protective factors, model disease mechanisms, and accelerate therapeutic discovery. Advances in trustworthy AI and informatics now enable the integration of multimodal genetics, omics, imaging, and outcome data from large biobanks, creating powerful opportunities for biomarker and gene discovery beyond categorical diagnoses. At the same time, generative AI and large language models (LLMs) extend these capabilities to text-rich sources such as the scientific literature, clinical notes, and caregiver narratives. When integrated with knowledge graphs, LLMs can dynamically retrieve and synthesize domain-specific knowledge.

CIS Seminar: "Good Old Fashioned Engineering Can Close the 100,000 Year “Data Gap” in Robotics"
2025-11-04 15:30 - 16:30 | Wu & Chen Auditorium
AI is rapidly advancing the way we think, but we live in a material world. We still need to move things, make things, and maintain things. We need AI-driven robots to support an aging human population that doesn’t have enough workers. Large vision-language models based on internet-scale data can now pass the Turing Test for intelligence. In this sense, data has “solved” language and many claim that data has solved speech recognition and computer vision.
Will data also solve robotics? Rich Sutton points out in the “Bitter Lesson” that data and black-box “end-to-end” models have surpassed all the best-laid analytic work in AI. I accept that this trend will eventually produce general-purpose robots.
But the question is: when?
Using commonly-accepted metrics for converting word and image tokens into time, the amount of internet-scale data (texts and images) used to train contemporary large vision language models (VLMs) is on the order of 100,000 years – it would take a human that long to read or view it [2]. However, the data needed to train robots must combine video with robot motion commands: that data does not yet exist.
One way to collect robot data is teleoperation – where human “trainers” use remote control devices to painstakingly choose every motion of a robot as it performs a task – like folding a towel – over and over again. This is a variant of puppeteering, an ancient artform, that requires extensive human skill and patience. Unlike puppets however, robot joint angles can be precisely recorded, so the exact position history of each motor can be combined with videos from cameras that record the scene from different angles. The data for each “trial” or “trajectory” from start to finish includes a few minutes of video and the position history of all robot motors. Many companies are gearing up with fleets of robots and humans to collect data this way.
However, the largest such dataset reported so far is on the order of 1 year of data (it was collected in under a year by many human-robot systems). This data has been used to train large models and initial results are intriguiging. But this suggests that at current data-collection rates, a general-purpose robot, based on a ChatGPT-sized set of robot data, will be available in…100,000 years.
So how can we close this 100,000-year “Data Gap”?
Researchers are actively pursuing 2 additional methods for generating robot data: simulation and 3D analysis of internet videos.
Digital simulation today looks incredibly life-like – consider the special effects in action movies and the deepfakes generated by AI. It’s relatively easy to create life-like simulations of robot drones flying or robot dogs walking down stairs and doing backflips. Simulations can also provide videos and motor data to train large robot models. Simulation data works well for robots that fly or walk, or even for doing backflips. But it turns out that simulation is notoriously unreliable for robot manipulation.
This Sim2Real “gap” arises because physical manipulation involves precise and changing contacts between the edges and surfaces of objects and grippers, very small but important material deformations, and very nuanced and changing frictional forces due to microscopic surface variations. These factors are extremely difficult to measure and to accurately model. But these very small errors result in simulation data that looks realistic but is physically inaccurate. A submillimeter inaccuracy can make the difference between carrying a glass of water and dropping it. Robots trained on simulation data can work well in simulation but they often fail when manipulating physical objects. Researchers agree that physically-accurate simulation of manipulation is a Grand Challenge.
The third potential source of robot data is videos on the Internet. YouTube includes about 35,000 years of videos. Many of these videos show people manipulating objects, cooking, stacking cups, folding laundry. However, it is extremely difficult to extract precise 3D motion from 2D videos. Computer vision researchers can approximately track the motion of human hands and objects in a video, but the same issues of noise and precision make data from videos unreliable for robot learning. Accurately “lifting” a video image back into 3D to recover precise finger and object motions is a Grand Challenge for computer vision that is not expected to be solved in the forseeable future.
There is a 4th option.
Robot data can be collected from real robots working with real objects in real environments. Industry has thousands of robots doing useful work around the clock. Today, little of this real robot production data is saved. This is partly because most industrial robots perform extremely repetitive tasks like automotive welding and spot-painting that do not vary much. Data to train large models is often diverse – think of the massive range of texts and images on the internet. General-purpose robots need a broad range of data with variations in tasks, objects, and environments.
But real general-purpose robots don’t exist yet, so we can’t collect real robot data from them.
One option is to bootstrap, starting with specific tasks like driving or e-commerce package sorting, where the objects vary but the task and environment don’t vary much, and gradually expanding as specific skills are mastered into adjacent skills. Some companies are developing such robots and putting them to work.
One example is Waymo, which has robot taxis operating in several US cities. These robots have “level 4” autonomy – they rely on human operators who log in remotely to guide robot taxis when unfamiliar circumstances arise.
Another example is Ambi Robotics, which has package sorting and stacking machines operating in postal and warehouse facilities. These robots are fully autonomous – but a few times an hour they drop a package. As with Waymo, human operators help out in such cases.
Both Waymo and Ambi have created a “data flywheel”, where working robots constantly collect data that is used to improve robot performance and to enable adjacent robot skills, like highway merging for Waymo and package stacking (very different from sorting) for Ambi.
One thing that Waymo and Ambi also have in common is that they don’t rely only on “end-to-end” AI models. These companies combine advances in AI and learning from data with rigorous engineering methods like inverse kinematics, 6d motion planning, and digital signal processing.
I call this GOFE (Good Old-Fashioned Engineering). GOFE was developed long before modern AI. GOFE is based on modularity, metrics, and step-by-step algorithms based on geometry and physics that can be fully understood and often guaranteed to perform reliably. GOFE includes Kalman Filters, RANSAC outlier rejection, PID and MPC controllers, etc [3].
Whereas “end-to-end” AI methods are “model-free”, GOFE is model-based. GOFE segments problems into modules, so that each module can be tested, fixed, or fine-tuned independently, and replaced when a better module becomes available. Model-free methods can be combined with model-based methods to “kickstart” robots to achieve the levels of reliability required for adoption in real commercial environments, where they can then begin generating real robot data. I’ve been told such a combination is what’s behind the current success of Waymo, and I know that a combination of model-free and GOFE is behind the success of Ambi. Waymo’s robot taxis are collecting vast amounts of real data, and over the past 4 years, Ambi has collected 22 years of real robot data as they have sorted over 100 million real packages [4].
As noted at the beginning, I don’t disagree with Rich Sutton – I believe that model-free AI will eventually surpass GOFE and that general-purpose robots will be common at some point in our future. I look forward to that future and hope I get to see it.
But when will the general-purpose robots arrive? I’m not sure that the public (or investors) are willing to wait very long. For the next few years, the safest bet for closing the 100,000 year data gap is to get real robots into production by combining GOFE with model-free methods. These real robots can collect data as they perform useful work such as taxi driving and sorting packages. That high-quality data will improve their performance and enable robots to perform adjacent skills, spinning up the data flywheel until it collects enough data to enable general-purpose robots.

ASSET Seminar: "The coverage principle in language models: From pre-training to test-time scaling"
2025-11-05 12:00 - 13:15 | Amy Gutmann Hall, Room 414, 3333 Chestnut Street, Philadelphia, 19104, United States
Test-time compute has emerged as a new axis for scaling language model capabilities, yet we lack a principled understanding of this paradigm. What are the right algorithms and trade-offs for test-time scaling? What properties of the pre-trained model enable it? And can we better align pre-training recipes for test-time success? This talk addresses these questions through a unified lens of coverage. We first show that test-time scaling strategies like best-of-N sampling succeed if and only if the pre-trained model has coverage over high-quality responses. We then demonstrate that coverage, and hence best-of-N performance, can be improved through deliberate exploration, either purely at test time or via RL-style post-training. Finally, we ask why pre-training via next-token prediction yields models with good coverage in the first place. We uncover a rich theoretical landscape driven by an implicit bias of the next-token prediction objective, while also identifying a fundamental misalignment between next-token prediction and coverage, raising the possibility of future algorithmic innovations.

Zoom: https://upenn.zoom.us/j/95189835192
Passcode: [REDACTED]

Fall 2025 GRASP SFI: Huy Ha, Columbia University & Stanford University, “UMI on Legs: Making Manipulation Policies Mobile with Manipulation-Centric Whole-body Controllers”
2025-11-05 15:00 - 16:00 | Levine 307, 3330 Walnut Street, Philadelphia, PA, 19104, United States
This is a hybrid event with in-person attendance in Levine 307 and virtual attendance via Zoom.
ABSTRACT
We introduce UMI-on-Legs, a new framework that combines real-world and simulation data for quadruped manipulation systems. We scale task-centric data collection in the real world using a hand-held gripper (UMI), providing a cheap way to demonstrate task-relevant manipulation skills without a robot. Simultaneously, we scale robot-centric data in simulation by training whole-body controller for task-tracking without task simulation setups. The interface between these two policies is end-effector trajectories in the task frame, inferred by the manipulation policy and passed to the whole body controller for tracking. We evaluate UMI-on-Legs on prehensile, non-prehensile, and dynamic manipulation tasks, and report over 70% success rate on all tasks. Lastly, we demonstrate the zero-shot cross-embodiment deployment of a pre-trained manipulation policy checkpoint from prior work, originally intended for a fixed-base robot arm, on our quadruped system. We believe this framework provides a scalable path towards learning expressive manipulation skills on dynamic robot embodiments.

MSE David P. Pope Distinguished Lecture: Ion Migration and Its Impact on the Stability of Halide Perovskite Solar Cells - Prashant Kamat - University of Notre Dame
2025-11-06 10:30 - 12:00 | Wu and Chen Auditorium (Room 101), Levine Hall, 3330 Walnut Street, Philadelphia, PA, 19104, United States
The ability to tune the bandgap of metal halide perovskites by alloying different halide ions is key to advancing tandem solar cells and light-emitting displays. However, this compositional flexibility also introduces challenges, most notably, the photoinduced migration of halide ions, which can degrade device performance. A prominent manifestation is photoinduced phase segregation in mixed-halide perovskites (MHPs), leading to the formation of iodide-rich and bromide-rich domains. These inhomogeneous regions act as charge carrier traps, reducing device efficiency. The thermodynamic and redox characteristics of halide perovskites create a strong driving force for hole trapping and the oxidation of iodide species. As a result, the mobility of halide ions and their vulnerability to hole-induced oxidation are major factors governing the long-term stability of perovskite solar cells.

Surface passivation of 3D perovskites using 2D perovskites, carbazole derivatives has been reported widely. However, interfacial chemistry can pose significant challenges during long-term solar cell operation. Under light and heat, cation migration can substantially alter the 2D/3D interface, leading to performance degradation. Therefore, suppressing both halide and cation migration is essential for enhancing the long-term stability and efficiency of perovskite solar cells and light-emitting devices.
Suggested Readings:
DuBose, J. T.; Kamat, P. V. Hole Trapping in Halide Perovskites Induces Phase Segregation, Accounts of Materials Research 2022, 3, 761-771.
DuBose, J. T.; Kamat, P. V. Energy Versus Electron Transfer: Managing Excited-State Interactions in Perovskite Nanocrystal–Molecular Hybrids, Chemical Reviews 2022, 122, 12475–12494.
Chakkamalayath, J.; Hiott, N.; Kamat, P. V. How Stable Is the 2D/3D Interface of Metal Halide Perovskite under Light and Heat?, ACS Energy Letters 2023, 8, 169-171.
Szabó, G.; Kamat, P. V., How Cation Migration across a 2D/3D Interface Dictates Perovskite Solar Cell Efficiency ACS Energy Letters 2024 9 (1), 193-200
Chakkamalayath, J. et al., Photon Management Through Energy Transfer in Halide Perovskite Nanocrystal–Dye Hybrids: Singlet vs Triplet Tuning. Accounts of Chemical Research 2025, 58, 1461–1472.

FOLDS Seminar: ACS: An interactive framework for machine-assisted selection with model-free guarantees
2025-11-06 12:00 - 13:00 | Amy Gutmann Hall, Room 414, 3333 Chestnut Street, Philadelphia, 19104, United States
Zoom link: https://upenn.zoom.us/j/98220304722

In this talk, I will introduce adaptive conformal selection (ACS), an interactive framework for model-free selection with guaranteed error control. Building on conformal selection (Jin and Candès, 2023b), ACS generalizes the approach to support human-in-the-loop adaptive data analysis. Under the ACS framework, we can partially reuse the data to boost the selection power, make decisions on the fly while exploring the data, and incorporate new information or preferences as they arise. The key to ACS is a carefully designed principle that controls the information available for decision making, allowing the data analyst to explore the data adaptively while maintaining rigorous control of the false discovery rate (FDR). Based on the ACS framework, we provide concrete selection algorithms for various goals, including model update/selection, diversified selection, and incorporating newly available labeled data. The effectiveness of ACS is demonstrated through extensive numerical simulations and real-data applications in large language model (LLM) deployment and drug discovery.
The talk is based on https://arxiv.org/pdf/2507.15825.

BE Seminar - Rohit Bhargava, "Chemical imaging: engineering a bridge between morphology and molecular composition in biomedical sciences"
2025-11-06 15:30 - 16:30 | 216 Moore Building

CIS Seminar: "Exams with More Learning and Less Stress with a Computer-Based Testing Facility"
2025-11-06 15:30 - 16:30 | Berger Auditorium (Room 13), Skirkanich Hall, 210 South 33rd Street, Philadelphia, PA, 19104, United States
In this talk, I’ll share (1) research on the benefits of frequent testing and “second-chance testing” (optional exam re-takes) on increased student learning and decreased test anxiety, (2) research on patterns of cheating on unproctored online assessments, and (3) how we’ve reduced the instructor workload at Illinois to implement frequent testing through our Computer-Based Testing Facility (CBTF). The CBTF is a collection of proctored computer labs that, in conjunction with the PrairieLearn open-source question-asking platform, enable our faculty to run sophisticated exams with almost no recurring effort even in the largest classrooms. For example, our CS 1 course for majors (run by a single faculty member) ran weekly exams for 1,150 students. Key enabling ideas for the CBTF include: (1) sophisticated auto-grading questions, (2) question generators, (3) asynchronous exams, and (4) dedicated testing space and proctors. The CBTF has been running for over 10 years and proctored over 100,000 exams last semester.

CIS Seminar: "Exams with More Learning and Less Stress with a Computer-Based Testing Facility"
2025-11-06 15:30 - 16:30 | Berger Auditorium (Room 13), Skirkanich Hall, 210 South 33rd Street, Philadelphia, PA, 19104, United States
Exams are an important tool for summative assessment, whose utility has only grown with the advent of large language models (LLMs) like ChatGPT, because they can be implemented in a trustworthy manner. But exams are generally not well liked by either students or faculty. Students find them stressful. For faculty (and their course staff), they represent a large adminstrative burden to write, proctor, and grade. This large burden means they are done infrequently in many classes, but this infrequent testing encourages cramming and leads to high test anxiety.
In this talk, I’ll share (1) research on the benefits of frequent testing and “second-chance testing” (optional exam re-takes) on increased student learning and decreased test anxiety, (2) research on patterns of cheating on unproctored online assessments, and (3) how we’ve reduced the instructor workload at Illinois to implement frequent testing through our Computer-Based Testing Facility (CBTF). The CBTF is a collection of proctored computer labs that, in conjunction with the PrairieLearn open-source question-asking platform, enable our faculty to run sophisticated exams with almost no recurring effort even in the largest classrooms. For example, our CS 1 course for majors (run by a single faculty member) ran weekly exams for 1,150 students. Key enabling ideas for the CBTF include: (1) sophisticated auto-grading questions, (2) question generators, (3) asynchronous exams, and (4) dedicated testing space and proctors. The CBTF has been running for over 10 years and proctored over 100,000 exams last semester.

Fall 2025 GRASP on Robotics: Neville Hogan, Massachusetts Institute of Technology, “Sensory-motor control in humans and robots”
2025-11-07 10:30 - 11:45 | Wu and Chen Auditorium (Room 101), Levine Hall, 3330 Walnut Street, Philadelphia, PA, 19104, United States
This event will be in-person ONLY in Wu and Chen Auditorium.
ABSTRACT
Despite recent advances, humans are still more agile and dexterous than robots; yet human communication (nerves) and actuation (muscles) are slower and our musculo-skeletal system is more complex. This presentation will consider features of neuro-mechanics that may confer advantage. However, they also impose limitations.
Muscle is highly ‘back-drivable’, enabling our ease with (even preference for) ‘contact rich’ tasks. However, muscle is not just a force-generator. Our endo-skeleton requires muscle stiffness for stability; moreover, stiffness must increase at least in proportion to tension.
Consequently, human strength is not limited by force production but by stiffness production. Recent experiments confirm this.
Measuring stiffness (or its dynamic generalization, mechanical impedance) requires access to three variables, but only two are directly measurable: force and position. ‘Subtracting’ a model of limb mechanical impedance enabled estimating the neurally-defined reference trajectory (the third variable) underlying a simple ‘contact-rich’ task: turning a circular crank. It displayed a coincidence of curvature maxima and speed minima, despite the strictly-constant curvature of the constrained hand path. This feature, as well as an observed dependence on turning direction, was reproduced by a model composing the neurally-defined reference trajectory from superimposed oscillations.
Composing cyclic movements from ‘primitive’ oscillations simplifies control but implies a speed-curvature constraint that is widely reported; it significantly limits human performance. It also accounts for our remarkable inability to control force exerted on a moving robot.
The composability of motion is complemented by the composability of mechanical impedance. That enables a truly modular approach to robot programming. It simplifies transitions between free and constrained motion; manages redundancy without inverse kinematic computations; and enables operation into, at, and out of singular configurations—all features of human sensory-motor control that may benefit robots.

Homecoming 2025: Penn Engineering Faculty Lightning Talks
2025-11-07 13:00 - 14:15 | Amy Gutmann Hall, Auditorium, 3333 Chestnut Street, Philadelphia, PA, 19104, United States
Step back into the classroom with Penn Engineering!
Join us for a series of TED Talk-style Lightning Talks featuring some of our most popular professors as they share their latest groundbreaking research.
“Fragility and Resilience of the Soft Earth”
Doug Jerolmack
Edmund J. and Louise W. Kahn Endowed Term Professor of Earth and Environmental Science
Professor of Mechanical Engineering and Applied Mechanics
“Immune Engineering for Cardiovascular Health”
Noor Momin
Stephenson Foundation Term Assistant Professor of Innovation
Bioengineering
“Advancing Genome Editing for Precision Therapeutics and Molecular Innovation”
Sherry Gao
Associate Professor
Chemical and Biomolecular Engineering
“Building the World’s Smallest Robot”
Marc Miskin
Assistant Professor
Electrical and Systems Engineering
“Uncovering Regulators of Aging with Dynamic Biomaterials”
Chris Madl
Assistant Professor, Chemical and Biomolecular Engineering, Materials Science and Engineering

PICS Colloquium: Influence of particle size distribution on random close packing with Eric Weeks
2025-11-07 14:00 - 15:00 | PICS Conference Room 534 – A Wing , 5th Floor, 3401 Walnut Street, Philadelphia, PA, 19104, United States
The volume fraction phi for random close packed (RCP) spheres is approximately 0.64. It is well known that higher RCP volume fractions are achieved by using collections of particles with a variety of sizes. The variety of sizes is often quantified by the polydispersity of the particle size distribution: the standard deviation of the radius divided by the mean radius. I’ll show that for 2D and 3D packings, the skewness also plays an important role (related to the third moment of the size distribution). I will also discuss some of our work on random close packing in confined spaces.

Building a Sustainable Future: Empowering the Next Generation.
2025-11-07 15:00 - 16:00 | Amy Gutmann Hall, Auditorium, 3333 Chestnut Street, Philadelphia, PA, 19104, United States

ESE Guest Seminar - "Challenges and Opportunities in Radio Frequency Power Conversion for Semiconductor Plasma Applications"
2025-11-10 10:30 - 11:30 | CTA
Radio frequency (RF) plasma technology is essential in modern semiconductor fabrication, enabling precise processes such as etching and deposition. As the industry advances toward increasingly complex three-dimensional structures and smaller nanoscale features, the demands on plasma-based processing continue to grow. Meeting these demands requires a new generation of RF power conversion and control systems that are robust, accurate, agile, and efficient. Key challenges include delivering precise power across wide dynamic ranges and frequencies, operating under rapidly varying load impedance conditions, maintaining high power efficiency, and incorporating advanced sensing and system analytics. These requirements reveal fundamental limitations in conventional RF power delivery systems.
This presentation offers a brief overview of key plasma processes and typical inductively and capacitively coupled plasma systems. It highlights the core RF power challenges encountered in these platforms and presents recent innovations aimed at addressing them. These include RF inverter designs that maintain high efficiency under varying load conditions, scalable power combining techniques for rapid power control, and phase switched impedance modulation (PSIM) for high-speed impedance matching and transformation. Together, these advances support the development of next generation RF power architectures that enable more capable and efficient semiconductor plasma processing systems.
