Abstract:

Recently, the transfer learning paradigm has seen a surge of interest due to its impressive capabilities in vision and language. Models are pretrained on ever-growing datasets with enormous parameter counts, trending towards being monolithic and opaque. How can we understand the underlying process? This talk will provide, to some degree, insight on how data affects transfer learning using the framework of influence functions. These findings include pinpointing subpopulations and biases in the pretraining data that help (or hurt) transfer performance.