As Artificial Intelligence (AI) finds increasing applications in industry and society. Responsible deployment demands that we measure and correct algorithmic biases vis-a-vis protected attributes such as sex, age and ethnicity. State of the art methods for measuring algorithmic bias rely on test sets that are collected in the wild and are then annotated for the protected attributes. Such methods are therefore observational and yield correlational information. I will argue that in order to obtain useful information to discover and correct biases we need causal information which is only available if we use an experimental method. I will show that modern generative models offer a promising starting point to develop experimental testing methods. I will review our recent work in face synthesis and demonstrate its application to the study of algorithmic bias in gender classification, face recognition, and social judgment of faces.