ABSTRACT: 

Algorithm audits are powerful tools for studying black-box systems without direct knowledge of those systems’ inner workings. While they have been effectively deployed to identify harms and biases in algorithmic content, algorithm audits’ narrow focus on technical components stop short of considering users themselves as integral and dynamic parts of the system, to be audited alongside its algorithmic components.

After an overview of the state of the art in algorithm auditing, this talk will introduce sociotechnical auditing: evaluating algorithmic systems at the sociotechnical level, focusing on the interplay between algorithms and users as each impacts the other over time. I will also present Intervenr, a platform we developed to conduct browser-based, longitudinal sociotechnical audits, and a case study in which we deployed Intervenr to investigate the central claim of online targeted advertising systems: that targeted ads perform better with users. Finally, I will conclude by discussing some of my group’s current work, expanding the auditing method to novices and youth, and developing post-auditing tools for collective action.