Catch Me If You Can: The Dynamic Nature of Bias in Machine Learning Applications

Abstract

Bias in machine learning (ML) applications represents systematic differences between expected and actual values of the predicted outputs, such that certain individuals or groups are systematically and disproportionately (dis)advantaged. This paper investigates the dynamic nature of bias in ML applications. We conducted an empirical study of bias in the context of an ML application that assesses applicants' competencies for a given job based on their job interviews. We analysed primary data (27 interviews with 21 members from six departments) and secondary data (e.g., company reports, webinars, and software demonstrations) from the organisation that develops and deploys the application. Drawing on the concept of reflexivity in digitised processes and associated notions of drift and control, we theorise the emergence and evolvement of bias as constituting dynamic relationships among sources of bias and associated effects of drift, and actions of bias mitigation and associated effects of control. We further demonstrate how actions aimed at mitigating bias, while weakening some sources of bias, also create additional sources through unintended side effects. We explain how sources of bias and bias mitigation actions continually (re)shape each other through dynamic and reflexive effects. For HR practitioners and policymakers, we provide actionable insights into bias detection and mitigation in ML‐based job competency assessment.