The Electronic Frontier Foundation's Jamie Williams and Lena Gunn have drawn up an annotated five-point list of questions to ask yourself before using a machine-learning algorithm to make predictions and guide outcomes.
The list draws heavily on two essential recent books on the subject: Cathy O'Neil's Weapons of Math Destruction and Virginia Eubanks's Automating Inequality, both of which are essential reads.
The list's five questions are:
1. Will this algorithm influence—or serve as the basis of—decisions with the potential to negatively impact people’s lives?
2. Can the available data actually lead to a good outcome?
3. Is the algorithm fair?
4. How will the results (really) be used by humans?
5. Will people affected by these decisions have any influence over the system?
Algorithmic-decision making is often touted for its superiority over human instinct. The tendency to view machines as objective and inherently trustworthy—even though they are not— is referred to as “automation bias.” There are of course many cognitive biases at play whenever we try to make a decision; automation bias adds an additional layer of complexity. Knowing that we as humans harbor this bias (and many others), when the result of an algorithm is intended to serve as only one factor underlying a decision, an organization must take care to create systems and practices that control for automation bias. This includes engineering the algorithm to provide a narrative report rather than a numerical score, and making sure that human decision makers receive basic training both in statistics and on the potential limits and shortcomings of the specific algorithmic systems they will be interacting with.
And in some circumstances, the mere possibility that a decision maker will be biased toward the algorithm’s answer is enough to counsel against its use. This includes, for example, in the context of predicting recidivism rates for the purpose of determining prison sentences. In Wisconsin, a court upheld the use of the COMPAS algorithm to predict a defendant’s recidivism rate on the ground that, at the end of the day, the judge was the one making the decision. But knowing what we do about the human instinct to trust machines, it is naïve to think that the judge’s ‘inherent distraction’ was not unduly influenced by the algorithm. One study on the impact of algorithmic risk assessments on judges in Kentucky found that algorithms only impacted judges’ decision making for a short time, after which they return to previous habits, but the impact may be different across various communities of judges, and adversely impacting even one person is a big deal given what’s at stake—lost liberty. Given the significance of sentencing decisions, and the serious issues with trying to predict recidivism in the first place (the system “essentially demonizes black offenders while simultaneously giving white criminals the benefit of the doubt”), use of algorithms in this context is inappropriate and unethical.
Math Can’t Solve Everything: Questions We Need To Be Asking Before Deciding an Algorithm is the Answer
[Jamie Williams and Lena Gunn/EFF]
