I was hugely impressed with Cathy “Mathbabe” O’Neil’s talk at Personal Democracy Forum 2015, “Weapons of Math Destruction,” in which she laid out the way that the “opaque, unregulated, and uncontestable” conclusions of Big Data threaten fairness and democracy.
I’d read her excellent site for years, but the presentation crystalized much of what I’d read of hers in a crisp, compelling argument.
That talk will be expanded into a nonfiction book that Crown will publish next September, and which is available for pre-order from today. I just ordered my copy.
We live in the age of the algorithm. Increasingly, the decisions that affect our lives—where we go to school, whether we get a car loan, how much we pay for health insurance—are being made not by humans, but by mathematical models. In theory, this should lead to greater fairness: Everyone is judged according to the same rules, and bias is eliminated. But as Cathy O’Neil reveals in this shocking book, the opposite is true. The models being used today are opaque, unregulated, and uncontestable, even when they’re wrong. Most troubling, they reinforce discrimination: If a poor student can’t get a loan because a lending model deems him too risky (by virtue of his race or neighborhood), he’s then cut off from the kind of education that could pull him out of poverty, and a vicious spiral ensues. Models are propping up the lucky and punishing the downtrodden, creating a “toxic cocktail for democracy.” Welcome to the dark side of Big Data.
Tracing the arc of a person’s life, from college to retirement, O’Neil exposes the black box models that shape our future, both as individuals and as a society. Models that score teachers and students, sort resumes, grant (or deny) loans, evaluate workers, target voters, set parole, and monitor our health—all have pernicious feedback loops. They don’t simply describe reality, as proponents claim, they change reality, by expanding or limiting the opportunities people have. O’Neil calls on modelers to take more responsibility for how their algorithms are being used. But in the end, it’s up to us to become more savvy about the models that govern our lives. This important book empowers us to ask the tough questions, uncover the truth, and demand change.
Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy [Cathy O’Neil/Crown]
Finishing up Weapons of Math Destruction
[Cathy O’Neil/Mathbabe]