Weapons of Math Destruction

Before government-sanctioned and other electronic credit scoring systems, individuals seeking a loan would have to physically go to a bank and request the loan. Generally, they would provide evidence of their fiscal responsibility and general financial portfolio. However, personal connections and biases played a large role in the ways these loans were distributed. For example, a banker might be more likely to give a loan to a friend, but women and minorities often had the doors closed to them before the conversation began, forcing them to arm themselves with financial evidence that may or may not sway the banker.

Since this system was flawed, the idea of FICO and other calculated scores appears fairer. Rather than focusing on connections, these scores focus on an individual’s history with money. On the surface, this seems fair and by WMD standards, O’Neil considers this a more innocuous one to a point. After all, credit scores are relatively transparent and have a clear feedback loop. If a person pays their bills on time, their credit score reflects this and vice versa.

However, O’Neil argues that the appearance of fairness doesn’t make it so. Credit reports are often used in other domains, like job-hunting. People with poor credit may be denied a job, further preventing them from paying bills on time. O’Neil also asserts that there’s a false link between creditworthiness and morality, a practice she calls “both toxic and ubiquitous” (147). There are many legitimate reasons people fail to pay bills on time, from losing a job to medical debt. Another reason people fall into these poverty traps is the use of electronic credit scoring, or e-scores, which companies and banks use to guess at an individual’s credit score based on a variety of factors, including zip codes. These e-scores prey upon individuals they consider high risk and offer them loans with much higher interest rates, which threaten to lock the individuals receiving them into cycles of poverty, despite their actual credit score or circumstances. 

Insurance policies, whether for health or cars, often use proxies of a consumer’s past behavior to determine rates. For example, the auto industry typically relies on credit scores to determine insurance prices. While age and driving record factors into these decisions, the composite relies mostly on broad assumptions based on a person’s financial history and demographics. Some companies have attempted to make their metrics for payment more directly contingent on a driver’s ability, rather than the proxies of age, academic record, and creditworthiness. They promise discounts for careful driving, and while this is a desirable quality for a pricing algorithm, it does have the potential to expand into unwelcome territory.

O’Neil notes that these careful driving benefits are opt-in for now, but other companies may expand to track where individuals are driving and when. Companies, like Sense, sort people into different groups based on their activities and calculate their risk that way. Naturally, people who frequent certain more “dangerous” zip codes or travel at odd hours will be considered higher risk and therefore incur more costs, harming those who live in low-income areas or who do shift work.

This type of nefarious modeling is also seen in companies’ health care incentives. Some companies charge more for health insurance based on an employee’s BMI, and O’Neil posits that the metrics collected by company health programs, Fitbits, and the like could be used to avoid hiring a candidate with health issues altogether. The danger is that these models rely on proxies to make determinations that affect people’s lives, and many individuals aren’t even aware it’s happening.

It’s no secret that social media relies on algorithms to shape the content consumers see and even sometimes to conduct social experiments on unwitting participants. O’Neil analyzes Facebook’s influence over individuals’ newsfeeds and how this influenced the election. Social media companies act as neutral platforms to disseminate thought, and this false assumption is cemented in the fact that the opinions and news consumers receive comes from people they know and often trust, like friends.

Facebook has also explored social media’s impact on an individual’s mood by altering the algorithm to show either more positive or negative content. There are likely many other experiments taking place on social media that companies do not publicize, and while O’Neil thinks these don’t rise to the level of WMDs yet since they don’t have a clear agenda to cause harm, they certainly have the power to do so. Other online algorithms have revealed a subset of consumers who show no brand loyalty and that there may be a cross-section of these individuals who are also likely to be swing voters. The value of marketing to these voters depends on their state’s role in the election, and the nature of the algorithms used by social media or politicians to target some demographics depend primarily on their objective, which O’Neil argues could even be tweaked and used for mass good.

O’Neil closes by reasserting the mechanics of WMDs and posing questions about their role in society. She sees them as tools to “[c]odify the past,” rather tools for creating the future (204). The best way to prevent WMDs from causing damage in the future is not to try to eliminate them altogether. They are far too profitable for the companies that use them, so this solution will never be viable. The answer is for brilliant minds capable of creating WMDs to use their abilities to learn the process and make it better.

She also states that consumers should demand more transparency in the algorithms that affect our lives so that individuals can exert greater personal control over them and understand how they affect their lives. Society cannot act as those WMDs are these systems beyond control, but rather, society must demand accountability, accessibility, and improvements. Transparency of algorithms and coverage of the discriminatory practices they perpetuate for profit is essential to making progress and having more equitable systems.