Forget those cute pastel illustrations from the fifties with their flying cars, robot servants and dreams of unlimited leisure. Our future has finally arrived and for most of us, especially for the less rich and less privileged who won’t qualify for individualized attention, the computer says “No”. ‘Weapons of Math Destruction’ is a timely book about the increasing influence of algorithms to control the news we see, the jobs we can get and the politicians we vote for; algorithms working tirelessly on someone’s behalf (not yours), unseen and unaccountable. The book explains statistical and methodological problems with these algorithms and illustrates how these same problems manifest themselves when they are applied to real world situations. An example gives a flavor of the issues that recur throughout the book: An algorithm which compared changes in student performance year-on-year was used to decide which teachers were a poor performers who would be let go. In one terminated teacher’s case subsequent evidence suggested that student’s scores had been tampered with the previous year and as a result she had inherited a class whose performance had been overstated, so was bound to deteriorate with the algorithm marking her out as a poor teacher. The teacher was not told how the algorithm applied nor allowed to appeal the decision; the algorithm was a black box that produced a result the school system wanted – terminations. And, notwithstanding the lack of transparency in this decision, a sample of scores from one class alone - perhaps twenty to thirty students - was not sufficient to produce a statistically valid result in any event. One more reason why being an American public school teacher must be one of the worst teaching jobs in the developed world. Please read the book for more egregious examples of “algorithm abuse”. Below is my summary are some of the key problems the book outlines. It is helpful to think about these problems in two categories, problems which come from poor application of the technical aspects of the algorithms (bad programming, misapplication of statistical or machine learning methods) and problems which come from how the algorithms are developed and used, how they incorporate hidden biases or value judgments, poorly thought out objectives and other ‘human factors’ beyond just poor maths: Some of the common issues were: Lack of transparency in how the models operate and how they make a decision, often leading to no means of appeal against a clearly unjust outcome. Related problems include over reliance on models in the face of contradictory data or, where people do understand the models but are benefitting from them, a lack of integrity in applying them. Models used to price mortgage backed securities during the financial crisis are a leading example. The use of “proxy data” for model input or output because the real data desired is unavailable, too expensive to obtain or cannot be objectively measured. Does sending out more e-mails with “creative phrases” mean that you are really a more creative and innovative person? FICO scores are a relatively good model for use in predicting credit risk, but not as a proxy when used for a whole host of other unrelated things such as predicting future job performance when used in the hiring process. Feedback loops, whereby a model increasingly encourages non optimal outcomes by rewarding certain behaviors at the cost of intended benefits. An example given is US college rankings which increasingly reward “user experience” and “research citations” rather than the actual educational outcomes for students. Algorithms that, mainly for efficiency purposes, use data from arbitrarily selected groups when individual data is available. Why should an algorithm price insurance for a driver based on the experience of other drivers that live near him or are in a similar economic position instead of on his own individual driving record? Optimization that is good for an algorithm’s owners but not for society as a whole. Monetary return is the most common priority for an algorithm used in the private sector, but is this what society wants when it results in micro-targeting the poor with the marketing of for-profit colleges that provide a below average education at great cost? Hidden biases and unfairness when assumptions are built into an algorithm that are reflective of social factors rather than individual experience. Algorithms used by the police to forecast crime are first used to predict easy to identify nuisance crimes which, unsurprisingly, occur mainly in deprived neighborhoods. The police have yet to develop an algorithm that forecasts where white collar crime takes place, although if they did Wall Street would surely light up red. Incorrect use of statistics; lack of feedback of model results into predicting outcomes. Baseball is a good field for prediction because outcomes – homes runs, strikes, batting averages - can be objectively measured and predictions can be fed back into the model to improve it. Using FICO scores in recruitment is not what the developers of FICO intended; there is no study suggesting statistical correlation between FICO and subsquent good job performance; good or bad job performance is also never used to assess HR models or improve their reliability. The last chapter of the book looks at the micro targetting of voters with political messages through Facebook and similar social media sites. This was written in September 2016, a respectable amount of time before the world had a glimpse into the dizzying vortex of fake news, Russian hacking and tweeting Presidents. This is an excellent chapter and may well be seen as prophetic once we look back over our current period of political chaos, if it ever ends. I resolved to write notes on the books I read as I get so much more out of them if I do. This resolution will last about one and a half books I’m sure, but if anyone is interested here they are: Introduction: Mathematical models are being increasingly used to make decisions that have real world impacts. However the algorithms they use are opaque except to a limited number of mathematicians or computer scientists and may, sometimes unknowingly, encode human prejudices, misunderstandings and biases. Assumptions may be camouflaged by the maths and go untested and cannot be questioned by the persons to whom they are applied. These algorithms are often used in inappropriate contexts where there is insufficient objective data to properly apply the underlying statistical theory; algorithms may work for baseball with a many tens if not hundreds of thousands of objective, constantly updated data points but not to evaluate a teacher with a class of thirty. The algorithms produce “feedback loops” under whereby they produce output that eliminates particular classes of result based on false assumptions, but as a result reinforce the weighting given the assumption in the model. Injustice is reinforced as these algorithms are applied to ever larger numbers of people, although the rich and privileged may still be assessed on an individual basis rather than as one of the masses assessed by machine. People unfairly denied opportunities as a result of this software are “collateral damage”. The algorithms optimize a payoff designed by the developers of the model - in the case of many private sector models, monetary profit - but who is to say that this is the optimum payoff for society as a whole? Chapter 1, Bomb Parts - What is a Model? The author explains how her managing of her family cooking is a form of “data model”: inputs are family preferences, appetite that day; available food, special cases such as cooking on a birthday, the output is “family satisfaction” and the model determines menu for that day The model could be pre-programmed with a set of rules to determine its menu or could be trained through observing many examples. In either case mistakes may be made, perhaps through forgetting a rule or not including a rare case in the training data. The key point is that the model can incorporate personal biases that are not visible, in the case of the menu model towards healthy food and away from ice-cream. Models don’t have to be complicated to be effective; a smoke alarm is a model intended to identify fire that operates on only a single input, the concentration of smoke particles. Three questions are posited to evaluate models: Firstly, is the model understandable - or even visible to - by the people to whom it is applied? Secondly, does the model work in the subjects’ interests? Is it fair or can it cause unjust damage? Thirdly, does the model scale? May it be used in ever wider circumstances, at the same time reinforcing its hidden biases as it is applied in ever wider circumstances? An example is the model used to determine sentence length in US courts based on a model about the risk of recidivism. The model takes into account factors, such as previous criminal record or age of first contact with the police, which would not be admissible as evidence in court and which may unfairly discriminate against certain sections of the population. Chapter 2 talks about the role of algorithms in the financial crisis, noting two key issues that lead to the collapse in the mortgaged backed securities market. Firstly the assumption that models had been subject to proper mathematical vetting; in reality few people understood the mathematical and statistical issues and many that did lacked the integrity to speak up, especially as initial success had created its own feedback loop encouraging growth in the market. Secondly modern computing power had allowed a massive secondary infrastructure to grow around the market - credit default swaps, CDOs etc - that instead of diversifying the risk masked, magnified and concentrated it. A very interesting point was made based on the author's experience in a risk assessment firm, that many traders are remunerated based on their Sharpe ratios, the ratio of revenues to risks taken, and accordingly are motivated to “...actively seek to underestimate…” risk in order to effectively manipulate the Sharpe ratio and hence their bonuses. This contrasts with the approach of hedge funds, which genuinely care about risks (given their own money is at risk) and traders at large financial institutions which don’t have their own financial capital at stake. Chapter 3 looks at the impact of the algorithm developed by US News to rank colleges in the US. 75% of the ranking was based on proxy items intended to measure college success and 25% on subjective evaluation. The proxies selected, for example admission ratios, SAT scores, invited gaming that distorted applications; colleges would invest in sports in order to encourage applications that could be rejected; in an extreme case a new Saudi college required part time professors with large numbers of citations to change their site reference to the college in order to move up the rankings. The system does not measure the key success of education - what the students learnt at each school. The ranking generates its own feedback loop; colleges that rank high attract more applicants, generating more rejections thus moving them further up the rankings. Wealthy applicants pay consultants to game the system. Crucially the original algorithm did not take into account college tuition costs. This guaranteed the early rankings being in line with existing “common sense” with Yale, Harvard and other wealthy colleges ranking high but thereby excluded an issue critical to students, value for money from an education, while encouraging colleges to spend excessively in order to improve student experience and hence ranking without regard to cost, ultimately leading to more student debt. US govt has now made available data on schools allowing students to check directly. Chapter 4 looks at targeted online advertising, taking for profit colleges that target poor and vulnerable people as a particularly nefarious case. The internet gives instant feedback for targeted marketing campaigns through Facebook or Google in particular where successful or failing ads can be evaluated in real time. Bayesian analysis is used to evaluate success. 20-30% of a for profit college’s budget may go on lead generation with more spent on recruitment than on education itself. Specialist lead generation firms exist targeting particular communities, posting fake job ads or promising health coverage. Chapter 5 looks at “Predpol” an algorithm to predict crime which has a key input the geographic location of crime but which excludes data on race. The algorithm is successful in forecasting where crimes can occur, but mainly because it includes low level crimes - public drunkenness, jaywalking - which are better correlated with geographic location. However these crimes are also often associated with poverty and indirectly with race; more serious crimes are more difficult to detect while the system ignores some crimes altogether, such as white collar fraud. On the surface the algorithm is objective but under the hood it reflects value judgments around where the police direct their attention. Issues of probable cause are also raised by the use of algorithms that predict whether a person will commit crime based on proxy data such as location of residence, whether or not employed or similar. These algorithms raise questions around to what extent the public is prepared to balance efficiency in police work against fairness, but without any public debate. Chapter 6 looks at the use of algorithms in employee hiring decisions.. Personality tests are often used for screening job applicants, but these may be “run arounds” laws preventing discrimination against people with disabilities and, a key consideration for the correct use of algorithms, are rarely updated through monitoring their actual success in predicting good job performance. This may not be a problem for a system evaluating baseball stars, whose performance can be objectively statistically measured and where the individuals may be paid millions of dollars, rather the burden falls on lower paid staff in the collateral damage suffered by individuals that are screened out by the algorithms but are capable of doing the job. These algorithms include negative feedback - those discriminated against fail to get good jobs, thus justifying the discrimination - and may be based on data that reflects historically discriminated hiring practices; an example is given of a system used in a British hospital that after many years of use was held to discriminate against women and immigrants, repeating discrimination reflected in the original data on which the system was based. Chapter 7 examines the impact of algorithms in the workplace. Job scheduling software uses data to schedule peaks and troughs of staffing, dehumanizing employees in the process who are unsure of their work schedule until only a day or two before being called on. Another algorithm - ‘Cataphora’ - attempted to identify the most creative and innovative employees through tracking the flow of e-mail including certain key phrases through the e-mail system. There is little evidence that this approach worked, but employees who were not among those identified as the most creative may be first in line for termination. This algorithm suffers from the two classic problems identified early in the book, the difficulty of finding measurable proxy data for the items (soft skills such as creativity) that you want to measure and the lack of any feedback on success and failure of those measured to help the algorithm learn. The chapter explained an egregious error in the evaluation of teacher performance through the use of SAT scores. The ‘Nation at Risk’ report issued by the Reagan administration was intended to address the decline in teaching standards as measured by a gradual decline in SAT scores of graduating students over the years. In fact the data illustrated an example of ‘Simpson’s Paradox’ in which the whole body of data showed one trend but when examined on a segmented trend the opposite trend was apparent. In this particular case the scope of people taking SAT tests had expanded over time including enough people at the lower end of the scoring range to lower the overall average (i.e. elite students had been sitting the test for many more years, so there was little room for scope to increase at the higher end of the scoring range). When the data was segmented in narrow SAT ranges SAT scores had increased in all segments over time. The whole premise of the report was false. Chapter 8 looked at FICO scores. FICO scores themselves have some good features as data; they are based on an individual’s history (in contrast to being based on aggregate data inferred from people who resemble the individual), default on a loan is relatively objectively measured and feedback is used to improve scoring. Problems arise when FICO scores are used as proxies for other data that is not so easily or objectively identified, for example future job success when used as screening in the hiring process, or when combined with other data in developing proprietary “e-numbers” used by firms for purposes such as marketing. These metrics are not transparent to consumers and risk being a backdoor for discrimination because they may use factors that are indirectly linked to poverty or race such as residence. Chapter 9 considered how the whole insurance business model may be undermined by data algorithms. Insurance relies on the pooling of a wide range of risks, good and bad, to allow the pricing of the risk in aggregate. Algorithms allow such risks to be segmented in a non-transparent way which, in addition to undermining the insurance model itself by pricing out high risks, permits price gouging of other segments. The use of non risk related indicators, for example credit scores or location of residence, in pricing motor insurance also allows price gouging. Why should a drop in a driver’s credit rating impact his premium when the risk that he has an accident is unchanged? Drivers are being assessed on the consumer patterns of their friends and neighbors rather than on their accident record. Chapter 10 looks at the application of algorithms to civic life, in particular the use of micro targeting of political messages through Facebook. Facebook has experimented with manipulating emotional responses of users through manipulating their newsfeeds. Facebook’s approach to disseminating news is contrasted to a conventional newspaper in which the Editor makes a decision on what to put on the front page but that decision is seen by everyone and open to public debate whereas the criteria for selection of articles for a newsfeed by Facebook are opaque and unique to the viewer. Facebook micro targeting may be one of the reasons many Republican voters still believe Obama ‘birther’ and other conspiracies. Micro targeting of voters risks disenfranchising everyone to the benefit of those paying for it. Voters in swing states are subject to more focused and intensive campaigns which are relevant only to themselves to the detriment of the democratic process for all.
