The Best-Selling Book of Bad Science
December 4, 2010 § 3 Comments
Freakonomics (Levitt, Steven, and Dubner, Stephen)
“The typical … expert … is prone to sound exceedingly sure of himself… An expert must be bold if he hopes to alchemize his homespun theory into conventional wisdom. His best chance of doing so is to engage the public’s emotions, for emotion is the enemy of rational argument.”
– Steven Levitt and Stephen Dubner, Freakonomics
“Many people – including a fair number of his peers – might not recognize Levitt’s work as economics at all. But he has merely distilled the so-called dismal science to its most primal aim: explaining how people get what they want.”
– Steven Levitt and Stephen Dubner, Freakonomics
The problems I have with this book begin with its name. Just because Steven Levitt is an unconventional economist doesn’t make everything he writes economics, freak or otherwise. “Explaining how people get what they want” is rather broad, and isn’t quite the “primal aim” of economics according to any definition I have seen; nor, in any case, is that the subject of Levitt’s book. The subject, in so far as I could discern one, is the examination of causality and correlation between social actions and outcomes, using basic statistical tools. I guess ‘Freak Sociology’ or ‘Freakistics’ do not have as catchy a ring to them as ‘Freakonomics’: hence, I am willing to bet, the name.
The same problem persists through the book. I have very few bones to pick with the authors’ politics, opinions or conclusions; I have several skeletons to pick with the somewhat dishonest representation of their studies. I accuse them, not of being bad people, but of purveying bad science in parts, and mainly, of bad faith. They had a book to sell, and they know that nothing sells like sensationalism (refer lead quote about experts), and so, they have occasionally resorted to deliberate misrepresentation of ambiguous statistical results in such a way that they sound definitive and authoritarian to careless readers.
This, in an adman, a PR lobbyist or a non-serious journalist, is perfectly understandable, if not exactly acceptable – so you, Dubner, are off the hook, with a rap on your knuckles; however, it is simply inexcusable in an academic. After a BA in Harvard, a Ph.D from MIT, and professorship at the University of Chicago, what forgiveness, Levitt?
What forgiveness, for the fact that someone like me, with not much more than undergraduate training in statistical analysis, could spot four glaring examples of dishonesty while breezing through the book in a couple of days?
Example One: The authors want to justify their view that the truism that electoral candidates spend too much money is false. This is what they say:
“In a typical election period that includes campaigns for the presidency, the Senate and the House of Representatives, about $1 billion is spent per year – which sounds like a lot of money, unless you care to measure it against something seemingly less important than democratic elections.
It is the same amount for instance, that Americans spend every year on chewing gum”
Observe the cunningness here. The “amount that Americans spend on chewing gum” is expected to remind the casual reader of the trivial amount that he, as an individual, spends on gum. Indeed, a billion dollars a year doesn’t add up to much per individual per year. However, this total sum, spent by 300 million Americans, is being compared with the sum spent on election campaigns every year – by roughly one thousand Americans who stand for public office. The former adds up to three dollars per person-year; the latter is a million. Are they comparable? Levitt clearly wants you to agree with him without thinking for yourself.
Example Two: The authors want to justify their controversial, celebrated – and in my opinion, reasonably plausible – view that Roe v Wade in 1973 helped in the drastic reduction of crime rates in the US in the 1990’s (by 50% over 5 years). In their introductory chapter, they mention a few competing reasons for this reduction (booming economy, gun control, innovative policing) and dismiss these theories as false. Carefully, they make no mention, in that chapter, of other competing theories that are actually valid. They then present, with drum-rolls and fanfare, legalized abortion, as the real reason for the drop in crime rates – omitting to tell us that as a factor, the consequence of abortion laws accounts for just about 40% of the drop, even according to their own estimates. It was only much later, by piecing together statistical tidbits strewn around a later chapter, that I was able to ascertain that nearly 60% of the decline in crime was attributable to three other factors – a surge in prison populations (33%), the cocaine market crash (15%) and an increase in the police force (10%). So, while Roe v Wade is still possibly the largest single factor, it doesn’t explain even half the phenomenon on its own. The authors gloss over this inconvenience. “Legalization of abortion causes 20% drop in crime” is simply not jazzy enough to sell a newspaper, let alone a book. It is also, unfortunately, the truth.
Example Three: This one is a shocker, because it is either badly worded or poorly thought through. While examining the merits of the theory linking increased policing to the decline in crime, they say:
It may help to look backward and see why crime had risen so much in the first place. From 1960 to 1985, the number of police officers fell more than 50 percent relative to the number of crimes. …the 50% decline in police translated into a roughly equal decline in the probability that a given criminal would be caught. … this decrease in policing created a strong positive incentive for criminals.
Why did crime rise? Because, Levitt says, the police officers-to-crimes ratio declined. Now, ratios reduce in value when either the numerator decreases or the denominator increases – in this case, the ratio reduces when there is a reduction in police officers, or an increase in crimes. Here, as we know, the latter took place. So, what Levitt is saying is, crime increased because the ratio of officers to crimes reduced, but the ratio reduced because … crime increased. This is worse than confusing causality with correlation; it is confusing causality with consequence.
If Levitt meant, instead, that the process was auto-catalytic, and that a marginal increase in crime over police would result in criminals not getting caught, which would incentivize further criminal behavior, and so rapidly escalate into a spiralling crime situation, now that is an interesting idea and makes for an intriguing non-linear dynamical model, but he should then spell it out, because otherwise, saying only what he does, it just sounds as if he doesn’t believe his readers are smart enough to understand anything beyond over-simplified generality.
Oh, and irrespective of anything else, surely, a 50% decline in the officers-to-crimes ratio is not the same as a “50% decrease in police“. It is the kind of loose statistical statement that high school students get penalized for in term tests. Is this the kind of math they teach in Chicago these days? Good grief.
“There is one drowning of a child for every 11,000 residential pools in the United States. (in a country with 6 million pools, this means that roughly 550 children under the age of ten drown each year). Meanwhile, there is 1 child killed by a gun for every 1 million-plus guns (In a country with an estimated 200 million guns, this means that roughly 175 children under ten die each year from guns). The likelihood of death by pool (1 in 11,000) versus death by gun (1 in 1 million-plus) isn’t even close.”
Observe carefully: each of these five statements seems to say a lot; together, the argument appears to be a slam dunk. Or so you would think at first sight. But then you will realize that there is something terribly puzzling about the first two statements. The authors have deliberately sequenced them as if the first were a prominent, publicly known statistic, and the second were derived from it (note the use of the words “this means“). But the truth is exactly the opposite. Something similar happens in statements three and four. They were placed in that order only so that we accept the numbers “11,000” and “1 million” as numbers that are important, numbers that are unquestionable, numbers that need to be compared with each other. Then, with a wave of his hypnotic word wand, Levitt tells us that “the likelihood of death by pool” for the average American kid is 1 in 11,000.
Er, no, it is not.
If only a thousand people in the US owned pet boa constrictors, and if, further, exactly one child under ten were to be accidentally crushed to death by his pet, would you say the ‘likeliness of death by boa constrictor’ for the average American kid is 1 in 1000, and therefore, that American parents should be extremely paranoid about this eventuality? Surely the number of American kids who come in frequent contact with people carrying one of those 200 million guns needs to be compared with the number who come in frequent contact with the 6 million pools, or the number who come in close contact with pet boa constrictors, in order to figure out which is the likelier way to die? The absence of lifeguards hovering around people carrying guns may also be a factor to take into account.
And what, in any case, does a national average prove, and why should it affect a parent’s behavior any more than an understanding of her child’s specific circumstances? If more American kids die each year of frostbite than of sunburn, should a Miamian mother dress her child warmly in July? Averages make for lazy generalizations, which the public likes and is willing to pay good money for, but is that what reputed professors are supposed to dish out? I suppose proper analysis is far too boring to make for a popular book, but it is eye-opening to see an MIT alum strip off his intellectual integrity and reveal his naked greed for an extra dollar or two.
Mind you, for all I know, Levitt’s conclusion may be accurate, and parents should worry more about swimming pools than guns (or boa constrictors). But that was never the point. And hey, Levitt definitely got one thing right:
“An exclamation point in a real-estate ad is bad news for sure, a bid to paper over real shortcomings with false enthusiasm.”
An over-exuberant use of adjectives like “charming” and “fantastic“, Levitt says, in one of his better pieces, is likely to mean that what is being sold has no real qualities of its own and needs hyping up. I totally agree. In fact, one of the most irritating things about this book is the fact that it is littered with the most unnecessary, effusive, eulogical descriptions of Levitt himself:
“the most brilliant young economist in America – the one so deemed, at least, by a jury of his elders”
“…heralded young economist at the University of Chicago…”
“He sifts through a pile of data to find a story that no one else has found…”
“‘Levitt is considered a demigod, one of the most creative people in economics and maybe in all social science,’ says Colin F Camerer, an economist at the California Inst. Of Technology. ‘He represents something that everyone thinks they will be when they go to grad school in econ…'”
“He is a noetic butterfly that no one has pinned down … but who is claimed by all. He has come to be acknowledged as a master of the simple, clever solution.”
“He is genial, low-key and unflappable, confident but not cocky.”
Uh oh. Sounds like this property has problems. The school district is probably terrible, and the owner is not to be trusted. Take your money and run.