Steven Levitt and Thomas Miles’ “The Role of Skill Versus Luck in Poker: Evidence from the World Series of Poker”

You might have heard something a couple of weeks ago about the appearance of another one of those academic studies into the old “skill-vs.-luck” debate poker. In this one, researchers Steven D. Levitt and Thomas J. Miles analyzed results from the 2010 World Series of Poker in order to shed some light on the issue. Their findings were published last month by the National Bureau of Economic Research in a paper titled “The Role of Skill Versus Luck in Poker: Evidence from the World Series of Poker.”

I was recently reminded of this study, and then reminded myself that since I’m teaching that “Poker in American Film and Culture” class I have an academic affiliation giving me access to the full essay. Incidentally, the skill-vs.-luck question was a constant topic of discussion in my class this spring, coming up either directly or indirectly in practically everything we read and watched.

I had a chance to read the study this week, and thought I’d pass along a summary of its methodology and findings along with some reaction.

Summary

Levitt you might know as the co-author of the best-selling Freakonomics (2005), one of those books that like Malcolm Gladwell’s The Tipping Point (2000) found a wide audience with its accessible explanations of various cultural phenomena, in this case using economics to guide its causal arguments. He’s currently a professor of economics at the University of Chicago, where Miles also teaches in the university’s law school.

Legislative battles over poker (especially online poker) appear to have motivated the pair’s study. Since the question of whether or not poker is a game of skill comes up so often in legal contexts -- and since the UIGEA “makes the legality of poker under federal law also depend on a skill-versus-luck inquiry” (the authors point out) -- there exists a need for such a study, especially since Levitt and Miles believe “there is little academic research on the subject.”

The pair spend some time at the beginning referencing the few other studies there have been and pointing out flaws in those studies’ methodologies before introducing their own. Essentially, what Levitt and Miles did can be summarized as follows: (1) looked at complete lists of all of the entrants of all 57 events at the 2010 WSOP, noting how each performed; (2) identified a select group of “high skill” players among the total entrants and compiled their results separately; (3) compared how the “high skill” players did versus everyone else.

Their conclusion? “High skill” players did better than everyone else at the 2010 WSOP. Much better, in fact. Of the 32,496 individuals who entered events last year, the 720 players the researchers pegged as “high skill” collectively generated an average return on investment (ROI) of 30.5%, averaging a profit of more than $1,200 per event. Meanwhile, the other 31,776 players’ average ROI was -15.6%, which translated to an average loss of over $400 per player per event.

For Levitt and Miles, this result provides “strong evidence in support of the idea that poker is a game of skill.”

I know what you’re thinking. How exactly did they determine who among those who came to play at the WSOP last year were “high skill” players?

They identified six factors, all of which related to previous demonstrations of success in tournament poker. They looked at 2009 rankings by BLUFF Magazine, Card Player, and PokerPages.com, identifying players finishing in the top 250 of those lists as “high skill.” They looked at POY stats from Season 8 of the World Poker Tour, counting anyone who had accumulated any POY points as “high skill.” (In the WPT POY system, points are awarded for deep finishes, with the amount of prize pool affecting how deep one must go to earn points.) Additionally, they identified any player as “high skill” who had previously won a WSOP bracelet as well as anyone finishing in the top 250 in earnings at the 2009 WSOP.

Obviously this represents a fairly subjective system for determining who is a “high skill” player and who is not, but that’s the approach they used. Predictably, the 720 players who made the cut as “high skill” -- only about 2% of the total -- on average entered 2010 WSOP events about six times as much as everyone else, and thus comprised about 12% of the total entries. On average, the “high skill” folks entered about 12 events each, while everyone else averaged playing about two, with the “high skill” folks laying out nearly $50,000 each over the summer while the rest each spent an average of a little over $5,200.

Even so, it is safe to say the 720 players the researchers highlighted had all certainly demonstrated at least some success in tournaments previously, and as noted they outperformed the rest of the field significantly at the 2010 WSOP.

Response

My initial reaction to the study -- aside perhaps from wondering a little about that set of criteria for identifying “high skill” players -- is to say that the findings are utterly unsurprising. Anyone who has paid even a little bit of attention to the professional poker circuit knows that many of those who win these suckers or who appear at final tables are players we’ve seen before. As much as luck plays a role in tournament poker, skilled players do tend to succeed at higher rates than do unskilled players (however one wants to define what it means to possess “high skill”).

Then I thought a little more about the study and its methodology, and had a couple of other, more skeptical thoughts about its findings.

For one, I feel like a study that focuses solely on results at WSOP tournaments can’t possibly be comprehensive enough to build a convincing argument “that poker is a game of skill.” What about cash games? And what about that much-shared opinion among the community of poker players that in fact cash games are a better indicator of skill than tournaments?

Secondly, I think the researchers have overlooked one crucial factor here when making the case that past winners continuing to be successful in poker proves that we are talking about a skill game.

At the end of the study, an analogy is suggested between poker and baseball. Over the past few years, teams that made the MLB playoffs have on average come back the following year to win more games than they have lost. So, too, have we seen poker players who performed well in 2009 winning more than they lost in 2010 (at the WSOP, anyway). “To the extent that baseball would unquestionably be judged a game of skill,” say Levitt and Miles, “the same conclusion might reasonably be applied to poker in light of the data.”

On the surface, the continued success of the researchers’ “high skill” players might indeed appear to support the idea that poker is like baseball -- that the good players keep winning because their skill outstrips that of their opponents.

But there’s another obvious reason why the good players keep winning at poker. The money. Unlike in baseball, poker players have to win in order to stay in the game at all. Not only that, since money is part of the game, it necessarily affects how the game is played.

The researchers do acknowledge how stack size influences strategy within tournaments, noting how players who have accumulated big stacks have more options available to them than do their short-stacked opponents. But they don’t say anything about how this same discrepancy works with regard to players’ bankrolls and approaches to tournament play. Not only can their “high skill” players play more tournaments, but in many cases they can play them differently than can everyone else!

Not to get too abstract here, but it seems to me that it wasn’t just skill or experience that helped those 720 players do well at the WSOP. Having financial security to play six times as many events -- to risk about 10 times as much money at the WSOP (on average) than everyone else -- necessarily put them in a position where they’d be more likely to reap greater rewards.

All of which is to say I’m not too sure whether this study is going to make its way into the courts as unambiguous support for the argument that poker is a skill game. Still, it was nice to see some poker-positive press about the study come out last month in the wake of the hugely hurtful hit poker took on Black Friday.