More on Luck vs. Skill in Poker: Two New Academic Studies Reach Opposite Conclusions

Continuing with this “luck-vs.-skill” debate in poker mentioned yesterday in the context of that Brooklyn federal judge’s ruling that Texas hold’em did not constitute illegal gambling as defined by the IGBA, I’m seeing two different academic studies appearing this month attempting to tackle the same issue.

One study comes from a trio of economists working at the Erasmus School of Economics located at Erasmus University in Rotterdam in the Netherlands. Their paper, “Beyond Chance? The Persistence of Performance in Online Poker,” examines a fairly large database of online poker hands to conclude “that players whose earlier profitability was in the top (bottom) deciles perform better (worse) and are substantially more likely to end up in the top (bottom) performance deciles of the following time period.” Such “persistence of performance” thus leads the researchers to conclude “that skill is an important factor in online poker.”

Meanwhile, a second study by another team of researchers working at the Institute of Psychology and Cognition Research at the University of Bremen in Germany arrives at a different conclusion. The four authors’ study, titled “Is Poker a Game of Skill or Chance? A Quasi-Experimental Study,” is due to appear in the September 2012 issue of the Journal of Gambling Studies, although it was made available online last week. Analyzing a much more modest sample size, the Bremen folks determined that “card distribution was the decisive factor for successful poker playing” rather than players’ skill levels, and thus their “findings indicate that poker should be regarded as a game of chance.”

The study by the Dutch group is a 32-page first draft that has been submitted to a journal for possible publication, and is available online for anyone to read at the Social Science Research Network site (click here or click the study’s title above). Meanwhile, the Germans’ full article can only be purchased unless you happen to have an academic affiliation and your school or university subscribes to the journal. Luckily for me I do have such an affiliation and my university does subscribe, so I’ve had a chance to download and read the 16-page study.

I’m neither an economist nor a psychologist, and so have to admit some of the methodology employed in both studies is a bit foreign to me. Still, I was able to follow both arguments reasonably well, and so thought I’d try to provide summaries of each here.

“Beyond Chance? The Persistence of Performance in Online Poker”

The Dutch economists begin by providing some context for their study, including explaining how both the popularity of online poker (and live poker) worldwide and ongoing battles over its legality inspired the study. They show a keen awareness of other research, too, including what behavioral research has found regarding decision-making in poker.

Thus do they begin with an understanding that players bring varying degrees of skill to poker, but also that players behave in ways that affect their performance, too (e.g., overestimating expectations, letting results affect decision-making, etc.). I’m glossing over specifics here, but mean to point out that I’m convinced the researchers appreciate the complexity of poker and how the skill component of the game involves a lot more than understanding probabilities, but also being able to manage one’s emotions and discipline oneself to make sound decisions consistently.

They go on to allude to other luck-vs.-skill studies of poker, including the one co-authored by Steven “Freakonomics” Levitt last year, positioning theirs as being methodologically similar to empirical studies like the one Levitt helped perform.

The researchers then describe how they analyzed a database of 76.7 million hands of real-money online NLHE played over the course of one year (from October 2009 to September 2010). Hands came from three different stakes -- 25NL, 200NL, and 1,000NL -- and involved over half a million different players. One player alone played 760,000 hands (!), while more than half of all players played less than 100 hands. The average number of hands each player played was around 1,000 for the middle and high stakes games and just over 400 for 25NL.

Of interest to many is the finding that 32% of all players in their sample “achieved a positive overall result after the deduction of rake,” a figure much higher than what I would’ve guessed. It should be noted the researchers ended up controlling for rake -- i.e., did not account for it when measuring performance -- since they’re aware of things like rakeback and bonuses and other factors that make it unclear just how much rake ultimately affects players’ profits.

Digging deeper, the researchers split the 12-month period into two 6-month periods, then looked at the first 6-month period, filtering out players who played less than 1,000 hands (the average ended up being about 5,800 hands per player). They worked out the BB/100 for each of those 32,716 players, then looked at those same players’ performances during the second 6-month period.

“In a nutshell,” they explain, “the results indicate that there is substantial and significant persistence in performance: deciles of players that performed relatively well in the first period on average continued to do so in the second period.”

The number-crunching continues from there, including performing similar analyses while breaking down the 12-months into four 3-month periods, as well as doing other work to try to account for varying sample sizes (i.e., to lessen the impact on the findings of those players playing fewer hands). The second half of the study then performs additional “regression analyses” that ultimately reinforce the initial findings that there does exist “persistence in performance” in poker, although to be honest this is where I have to admit my own limitations with comprehension.

The conclusion goes on to talk about brick-and-mortar games, tournament poker, and other variants not covered by the study. Suffice it to say, the study is quite thorough and self-aware, and its conclusion that “the results provide strong evidence against the hypothesis that poker is a game of pure chance” is quite persuasive.

“Is Poker a Game of Skill or Chance? A Quasi-Experimental Study”

I was very curious about this study by four German psychologists given its conclusion that luck rather than skill predominated in poker -- the first academic study I’ve heard of, actually, that does not come down on the side of skill. By the way, a “quasi-experimental study” refers to an empirical study in which the researcher gets to control the conditions. For example, rather than look at hands dealt by others (or by an online card room), the researchers can deal the cards and even minimize the randomness of how they are distributed.

Like the Dutch crowd, the authors of this study also begin with an overview of poker’s popularity, the abundance of online rooms (594 as of 2011, they say), and various legal battles in which the “luck-vs.-skill” issue is often central. They then look at previous studies which they characterize as “heterogenous and fragmentary” given the disparate types of methodologies employed. “Most research findings have suggested that skill is a meaningful element,” they admit, although qualify this statement by suggesting previous research has been “based on a relatively superficial examination.”

Moving on, the group next explains their methodology. They recruited 300 poker players via online advertisements, emailing them questionnaires about their poker skill and behavior, then paying their travel expenses to come participate. Those players were then given a second 27-item questionnaire designed to determine whether to classify each as an “average” or “expert” player.

The questions asked about their frequency of play, their self-perceptions about their own success and skill, and even what you might call their mental toughness (“How often do you mentally deal with poker, even if you do not play?”). A complicated method of grading the questionnaries was established, and ultimately the 300 players were divided evenly into two groups of 150 “average” players and 150 “expert” ones.

The 300 were then sat around 50 six-handed tables, half of which played no-limit hold’em while the other half played fixed-limit HE. The blinds were €0.03/€0.06 for both games, with each player given €10 with which to play. The LHE players got all €10 to start, while the NLHE ones got €5 with a chance to “rebuy” for €5 more.

“To provide an overall incentive to participate in the game and to ensure that the players’ game behavior would reflect actual play to the greatest extent,” there were prizes given to players who did well, although to be honest I’m not really following how those prizes were distributed. The study says “five special prizes of up to EUR 500 were offered to the best players at all of the tables,” whatever that means.

The games were played on computers (with players isolated from one another), following a “duplicate poker” format with all hands -- i.e., all the hole cards, flops, turns, and rivers -- having been predetermined. Players didn’t know this (i.e., they were led to believe the deal was random), nor did they know the game would be stopped after playing just 60 hands.

The deal was manipulated in a way that is also a bit ambiguous to follow, with players being dealt the “winning hand” and “losing hand” a predetermined number of times. I’ll boil it down to all players intentionally being dealt “good cards” and “bad cards” in a way the researchers believed to be evenly distributed, with the idea being to compare how well the “experts” did versus the “average players” in both cases (i.e., when running good and when running bad).

The results indicated what the researchers determine to be statistically insignificant differences between the performance of the “experts” and the “average players” when it came to playing good cards (or being dealt “winning hands”). That is, the “experts” did a little better, it sounds like, but not enough to matter. However, when dealt bad cards (“losing hands”), the “experts... outperformed average players... considerably.”

Again, as with the Dutch economists’ study, there are tables and breakdowns of the results, most of which extend beyond my understanding. But I think I’m following the overall argument, namely, that since “the only significant difference in cash balance outcome between experts and average players was under bad card conditions,” the researchers are concluding “there was no evidence for one group’s overall superiority in this context.”

“Therefore,” they go on to say, “it can be concluded that chance clearly dominates skill; thus, poker should be classified as gambling.”

To their credit, the researchers do admit to various limitations in their methodology, including a recognition that their subjects were self-selected and that their questionnaires had not been adequately tested for reliability. They also acknowledge they’ve gone with a super-small sample size in terms of number hands, although weirdly respond to that objection by speculating that “it could be assumed that with longer play periods, the difference in players’ level of skill would decrease,” a statement that seems to betray an obvious lack of understanding of the game they are studying.

There seem to be some huge problems with the Germans’ methodology that ultimately undermine the study in a significant way -- from the highly-suspect method of classifying of players as “expert” or “average” to the awkward dealing of “winning” and “losing” hands to the teeny-tiny sample size. More than enough to make their conclusion seem prematurely drawn, I’d say.

I remember reading an article in the Journal of Gambling Studies a couple of years ago, one by Kyle Siler called “Social and Psychological Challenges of Poker,” which seemed much more grounded and persuasive (discussed here). But that’s not the case with this one.

No, I’m still looking for a more convincing academic study proving once and for all that luck prevails in poker. After all, there seems to be a “persistence of performance” when it comes to these studies, with most ultimately having shown that skill is significant.

(Groovy “ambigram” graphic up top by Michael Irving.)