Thursday, August 23, 2012

More on Luck vs. Skill in Poker: Two New Academic Studies Reach Opposite Conclusions

Continuing with this “luck-vs.-skill” debate in poker mentioned yesterday in the context of that Brooklyn federal judge’s ruling that Texas hold’em did not constitute illegal gambling as defined by the IGBA, I’m seeing two different academic studies appearing this month attempting to tackle the same issue.

One study comes from a trio of economists working at the Erasmus School of Economics located at Erasmus University in Rotterdam in the Netherlands. Their paper, “Beyond Chance? The Persistence of Performance in Online Poker,” examines a fairly large database of online poker hands to conclude “that players whose earlier profitability was in the top (bottom) deciles perform better (worse) and are substantially more likely to end up in the top (bottom) performance deciles of the following time period.” Such “persistence of performance” thus leads the researchers to conclude “that skill is an important factor in online poker.”

Meanwhile, a second study by another team of researchers working at the Institute of Psychology and Cognition Research at the University of Bremen in Germany arrives at a different conclusion. The four authors’ study, titled “Is Poker a Game of Skill or Chance? A Quasi-Experimental Study,” is due to appear in the September 2012 issue of the Journal of Gambling Studies, although it was made available online last week. Analyzing a much more modest sample size, the Bremen folks determined that “card distribution was the decisive factor for successful poker playing” rather than players’ skill levels, and thus their “findings indicate that poker should be regarded as a game of chance.”

The study by the Dutch group is a 32-page first draft that has been submitted to a journal for possible publication, and is available online for anyone to read at the Social Science Research Network site (click here or click the study’s title above). Meanwhile, the Germans’ full article can only be purchased unless you happen to have an academic affiliation and your school or university subscribes to the journal. Luckily for me I do have such an affiliation and my university does subscribe, so I’ve had a chance to download and read the 16-page study.

I’m neither an economist nor a psychologist, and so have to admit some of the methodology employed in both studies is a bit foreign to me. Still, I was able to follow both arguments reasonably well, and so thought I’d try to provide summaries of each here.

“Beyond Chance? The Persistence of Performance in Online Poker”

'Beyond Chance? The Persistence of Performance in Online Poker' by Rogier J.D. Potter van Loon, Martijn J. Van den Assem, and Dennie Van DolderThe Dutch economists begin by providing some context for their study, including explaining how both the popularity of online poker (and live poker) worldwide and ongoing battles over its legality inspired the study. They show a keen awareness of other research, too, including what behavioral research has found regarding decision-making in poker.

Thus do they begin with an understanding that players bring varying degrees of skill to poker, but also that players behave in ways that affect their performance, too (e.g., overestimating expectations, letting results affect decision-making, etc.). I’m glossing over specifics here, but mean to point out that I’m convinced the researchers appreciate the complexity of poker and how the skill component of the game involves a lot more than understanding probabilities, but also being able to manage one’s emotions and discipline oneself to make sound decisions consistently.

They go on to allude to other luck-vs.-skill studies of poker, including the one co-authored by Steven “Freakonomics” Levitt last year, positioning theirs as being methodologically similar to empirical studies like the one Levitt helped perform.

The researchers then describe how they analyzed a database of 76.7 million hands of real-money online NLHE played over the course of one year (from October 2009 to September 2010). Hands came from three different stakes -- 25NL, 200NL, and 1,000NL -- and involved over half a million different players. One player alone played 760,000 hands (!), while more than half of all players played less than 100 hands. The average number of hands each player played was around 1,000 for the middle and high stakes games and just over 400 for 25NL.

Of interest to many is the finding that 32% of all players in their sample “achieved a positive overall result after the deduction of rake,” a figure much higher than what I would’ve guessed. It should be noted the researchers ended up controlling for rake -- i.e., did not account for it when measuring performance -- since they’re aware of things like rakeback and bonuses and other factors that make it unclear just how much rake ultimately affects players’ profits.

Digging deeper, the researchers split the 12-month period into two 6-month periods, then looked at the first 6-month period, filtering out players who played less than 1,000 hands (the average ended up being about 5,800 hands per player). They worked out the BB/100 for each of those 32,716 players, then looked at those same players’ performances during the second 6-month period.

“In a nutshell,” they explain, “the results indicate that there is substantial and significant persistence in performance: deciles of players that performed relatively well in the first period on average continued to do so in the second period.”

The number-crunching continues from there, including performing similar analyses while breaking down the 12-months into four 3-month periods, as well as doing other work to try to account for varying sample sizes (i.e., to lessen the impact on the findings of those players playing fewer hands). The second half of the study then performs additional “regression analyses” that ultimately reinforce the initial findings that there does exist “persistence in performance” in poker, although to be honest this is where I have to admit my own limitations with comprehension.

The conclusion goes on to talk about brick-and-mortar games, tournament poker, and other variants not covered by the study. Suffice it to say, the study is quite thorough and self-aware, and its conclusion that “the results provide strong evidence against the hypothesis that poker is a game of pure chance” is quite persuasive.

“Is Poker a Game of Skill or Chance? A Quasi-Experimental Study”

Journal of Gambling StudiesI was very curious about this study by four German psychologists given its conclusion that luck rather than skill predominated in poker -- the first academic study I’ve heard of, actually, that does not come down on the side of skill. By the way, a “quasi-experimental study” refers to an empirical study in which the researcher gets to control the conditions. For example, rather than look at hands dealt by others (or by an online card room), the researchers can deal the cards and even minimize the randomness of how they are distributed.

Like the Dutch crowd, the authors of this study also begin with an overview of poker’s popularity, the abundance of online rooms (594 as of 2011, they say), and various legal battles in which the “luck-vs.-skill” issue is often central. They then look at previous studies which they characterize as “heterogenous and fragmentary” given the disparate types of methodologies employed. “Most research findings have suggested that skill is a meaningful element,” they admit, although qualify this statement by suggesting previous research has been “based on a relatively superficial examination.”

Moving on, the group next explains their methodology. They recruited 300 poker players via online advertisements, emailing them questionnaires about their poker skill and behavior, then paying their travel expenses to come participate. Those players were then given a second 27-item questionnaire designed to determine whether to classify each as an “average” or “expert” player.

The questions asked about their frequency of play, their self-perceptions about their own success and skill, and even what you might call their mental toughness (“How often do you mentally deal with poker, even if you do not play?”). A complicated method of grading the questionnaries was established, and ultimately the 300 players were divided evenly into two groups of 150 “average” players and 150 “expert” ones.

The 300 were then sat around 50 six-handed tables, half of which played no-limit hold’em while the other half played fixed-limit HE. The blinds were €0.03/€0.06 for both games, with each player given €10 with which to play. The LHE players got all €10 to start, while the NLHE ones got €5 with a chance to “rebuy” for €5 more.

“To provide an overall incentive to participate in the game and to ensure that the players’ game behavior would reflect actual play to the greatest extent,” there were prizes given to players who did well, although to be honest I’m not really following how those prizes were distributed. The study says “five special prizes of up to EUR 500 were offered to the best players at all of the tables,” whatever that means.

The games were played on computers (with players isolated from one another), following a “duplicate poker” format with all hands -- i.e., all the hole cards, flops, turns, and rivers -- having been predetermined. Players didn’t know this (i.e., they were led to believe the deal was random), nor did they know the game would be stopped after playing just 60 hands.

The deal was manipulated in a way that is also a bit ambiguous to follow, with players being dealt the “winning hand” and “losing hand” a predetermined number of times. I’ll boil it down to all players intentionally being dealt “good cards” and “bad cards” in a way the researchers believed to be evenly distributed, with the idea being to compare how well the “experts” did versus the “average players” in both cases (i.e., when running good and when running bad).

The results indicated what the researchers determine to be statistically insignificant differences between the performance of the “experts” and the “average players” when it came to playing good cards (or being dealt “winning hands”). That is, the “experts” did a little better, it sounds like, but not enough to matter. However, when dealt bad cards (“losing hands”), the “experts... outperformed average players... considerably.”

Again, as with the Dutch economists’ study, there are tables and breakdowns of the results, most of which extend beyond my understanding. But I think I’m following the overall argument, namely, that since “the only significant difference in cash balance outcome between experts and average players was under bad card conditions,” the researchers are concluding “there was no evidence for one group’s overall superiority in this context.”

“Therefore,” they go on to say, “it can be concluded that chance clearly dominates skill; thus, poker should be classified as gambling.”

To their credit, the researchers do admit to various limitations in their methodology, including a recognition that their subjects were self-selected and that their questionnaires had not been adequately tested for reliability. They also acknowledge they’ve gone with a super-small sample size in terms of number hands, although weirdly respond to that objection by speculating that “it could be assumed that with longer play periods, the difference in players’ level of skill would decrease,” a statement that seems to betray an obvious lack of understanding of the game they are studying.

There seem to be some huge problems with the Germans’ methodology that ultimately undermine the study in a significant way -- from the highly-suspect method of classifying of players as “expert” or “average” to the awkward dealing of “winning” and “losing” hands to the teeny-tiny sample size. More than enough to make their conclusion seem prematurely drawn, I’d say.

I remember reading an article in the Journal of Gambling Studies a couple of years ago, one by Kyle Siler called “Social and Psychological Challenges of Poker,” which seemed much more grounded and persuasive (discussed here). But that’s not the case with this one.

No, I’m still looking for a more convincing academic study proving once and for all that luck prevails in poker. After all, there seems to be a “persistence of performance” when it comes to these studies, with most ultimately having shown that skill is significant.

(Groovy “ambigram” graphic up top by Michael Irving.)

Labels: , , ,


Blogger There are some who call me... Tim said...

From an academic standpoint, that first study holds much more water. Large sample size taken over a length of time where skill and/or luck could be accurately determined, and using stakes that matter.

That second study fails due to a number of factors:
- Small sample size (60?!? hands, 300 players)
- "Expert" vs. "Average"? I play in a relatively small circle of players and I will vouch that, based on self-analysis we should be playing in Lake Wobegon - we'd all be above average.
- Lack of time for skill to come into play. How ofter do you read poker blogs where the writer takes it easy during the first few orbits so that they can learn about the other players? 60 hands are just 10 orbits - quick ones at that. Even playing with the folks I play with regularly I take it easy when I change tables - I need to time to read each player and see how they are thinking that night.
- Inconsequential stakes. When online money games were legal in the US, I'd play the $.01/$.02 games a whole lot looser than the $.50/$1.00 games. Having the subjects play for pennies was ridiculous. Check the last time Phil Hellmuth played in a $.05/$.10 game - he was probably still in diapers.
- Self-selected subjects. See my comment about Lake Wobegon... "Hey - I'd be GREAT for this subject 'cuz I'm such a skilled player!" Yeah - I know half were split into "average", but what were the factors that were used to determine expert versus average?

I am no expert, but for the past two years I've taken the annual "crown" in the game I play most frequently (NL Holdem tournament style with 20-40 players, results measured by wins, knockouts, relative ending place, etc.). {No applause, please... I am trying to remain humble} However, I played quite a bit during that time. This year I've played much less frequently, and the results show - if I make the top 20 this year I will consider it a success.

Unfortunately, the "gambling is evil" cadre will use that second, horribly designed study as "proof" that poker is a game of chance, not skill.

8/23/2012 2:15 PM  
Blogger Rakewell said...

Interesting. If valid, the German conclusion seems to be that the difference between the most-skilled and least-skilled players is not how much they win with the best hand, but how little they lose with the worst (or second-best) hand. This is an observation I first heard in my very earliest days of learning about poker. I think that it was Mike Sexton in one of the first WPT broadcasts, and I think he was quoting Doyle Brunson to that effect, but my memory is admittedly hazy as to the details. In any event, it's a truism in which I have come to believe more and more as time goes by.

As one great example, think about the Greg Raymer/Tom Schneider hand from last year's Main Event. Any old fool can win given Raymer's cards. But it takes a top pro like Schneider not to go broke with a premium but second-best hand in that spot.

Judging from your take of the German study, I would guess that the researchers don't have any idea why 60 hands is a woefully inadequate sample size. What really separates the best players from the rest is how well they observe and subsequently adjust to exploit the mistakes and weaknesses of their opponents. That just cannot be done meaningfully over 60 hands. It often takes hours of play to wait for and/or set up a situation that allows one to take advantage of a given opponent's specific vulnerabilities.

8/23/2012 2:24 PM  
Blogger James DiGioia said...

I don't understand how the second study concluded that luck is primarily a factor when they found themselves this:

However, when dealt bad cards (“losing hands”), the “experts... outperformed average players... considerably.”

That would seem to point to skill > luck as the same hand in different players hand end up with the same result.

8/23/2012 3:04 PM  
Blogger Aztec said...

One thing these studies illustrate is that the validity of any conclusions one draws is dependent on the quality of the data being analysed. The German team do not seem to appreciate the flaws in the data they collected, and it suggests to me that any “quasi-experimental” data will inherently be less reliable than data collected in the real world.

For what it’s worth, I struggle to understand how this skill v luck argument just keeps on running. Surely, once you have sound data to work from, it is obvious that at small sample sizes luck plays the predominant factor, but as the sample size increases skill takes over. Those who say it is a game of chance and those who say it’s mostly skill are both right – it just depends how you look at it.

8/24/2012 1:07 PM  
Blogger xpxinvxpy said...

The German study is nonsense. The giveaway is that it's psychologists doing statistics. Gee, what kind of credibility would I (as a statistician) get if I wrote a psychology paper? Why, then, should one necessarily believe psychologists when they do statistics? Yes, there are some psychologists who know statistics, but they are few in number.

Anyhow, here is a fact ground in rigorous probability theory: Tom Dwan took nearly two million dollars from Patrick Antonius over 39,000 hands of poker, and this was not sufficient to show that Dwan is a better player than Antonius. Consult chapter seven of the superb book on holdem by F. Paik Schoenberg for the mathematical details.

Does anyone think that the Germans had enough hands to reach a conclusion?

8/24/2012 5:58 PM  

Post a Comment

<< Home

Newer Posts
Older Posts

Copyright © 2006-2021 Hard-Boiled Poker.
All Rights Reserved.