The Humans and the Bots

Had a thought today about the world in which we currently live. It was poker-related, too -- in fact, online poker-related -- so I figured I might share it here.

Post-Black Friday my online poker playing essentially dwindled to some half-hearted noodling on a couple of the small, remaining sites, then disappeared entirely save the occasional play money game on PokerStars.

Not long ago I got an account on this new site called Coin Poker. It went live in November, and I believe it was sometime in December or maybe early January when I hopped on there for the first time. The site is “powered by blockchain technology via Ethereum,” and in fact the games are played with a newly invented cryptocurrency, “Chips” or “CHP” (now listed on a couple of exchanges).

The site had an ICO (Initial Coin Offering) -- actually a “pre-ICO” and then two stages of ICOs -- in which a good chunk of these CHPs were sold for Ethereum. Meanwhile the site has been conducting tournament freerolls to give away the rest of the CHPs. There were a lot of those early on, though the schedule has thinned a little lately.

It’s through the freerolls that I won some CHPs and began a modest “bankroll” on the site, something with which to play in the “cash” games. I haven’t explored where exactly things stand as far as depositing and withdrawing are concerned, and don’t really anticipate doing so soon (unless perhaps I were to run my small total up significantly).

Playing on the site has been diverting, though, and for the first time in several years I have found myself genuinely invested in the games when playing poker online. I’ve even revived some of those earlier online poker memories of pleasure and pain associated with wins and losses, to a lesser degree of course.

When I first started on the site, I’d join the freerolls which like all the games on the site are played either four-handed or six-handed. Very frequently there would be players at the table shown as sitting out, something I grew accustomed to quickly. At a six-handed table there might be three or four seats occupied by the non-playing entrants, and occasionally at four-handed tables I might be the only live one there just scooping up blinds and antes until the field got whittled down.

At the time I assumed the site was filling the empty seats with these “dummy” players just to make the freerolls last a little longer, or perhaps to foster the impression of more traffic than there really was. Whatever the reason, I haven’t noticed the sitter-outers as much lately, or at all, really. As the site has grown a bit more popular, I imagine if there were such a strategy employed before it has now been withdrawn. (I’m only speculating.)

I wasn’t bothered too much by all the players sitting out, although the presence of all of those silent “zombies” at the table did cause me to recall the controversies and occasional hysteria surrounding the use of “bots” in online poker. Coupled with some of the news from the past few weeks (and months), that in turn has made me think about the significant influence such software applications running automated tasks or scripts online now have upon our lives.

It’s an enormous subject, but in particular I’m thinking about those indictments handed down last Friday by Special Counsel Robert Mueller that charge 13 Russian nationals and three Russian entities with conspiracy to defraud the United States via their attempts to meddle with the 2016 U.S. presidential election. If you’ve read through the 37-page document spelling out what happened (or heard it summarized), you’re familiar with some of the methods employed by these agents to manipulate news and opinion consumed by Americans during the campaign, especially via social media.

The report describes in detail how a Russian company called the “Internet Research Agency” (a name sounding equally generic and sinister) employed hundreds to help generate content published via fake accounts with invented personas on YouTube, Facebook, Instagram, and Twitter, content that was in turn disseminated far and wide “via retweets, reposts, and similar means.”

The network has been characterized as a “bot farm” and even this week there was evidence of the network or something similar continuing to operate via the rapid spread of various messages (including false ones) in the wake of the deadly school shooting in Lakeland, Florida a week ago.

One of the more curious aspects of the “disinformation operation” (as some have described it) is the way invented news and opinion gets picked up and further distributed by unsuspecting social media users (i.e., Americans not involved with the operation). The indictment describes “unwitting members, volunteers, and supporters” of the campaign the Russians were supporting as having performed such work along with others “involved in local community outreach, as well as grassroots groups.”

In other words, certain messages and information “campaigns” begun by this Internet Research Agency were initially promulgated by a vast number of fake accounts with programs or “bots” helping extend their reach and influence. Then actual, living and breathing humans receiving those messages (and unaware of or unconcerned about their origin) passed them along as well, increasing their audience and influence.

Setting aside questions of legality and jurisdiction (and ignoring entirely the many other areas being explored by Mueller and his team), I just want to isolate that phenomenon of an automated message sent via a “bot” being received and then resent by a human. The fake accounts being directed by the scripts are simply executing commands. The humans who then receive and resend those messages do so consciously, although they, too, act by rote in a sense, simply hitting “like” and “retweet” in what is often an uncritical fashion. (Bot-like, you could say, depending on your predilection for irony.)

When playing against the “dummy” non-players in those freerolls, I could comfortably bet or raise against them every single time, knowing full well that even though they might resemble “human” players sitting there at the table, they weren’t going to play back at me. They were programmed simply to fold every time the action was on them. If you’ve ever played against “the computer” in crude games (including poker games), you’ve probably similarly been able to pick up on the program’s patterns and exploit them to your favor.

Of course, increasingly sophisticated programs have been created to run much more challenging poker playing “bots,” including those powered by artificial intelligence. These programs can in fact exploit the tendencies of humans who often find it very difficult to randomize their actions and thereby avoid detectable patterns. It’s much harder to know what to do against these, as some of the more recent efforts in this area have demonstrated.

Many of those who forwarded along memes, photos, articles, and other bot-created content during the 2016 presidential campaign weren’t aware of the original source of that information (were “unwitting”). They were -- and are, still -- being exploited, in a way, by others who know how they tend to “play” when using social media.

The “game” is getting a lot harder. As far as social media is concerned -- and news and politics and everything else in our lives that has now become so greatly influenced by message-delivering mechanism of social media -- it’s becoming more and more difficult to know who is human and who is a bot pretending to be human.

Especially when the humans keep acting like the bots.

(EDIT [added 3/19/18]: Speaking of CoinPoker and bots, there’s an interesting new article on PartTime Poker sharing some research regarding the site’s unusual traffic patterns. The title gives you an idea of the article’s conclusion -- "CoinPoker’s Traffic is a Farce.")

Image: “Reply - Retweet- Favorite” (adapted), David Berkowitz. CC BY 2.0.

Playing Cepheus

A short follow-up regarding Cepheus, the heads-up fixed-limit hold’em playing program developed by the Computer Poker Research Group at the University of Alberta I was writing about yesterday -- i.e., the program that is said to have essentially “solved” heads-up LHE insofar as (the researchers claim) “a human lifetime of play is not sufficient to establish with statistical significance that the strategy [employed by Cepheus] is not an exact solution.”

I mentioned how they’ve put Cepheus online for the curious to play against. Today after queueing up for a long time I managed to get a game against the program. We played 100 hands of 10/20 LHE, after which I managed to finish up 105 units -- just about five big bets.

I ran hot early on, so hot it almost seemed like things were rigged in my favor as I built up a lead of over 250 through the first 30 hands. Then things evened out between us and after 49 hands we were dead even, and for a hand or two after that I was down briefly. But I won three big pots in a row to zoom back up over 200, and ultimately never lost the lead again.

I played tight-aggressive throughout, becoming a little more conservative during the last dozen hands or so as I wanted to preserve my lead. Both Cepheus and I were mindful of position, with Cepheus raising almost every single button and folding otherwise (i.e., never limping). Meanwhile I also mostly raised or folded my buttons (folding more than Cepheus did), though I limped occasionally, too.

Cepheus would three-bet me fairly often before the flop when I did raise, and probably bet when checked to around 80-90% of the time (I don’t have a log of the hand histories, so can’t say for sure). After about 75 hands I had just begun to become aware of the fact that Cepheus hadn’t seemed to have check-raised me on either the turn or river, then the program did it twice within just a few hands, both times successfully earning extra bets as a result.

In the first case I was playing from the button with K-7-offsuit with the king of clubs, and the flop had come all clubs with the ace to give me a nut flush draw. Cepheus check-called me there, then a king fell on the turn and that’s where Cepheus check-raised me. The river was a blank, and Cepheus won the hand with K-Q.

The second instance also involved Cepheus connecting on the turn -- that’s a screenshot of that hand above (click to enlarge). I also managed to check-raise a couple of turns after making hands to get extra value.

Obviously the tiny sample established practically nothing regarding either Cepheus or myself. I will admit that toward the latter part of the session I felt my attention flag just a touch, enough to remind me of the difference between myself and my non-human competitor. If we’d gone on, say, to play 1,000 hands or more I imagine it would have been very difficult for me to continue to make correct decisions (not that all of the ones I made were correct).

If you happen to play Cepheus, let me know how it goes and what impressions you get from the program.

Cepheus and the “Solving” of Heads-Up LHE

Yesterday I was writing about the “New York Times 4th Down Bot” and related efforts by those studying stats to present math-based models for dictating how best to coach a football game. There was a related story last week regarding the team of researchers at the University of Alberta who have been working for several years on “solving” heads-up fixed-limit hold’em, with their new study published in the January 2015 issue of the journal Science “announc[ing] that heads-up limit Texas hold’em is now essentially weakly solved.”

I remember many years ago -- way back in the summer of 2007 -- speaking with Jonathan Schaeffer of the University of Alberta about Polaris, the computer program developed by his team of researchers that comprised the university’s Computer Poker Research Group (CPRG) in the dept. of Computing Science which he chaired. An LHE match between Polaris and two pros, Phil Laak and Ali Eslami, took place that summer in which the humans won, and for PokerNews I spoke to Schaeffer about Polaris and the CPRG’s goals.

“One of these days -- within 5 to 10 years -- two-person, limit hold’em will be solved,” he said to me. Here we are about seven-and-a-half years later, and it sounds like those continuing the work there at Alberta have fulfilled Schaeffer’s prediction.

The new heads-up LHE-playing program is called Cepheus and is in fact available to play against online, although when I went to the site there were “too many in queue” and I was invited to come back later to try.

With my academic affiliation I have been able to get a copy of the Science study and have read through it. Simply titled “Heads-up limit hold'em poker is solved,” it begins by pointing out that while certain “perfect-information” games like Connect Four and checkers have been solved, others like chess have not even if much-celebrated events like Deep Blue’s victory over Garry Kasparov has led some to suggest it has. “Defeating top human players is not the same as ‘solving’ a game,” the study’s four authors point out.

The study then notes how only “perfect-information” games have been solved thus far, making their claim regarding the “imperfect-information” game of heads-up limit hold’em groundbreaking. Reference is made to the work of game theory pioneer John von Neumann and the element of bluffing that distinguishes poker, to Michael Craig’s The Professor, the Banker, and the Suicide King (which details the high-stakes LHE games between by Andy Beal and “the Corporation”), as well as to the previous work of Schaeffer and his team at Alberta.

An explanation of what it means to “solve” a game follows, with the distinction “weakly solved” referring to a game in which “for the initial position(s), a strategy has been determined to obtain at least the game-theoretic value, for both players, under reasonable resources.” From that definition, the researches extrapolate that as far as heads-up LHE goes, it is safe to say that the game is “essentially weakly solved... if a human lifetime of play is not sufficient to establish with statistical significance that the strategy is not an exact solution.”

From there comes further refinement of what is meant by an “imperfect-information game” then an explanation of the programming of Cepheus and the “solution” ultimately found. This admittedly is where your humble scribbler feels especially humbled, not being versed in the various fields of the researchers.

Back in 2007 there was already lots of talk about “poker bots” and online poker, so I had to ask Schaeffer about Polaris and how it might relate to the online game. “I want to be clear,” he told me. “We do not play online poker. None of our software is enabled to play online poker on any of the sites.”

Nor has being able to “solve” heads-up limit hold’em ever been the endgame for the researchers at Alberta. They conclude their study by noting how “the breakthroughs behind our result are general algorithmic advances that make game-theoretic reasoning in large-scale models of any sort more tractable.” In other words, as often gets pointed out by those who study game theory as it applies to recreational games or sports, the findings there have value in other realms involving human decision-making.

As a longtime fan of LHE, I’m curious to learn more about Cepheus, including how it was created and what it can do. If you are also curious, here’s a good piece on the FiveThirtyEight site summarizing the team’s work and placing it in a broader context by Oliver Roeder called “Computers Are Learning How to Treat Illnesses By Playing Poker and Atari.” And I’m about to go listen to the latest episode of the Thinking Poker podcast on which Andrew Brokos had on as guests two of the study’s authors, Michael Bowling and Michael Johanson of the CPRG at Alberta, to talk about Cepheus.

In the meantime I’m going to keep queueing up to try to play Cepheus. Last night Vanessa Selbst was on PokerStars playing at some some 8-game play money tables, and I actually sat at one for a few orbits. Feeling equally intimidated to sit across from Cepheus, I think.

Being Human

Victoria Coren wrote a neat, short Guardian piece yesterday titled “How do you find the best player in the world?”

There she reflects briefly on the recent International Federation of Poker event in which teams from 11 nations (plus a team from the “virtual” nation of Zynga) completed against one another using the duplicate poker format. I wrote a little about the IFP event, though not so much about duplicate poker, in my Community Cards column for Epic Poker this week, “Poker as a Sport.”

Coren’s succinctly-made point yesterday was to point out how difficult -- really, impossible -- it is to rank poker players according to any utterly unambiguous scale. “I rather like the impossibility of naming anyone ‘best,’” writes Coren, adding that “the ensuing, unceasing argument is so human.”

I rather like Coren’s choice of adjective to conclude that thought. It is “human” to attempt such futile tasks. And it’s our being “human” that helps contribute to the impossibility of objectively ranking poker players.

She ends her column with a quote from the last page of Richard Jessup’s novel The Cincinnati Kid, a book I wrote about here some years ago. The quote is in fact presented in the novel as an idea Christian (Eric’s girlfriend) tries to impart to the Kid. “For every number one man there is a number two man,” goes the idea, “and because of this a man cannot retreat from life.”

Then comes a pronouncement about the seemingly unbeatable Lancey Howard: “The difference is that the number one man is a machine and the Cincinnati Kid is not, and was not, and never will be a machine.”

The implication that Lancey is “a machine” sounds an ironic note when we recall his nickname -- “The Man.” Another implication, of course, is that being human means being capable of losing. That no “number one” can ever continue as such without being challenged. Not if he’s human, that is.

All of that talk resonated strongly with me today as we just happen to be reading and discussing “poker bots” and online poker in my “Poker in American Film and Culture” class. Our readings consider recent efforts in artificial intelligence to create poker-playing computer programs -- i.e., to make machines more human-like -- as well as how online poker might have the effect of making humans more machine-like.

All of these items -- artificial intelligence, poker bots, online poker, the fictional character Lancey Howard -- encourage us to consider the significance of the human element in poker. And how it is our flaws and our efforts to exploit those of others and suppress our own that make the game interesting and meaningful -- not a “retreat from life,” but an expression of it.

Robot Rights (and Wrongs)

I enjoy playing online poker. Strictly small stakes, natch. Been doing so for several years now, and hope to continue playing in the future, too.

With online poker, there’s always a leap of faith. One can never know with 100% certainty that one is not being dealt a crooked game. But I can say that almost every time I’ve ever sat down to play I’ve felt reasonably sure the games have been on the up-and-up.

How long, though, can online poker -- as we’ve been enjoying it over these last few years -- survive in its current form? Have to admit, this recent “bots” business has me wonderin’...

Some News About “Some of your opponents”

Over the weekend there was quite a lot of chatter on the forums, Twitter, and elsewhere in response to the fact that on Friday a large number of players had received some sort of refund or compensation from Full Tilt Poker.

Most of those receiving money were sent emails which began by announcing that the “Full Tilt Poker Security department has recently concluded an extensive investigation and we have determined some of your opponents were in violation of our site terms.” The email went on to note that the offending accounts had been “permanently closed,” the money confiscated, and refunds awarded to affected players “based on the number of tournaments or hands played against the offending players, and the amount won or lost against them.”

The emails provided no other specifics, only adding that “for a number of reasons, we are unable to provide any other additional information” regarding the violations of the site terms. It looks as though some players might have begun receiving refunds as early as late September, actually, then late last week came an especially high number of refunds, prompting players to congregate on the forums and elsewhere to figure out why exactly they were getting money returned to them.

By Friday afternoon, an explanation had been found, thanks primarily to the fact that those responsible for the site term violations had posted a complaint on their website about their accounts being shut down and funds confiscated. That’s right -- the cheaters have a website, and a forum, and what appears to be a thriving little community of bot-promoters and -users.

Turns out “some of your opponents” were not who you thought they were. There was a reason why that dude always seemed to three-bet from the blinds following your late-position attempts to steal. “He” was programmed to do so!

Beware the Bots

The site where the complaint appeared is called “Bonus Bots” and is operated by a group that identifies itself as “Shanky Technologies.” They have apparently been up and running for more than two years now, selling a number of different “poker bot” products that play all sorts of games and work on various sites. A bit of poking around shows one can download demo versions of the programs that’ll run for a couple hundred hands (for hold’em, Omaha/8, pot-limit Omaha, and even blackjack), then unlock them by purchasing licenses which range from $99 to $149.

The “bot” programs are obviously functional, playing both cash games and tourneys for the players who run them. Whether they have been programmed well enough to win consistently is difficult to determine, but it appears as though they had at least enabled many of those who were using them to profit via rakeback.

And while the number of players and accounts that had been using the Shanky Technologies bots is not known, from the forums it appears there were a lot of them, primarily hanging out in the lower buy-in SNGs, MTTs, and ring games, especially Rush Poker where they perhaps could more easily go undetected by players playing against the bots.

The use of “poker bots” of course violates one of Full Tilt Poker’s “site terms,” the one stating that “the use of artificial intelligence including, without limitation, ‘robots’ is strictly forbidden.” Full Tilt also states that by playing on the site one agrees to allow FTP to “take steps to detect and prevent the use of prohibited EPA [external player assistance] programs,” including the “examination of software programs running concurrently with the Full Tilt Poker software on a player’s computer.”

Other sites have similar restrictions, although the policing against bots is inconsistent. Some have noted that PokerStars appears to be the only U.S.-facing site that has remained constantly vigilant in detecting bots and keeping them off their games.

“An Incredibly Unethical Move”

The complaint from Shanky Technologies is fascinating for a number of reasons, including the fact that it begins by noting that Full Tilt “came out of nowhere” with “no warning” to stop the use of the bots, “and this was after years of tolerating us to such an extent that most of us felt they welcomed us with open arms.”

Indeed, the unapologetic manner in which the bots are discussed on the site’s blog and forums does suggest not only that they’ve been able to use their programs for some time, but that Full Tilt and several other sites -- not including PokerStars or PartyPoker -- have turned a blind eye to their use for quite some time. And some continue to do so, apparently, including UB and Bodog.

The complaint goes on to mention that “hundreds of us” had been using the programs without incident. The move by Full Tilt Poker to freeze their accounts and confiscate their funds is described as “a very sudden and calculated move on their part.” “It would be difficult to describe their actions as anything other than stealing,” says Shanky Technologies, “if you understand the environment.”

They go on to describe Full Tilt’s decision finally to enforce their site term as “an incredibly unethical move” and suggest players stay away from the site, even those who might wish to play there “manually” (i.e., without a “bot”).

Pretty friggin’ audacious, eh? Kind of reminds me of some of those sci-fi stories which try to explore the subject of “robot rights.” Usually such stories involve the development of consciousness or “feelings” in the robots, thereby raising questions about whether or not they might or should be treated similarly to humans and/or afforded the same “rights,” including the right to live.

But Bots are People, Too!

While acknowledging the fact that they’ve been violating one of Full Tilt’s site terms, the makers and users of the poker bots adamantly believe the use of bots to play online poker does not affect the game’s integrity. In other words, they appear ready to argue for the “right” of their bots to compete against humans, even if the humans -- both the players and those ones running the site -- do not wish to allow such to occur.

One of the users of the Shanky bots started a thread on Two Plus Two over the weekend defending their use as “a legit form of poker,” claiming “it’s my mind dictating my bot’s actions” and even going so far as to describe using the bot as “a form of poker at its purist.”

Hard to know for sure whether the poster was being truthful or simply provoking. As you might imagine, most of those responding to the post disagreed with its argument, pointing out how poker “at its purist” is a game between people, not a competition to see who can construct the best poker-playing program.

Even so, online poker has always been different from the live game. Sure, it’s a game between people, but we each are operating a machine in order to compete with one another. To what extent are those machines representing us?

Are you human? Am I?

The more I think about it, the more I find the whole incident troubling. Some are suggesting that at certain levels (NL2 and NL5), the games at Full Tilt were “saturated” by these poker-playing bots prior to last week’s action. Again, it is hard to know what to believe, but one assumes that those creating the programs will work further to try to develop ways of running their programs without being detected (“stealth measures,” as the Shanky guys put it).

I keep thinking of that one statement, the one where the makers of the bots try to characterize Full Tilt suddenly bringing down the hammer as “stealing,” explaining that it would appear as such to you, too, “if you understand the environment.”

The reference there is to the lackadaisical enforcement of site terms, although the line makes me think in broader terms about the game we have enjoyed for so long. And we thought we knew.

How well do most of us playing online poker really “understand the environment”?