Confessions of a Bot Runner Article

Check out this article, which is a pretty in-depth analysis of my poker botting activities based on what I’ve written here, my 2+2 posts, HackerNews comments, and more.

Quote:

  • Matt played HU SNGs on pokerstars part-time in 2006 and 2007. He seemed to have a fair success. He kept a blog in 2006 which is now archived on his mattmazur.com website. He was playing the 50s and 100s and taking looks at the 200 level, playing under the pokerstars name ‘kaon’. The blog stops in December, 2006. He states in the introduction that this is because he had started working on his poker bot project.
  • Matt posted on 2+2 under the name nichomacheo. You can see his profile and posts on the archive server and on the current server. He is still posting strategy up until late 2008.
  • He started posting his blog again in July 2007, a new domain. It was to log his return to HU SNGs playing on FTP. This was half way through his time running a pokerbot on stars (from screenshots, the the bot appears to have Full Tilt support). The blog only lasted 4 posts.

This was written by a professional poker player who goes by the name Hood who has “never written a bot … and advocate the strongest punishments for those who do

How to Calculate Your Equity vs Every Other Hole Card Combination

This is post #15 in an ongoing series of articles about my work as a poker bot developer.

One of the most useful command line tools I developed in the course of building the poker bot was a small program called eqall.

Eqall, short for Equity All, calculates your equity against every other hole card combination given the current board.

As a simple example, imagine you have Ks Qs and the board is Ad As Th 3h. Pretend your opponent has Js 9s. You’re in good shape vs this opponent–any jack will give you a straight and anything other than one of the three remaining nines gives you the win.

Your equity in this simple situation is 1 – the number of cards that will give your opponent the win over the number of cards left in the deck or 1 – 3/(52 – 4 – 2 – 2) or 1 – 3/44 or 0.931818.

What eqall does is it calculates your equity vs every possible hand your opponent can have. So it calculates your equity vs Js 9s, like in our example, plus Js Ts, Js 8s, Js 7s, and so on until it has calculated all of them.

You can download eqall.zip here (7KB).

Example:

>> eqall Ks Qs - Ad As Th 3h

Ks Qs - Js Ts = 0.204545
Ks Qs - Js 9s = 0.931818
Ks Qs - Js 8s = 0.931818
Ks Qs - Js 7s = 0.931818
Ks Qs - Js 6s = 0.931818
Ks Qs - Js 5s = 0.931818
Ks Qs - Js 4s = 0.931818
Ks Qs - Js 3s = 0.272727
Ks Qs - Js 2s = 0.931818
Ks Qs - Js Ac = 0.000000
Ks Qs - Js Kc = 0.921053
Ks Qs - Js Qc = 0.931818
...

The output is relatively simple: Ks Qs are your hole cards, the second column are the hole cards your opponent could have, and the last column is your equity, or your chance to win, against those hole cards.

Poker Bot Command Line Tool – AllHandsDesc

This is post #14 in an ongoing series of articles about my work as a poker bot developer.

Over the next several posts, I’m going to publish several command line tools that I developed in the course of building my poker bot.

None of these tools will enable anyone who can’t already build a poker bot to build one so I don’t think there’s much harm in posting them.

All of these were build on top of Poker Eval, an open source C library for doing poker calculations.

Tool #1: AllHandsDescC

Click here to download the ZIP file (6 KB)

Purpose: This tool will iterate over every possible hole card combination a player can have and spit out its rank when combined specified board cards.

Example:

>> allhandsdesc Td Ts 8h

As Ks - NoPair (A K 2 2 2) - OnePair (T 8 2 2) - OnePair (T A K 8) @ 280
As Qs - NoPair (A Q 2 2 2) - OnePair (T 8 2 2) - OnePair (T A Q 8) @ 292
As Js - NoPair (A J 2 2 2) - OnePair (T 8 2 2) - OnePair (T A J 8) @ 304
As Ts - NoPair (A T 2 2 2) - OnePair (T 8 2 2) - TwoPair (T 2 A) @ 282
As 9s - NoPair (A 9 2 2 2) - OnePair (T 8 2 2) - OnePair (T A 9 8) @ 316
As 8s - NoPair (A 8 2 2 2) - OnePair (T 8 2 2) - TwoPair (T 8 A) @ 119
As 7s - NoPair (A 7 2 2 2) - OnePair (T 8 2 2) - OnePair (T A 8 7) @ 328
As 6s - NoPair (A 6 2 2 2) - OnePair (T 8 2 2) - OnePair (T A 8 6) @ 340
As 5s - NoPair (A 5 2 2 2) - OnePair (T 8 2 2) - OnePair (T A 8 5) @ 352
As 4s - NoPair (A 4 2 2 2) - OnePair (T 8 2 2) - OnePair (T A 8 4) @ 364
...

Output Format:

There are five pieces of information per output line. Using the first line above as our example:

As Ks - NoPair (A K 2 2 2) - OnePair (T 8 2 2) - OnePair (T A K 8) @ 280

As Ks – Hole cards we’re checking

NoPair (A K 2 2 2) – This is the rank of the hole cards by themselves. It will either be NoPair or OnePair, in the case of a pocket pair. A K 2 2 2 is a way of representing the strength of the NoPair: Ace high, followed by king, and since we only gave it two hole cards, it defaults to twos for the rest of the five-card hand: 2 2 2.

OnePair (T 8 2 2) – This is the rank of the board cads by themselves. Td Ts 8h makes one pair: Two tens, followed by an eight, followed by two default 2’s. Note that the out shows “T 8 2 2 2” not “T T 8 2 2” because two tens are implied by its rank of “OnePair”.

OnePair (T A K 8) – This is the rank of the hole cards plus the board cards. As Ks Td Ts 8h makes one pair: two tens, followed by an ace, a king, and an eight.

@ 280 – This shows the number of hole card combinations that can beat these hole cards on this board. Consider a few example from this hand:

Tc 8d - NoPair (T 8 2 2 2) - OnePair (T 8 2 2) - FlHouse (T 8) @ 0

Since you hold a ten, it’s not possible for someone else to have quads, so you have the nuts–no hands can beat you.

Tc Th - OnePair (T 2 2 2) - OnePair (T 8 2 2) - Quads (T 8) @ 0

If you hold the two tens, you have quads, and there are no hands that can beat you.

Ks Tc - NoPair (K T 2 2 2) - OnePair (T 8 2 2) - Trips (T K 8) @ 10

If you hold Ks Tc, there are ten hands that can beat you: Six from full houses: Th 8c, Th 8d, Th 8s, 8d 8h, 8d 8s and four from higher trips: Th Ac, Th Ad, Th As, Th Ah.

If you have any questions, don’t hesitate to leave a comment below.

A Poker Bot Comment Worth Reading

This is post #13 in an ongoing series of articles about my work as a poker bot developer.

An anonymous poster left the following comment on my last poker bot post:

Hi Matt,

very, very interesting article series — please keep it up if you have more material.

I’ve also written a bot. It started as a little helper tool for my own, manual SNG-playing, showing a HUD on the table with the other players’ OPR stats etc etc. Having that in place, I couldn’t resist the temptation to see how hard it would be to do some table scraping, and when it turned out that was really straightforward, I started coding up a little rule-set that I believed would be able to win at low-stakes SNGs, where play is less than stellar…

After about 4-5 weeks, of off-hour, hobby programming during some evenings and weekends (I have a family with two small kids, so time is limited), I had it absolutely *crushing* the 18-man turbo $1.75 and $3.40 SNGs on Pokerstars — with a 50% ROI vs the field (ie not counting the rake). Actual ROI (counting the rake) was ~35%.

Table scraping etc was effective enough that the bot could easily 24-table (and probably way more — I never tested with over 24), but I kept it at 12-18 simultaneous games, to not go completely overboard on the volume.

After about 1.5 months, and 6-7k games, Stars’ security team busted me. I’m not sure what triggered it — it might have been a captcha I missed (I usually just left the bot playing overnight while I was sleeping…), it might have been decision patterns that set off an alarm, it might have been things they scraped off my computer (mouse movements or something like that — although as you also mention, I believe that’s actually quite unlikely, as they will get an insane amount of false positives, for TableNinja users etc), or — most likely — it might have been the volume that led to a manual investigation of the account.

Just before I was busted, there were seemingly random changes to the layout of the hole cards and board cards, and they were sometimes antialiased a little different. That was probably a trap in the Stars client that they can trigger from the server side, to see if the other side starts timing out or doing something else stupid. And my bot obviously fell right into it.

A couple of days after that happened, I then got the same email you got, about terminating my account, confiscating the money I had (I had made a withdrawal the day before, so that was a tiny sum), and expelling me forever from playing at Stars.

Oh, well. It was a short but interesting ride — but as you also say, it wasn’t really worth it.

Awesome.

And wow: 4-5 weeks to program a winning low stakes SNG bot with a 35% ROI. That’s pretty remarkable.

I’ve always had a suspicion that the abnormally high number of HUSNGs that my bot played/day was what ultimately led to its demise. I suspect that may be the case here too: 6-7K SNGs in 6 weeks–I bet that’s quite an outlier ;)

Another thing he mentions is the seemingly random changes to the layout, which I can also attest to. The tables would randomly resize by a few pixels, which would normally be imperceptible to a human player but for a bot that depends on certain pixels being in exact positions, it causes a lot of trouble.

Anonymous poster whoever you are, thanks for the fascinating story.

Building a Shortstacking Poker Bot – A Visual History

This is post #12 in an ongoing series of articles about my work as a poker bot developer.

In late 2006 and early 2007, I tried to build a shortstacking poker bot.

Shortstacking, in a nutshell, is a strategy that advocates buying in for a relatively small amount of money compared to your opponents and then playing very aggressive preflop poker. Most opponents do not adjust correctly to your stack size, and, in theory, you can make a lot of money from their mistakes.

Here’s a quote from shortstacking strategy guide on what used to be DonkeyDevestation.com:

By buying in with 20bb, you control the size of the stacks that the hand is being played for, not your opponents. Most of your opponents will be making plays preflop that are suitable for a full stack game but would be losers against 20bb stacks, and therefore losers against you. They take the worst of it against you to play better against the others, and you profit from this. Many players will complain about short stack players, claiming they are “lame” or “unskilled”. And they are right. It is lame, and it does take less skill to play a short stack. But it is very profitable, especially in today’s games. Furthermore, unlike most strategies, short stack play actually becomes more effective as you move up in limits, because other players tend to be more aggressive preflop, meaning they are putting more money in with hands they would not raise if they were only playing against you.

Shortstacking was appealing to me as a first time bot developer because it had been written about fairly extensively and the strategy, at least from a preflop perspective, was pretty well defined.

Here, for example, is a chart of a player named cuzco, who successfully employed a shortstacking strategy:

cuzco

How hard could it be to study his hand histories and have a bot emulate his decisions?

I already had a bit of experience writing add-ons for the PokerStars software (see PokerShark), so I figured it would only take a few weeks to get a prototype together.

In November 2006, I started programming.

The first major hurdle was figuring out how to read the chat box on the PokerStars client. PokerStars uses some custom controls which can’t easily be read using standard Windows API techniques, so I tried using Optical Character Recognition (OCR) to convert the pixels to words:

PokerStarsOCRInitial

And later:

PokerStarsOCRProgress

Eventually I figured out that PokerOffice, which makes a popular add-on for PokerStars, adds their own chat control on top of the default PokerStars one and it can be read using standard techniques. This was a big step because I no longer had to rely on OCR to get the text, which didn’t work very well (a lowercase L and a capital I look the same, for example).

Reading the text on the PokerStars Lobby was also important in order to automatically open new tables:

LobbyOCR

Using some shortstacking strategy guides as a starting point, I put together extensive preflop and postflop logic charts which were eventually implemented in the bot:

PreflopLogic

By the end of January, I had a working product:

InterfaceJan

It was capable of multitabling like the machine it was:

12Tables

Most days it lost money, which was to be expected during development, but some days it did very well:

PokerTrackerSwings

And that often gave me hope that the changes I had made the previous day were all that was needed to finally make it profitable.

I did a lot of A/B testing to try to measure what effect, if any, the changes I was making were effective:

ABTesting

Another thing I had to worry about were CAPTCHAs, which PokerStars is fond of using to identify bots:

CAPTCHA

Eventually I realized that conditional statements were not enough. There are simply too many situations in No Limit Texas Hold’em to account for–even when you’re shortstacking. For example, how do you take into account bet sizes and your opponent’s stack size? “IF … THEN” could not adequately handle that, so I started playing around with Expected Value (EV) calculations:

DecisionTreePostflop

Unfortunately at the time I didn’t have the tools to implement these techniques.

Nonetheless, I persisted with the shortstacking bot, believing that it only would take a few more changes before it was profitable.

March rolled by…

InterfaceMarch

And April came around…

InterfaceApril

But, despite my best efforts to turn it profitable, it kept losing.

By the time I decided to quit at the end April, the bot had played roughly 49K hands and blown through more than $1,300:

PokerTrackerGraph

Eesh.

Shortstacking, it seemed, was harder than it looked.

What I eventually realized is that your profits from shortstacking come from a few key decisions and if you didn’t make the correct moves at those moments, you couldn’t win enough to be profitable. In other words, I may have had a 90% solution–though I doubt it–but because it couldn’t make the correct decision that remaining 10% of the time, it could never be profitable in the long run.

Frustrated but determined not to let the work go to waste, I decided to start from scratch and to build a No Limit Heads Up Sit-n-Go (HUSNG) bot. That was my game of choice and I figured as long as I’m going to work on something, it might as well be with a game that I enjoy and had a lot of experience with.

Tatta, one of the most profitable Sit-n-Go players on PokerStars, was my inspiration:

tatta

“If I could only achieve a fraction of his success,” I reasoned to myself, “this will have been worth it.”

For the rest of this story, I’ll point you to the previous 11 posts about the HUSNG bot.

To sum it it: after several more months of work it was profitable, but shortly thereafter, PokerStars caught it and closed my account.

Doh. :)

Experimenting with a Neural Network-based Poker Bot

This is post #11 in an ongoing series of articles about my work as a poker bot developer.

At one point or another most poker bot developers have an epiphany. Their eyes open wide and they excitedly shout to the next stranger they see: “I’ll do it with a neural network!” Seems like such a good idea, right?

I tested it out and my definitive conclusion is “Maybe“.

A neural network (NN) is an AI technique that intelligently maps input values to output values. “Huh?” you say? Here’s the idea for poker: You give a NN data from previous hands you played (position, card values, hand rank, etc) and the decision you made in those situations (call, raise, fold, bet, check) and the NN will learn how to mimic those decisions. For a poker bot, this is a pretty appealing idea: you find the hand history of a winning, high stakes player, train the NN, and then set you poker bot loose to win a boat load of money.

Designing a Simple Test

A little background: My original goal for the poker bot was a full ring shortstacking bot. Shortstacking is a nasty little poker strategy that advocates aggressive play with with a relatively small amount chips. You see, when you don’t have a lot of chips to play with, you wind up making a lot of all in decisions preflop and relatively few postflop and most opponents do not adjust correctly to your strategy; most opponents play like they’re playing against someone with a normal stack, which is the absolute worst thing you can do against a talented shortstacker. This made shortstacking the perfect strategy for my fledgling poker bot.

At that time, the shortstacking bot made its decisions based on some elaborate conditional statements (ex: if you have QQ, KK, or AA and in early position, then raise). I had been testing the it for several weeks when I decided to try out the neural network idea, so I had plenty of data to work with.

To test it out, I picked a very specific situation that the shortstacking bot had faced many times in the past:  Everyone folds to you preflop at a full ring (8-9 players) table, do you raise or fold? The bot never called in those situations, so I didn’t have to factor that in.

There were a total of 10,461 hands that met that criteria. For the NN, I used 7 input values:

1. The numeric value of the first hole card scaled from 0 to 1. So, for example, 2 = 2/14 and Ace = 14/14.
2. The numeric value of the second hole card scaled from 0 to 1.
3. Whether or not they were suited. 0 = unsuited, 1 = suited
4. My position at the table where 0 = first to act, 7/9 = 0.778 = Dealer
5. The average value of the two hole cards
6. The difference between the first card’s value and the average
7. The difference between the second card’s value and the average

The output was simply a 1 if it had raised and a 0 if it had folded.

Here’s what the data looks like in an Excel sheet:

Poker Bot Neural Network

You can also download the spreadsheet by clicking here.

The Results

The predictions of the neural network were stunningly accurate:

Correctly predicted Raise: 776/865 = 88.6%
Correctly predicted Fold: 9475/9596 = 98.7%
Overall Accuracy: 10241/10461 = 97.9%

The predictions indicate that it is possible to make quality decisions based on the output of a neural network.

However, and this is a big however: the conditional statements that controlled the decisions in the training data were not very complicated so it wasn’t very hard for the NN to learn the pattern. Training it based on a human’s behavior may have led to very different results because a human’s thought process is much more complicated than “if this then that”. Normal decisions are not simply based on your hole cards and position at the table. You also have to take into account your stack size, your image, your opponents, the dynamics at the table, and a host of other factors. But, interestingly, this is exactly what a NN is good at: learning how a wide range of variables affect a decision.

Despite the success of this test, I ultimately decided not to pursue a neural network based poker bot. The problem is that you don’t have much control of the decision making process. You can’t, for example, look back at a hand and analyze why it made a specific decision. The neural network will spit out a number and the bot acts accordingly. There is no why; it’s merely a feeling it had. It’s also difficult to be precise. Say you want to always raise with AA, raise with KK half the time, and always call with QQ. It’s not a trivial task to adjust a neural network to make those type of decision if the training data indicates you did something else.

One final note: When I first started developing the poker bot in late 2006 I spoke with someone online who claimed to have built a profitable Heads Up No Limit Sit-n-go bot based solely on the predictions of a neural network that he had trained on his own hand histories. Legit? Who knows. Makes you wonder though…

Modeling Human Clicking Behavior on PokerStars

This is post #10 in an ongoing series of articles about my work as a poker bot developer.

There’s a lot of conjecture and speculation about what the online poker sites look at to detect bots. As bot developers, all we can really do is make educated guesses and hope that we fly under the radar long enough to make a profit.

If I had to guess on what they look at, mouse click location is near the top of the list.

We know that PokerStars records where you click because there is a log file (aptly named PokerStars.log) in the PokerStars directory which includes that information:

It’s not clear though whether they use this information to identify bots or as supplemental data in the event you need technical assistance from the PokerStars. Presumably if they were looking at mouse click location to detect bots they wouldn’t keep it in a log file which the user can edit.

Regardless, it’d be relatively easy for Jeff and team at PokerStars to do some statistical analysis on this data to flag suspicious activity, so if you’re going to develop a bot, you should try to make it act as human-like as possible.

But how do you know what’s normal activity?

For one, when the bot performs an action such as raising, don’t have it click the same exact location every time.

Here’s what I did:

That PokerStars log file contains the locations where you click, right? So why not take advantage of it.

I deleted the log file to reset the data and then joined a couple tables and played for a few hours. When I finished, I extracted the coordinates from the log file and plotted them on a screenshot of one of the table I had just finished playing at.

The end results show exactly where I clicked:

Most of the locations should be clear: Fold, Call, Raise, marking the Check box, viewing the hand history, and clicking the “Chat” and “Stats” tabs. The clicks around the center surprised me at first, but then I realized when I’m multitabling I click the center of the tables to bring the focus to it.

I based my bot’s click locations on this visualization. For example, the location of the clicks on the third button can be approximated by two overlapping normal distributions, one vertical and one horizontal with their intersection at the center of the button.

Here’s the code I used to implement it:

Is that the key to avoiding detection? Who knows.

I ran the bot for nearly two years, and while it may not have helped, it certainly did not hurt.

Best of luck —