At one point or another most poker bot developers have an epiphany. Their eyes open wide and they excitedly shout to the next stranger they see: “I’ll do it with a neural network!” Seems like such a good idea, right?
I tested it out and my definitive conclusion is “Maybe“.
A neural network (NN) is an AI technique that intelligently maps input values to output values. “Huh?” you say? Here’s the idea for poker: You give a NN data from previous hands you played (position, card values, hand rank, etc) and the decision you made in those situations (call, raise, fold, bet, check) and the NN will learn how to mimic those decisions. For a poker bot, this is a pretty appealing idea: you find the hand history of a winning, high stakes player, train the NN, and then set you poker bot loose to win a boat load of money.
Designing a Simple Test
A little background: My original goal for the poker bot was a full ring shortstacking bot. Shortstacking is a nasty little poker strategy that advocates aggressive play with with a relatively small amount chips. You see, when you don’t have a lot of chips to play with, you wind up making a lot of all in decisions preflop and relatively few postflop and most opponents do not adjust correctly to your strategy; most opponents play like they’re playing against someone with a normal stack, which is the absolute worst thing you can do against a talented shortstacker. This made shortstacking the perfect strategy for my fledgling poker bot.
At that time, the shortstacking bot made its decisions based on some elaborate conditional statements (ex: if you have QQ, KK, or AA and in early position, then raise). I had been testing the it for several weeks when I decided to try out the neural network idea, so I had plenty of data to work with.
To test it out, I picked a very specific situation that the shortstacking bot had faced many times in the past: Everyone folds to you preflop at a full ring (8-9 players) table, do you raise or fold? The bot never called in those situations, so I didn’t have to factor that in.
There were a total of 10,461 hands that met that criteria. For the NN, I used 7 input values:
1. The numeric value of the first hole card scaled from 0 to 1. So, for example, 2 = 2/14 and Ace = 14/14.
2. The numeric value of the second hole card scaled from 0 to 1.
3. Whether or not they were suited. 0 = unsuited, 1 = suited
4. My position at the table where 0 = first to act, 7/9 = 0.778 = Dealer
5. The average value of the two hole cards
6. The difference between the first card’s value and the average
7. The difference between the second card’s value and the average
The output was simply a 1 if it had raised and a 0 if it had folded.
Here’s what the data looks like in an Excel sheet:
You can also download the spreadsheet by clicking here.
The Results
The predictions of the neural network were stunningly accurate:
Correctly predicted Raise: 776/865 = 88.6%
Correctly predicted Fold: 9475/9596 = 98.7%
Overall Accuracy: 10241/10461 = 97.9%
The predictions indicate that it is possible to make quality decisions based on the output of a neural network.
However, and this is a big however: the conditional statements that controlled the decisions in the training data were not very complicated so it wasn’t very hard for the NN to learn the pattern. Training it based on a human’s behavior may have led to very different results because a human’s thought process is much more complicated than “if this then that”. Normal decisions are not simply based on your hole cards and position at the table. You also have to take into account your stack size, your image, your opponents, the dynamics at the table, and a host of other factors. But, interestingly, this is exactly what a NN is good at: learning how a wide range of variables affect a decision.
Despite the success of this test, I ultimately decided not to pursue a neural network based poker bot. The problem is that you don’t have much control of the decision making process. You can’t, for example, look back at a hand and analyze why it made a specific decision. The neural network will spit out a number and the bot acts accordingly. There is no why; it’s merely a feeling it had. It’s also difficult to be precise. Say you want to always raise with AA, raise with KK half the time, and always call with QQ. It’s not a trivial task to adjust a neural network to make those type of decision if the training data indicates you did something else.
One final note: When I first started developing the poker bot in late 2006 I spoke with someone online who claimed to have built a profitable Heads Up No Limit Sit-n-go bot based solely on the predictions of a neural network that he had trained on his own hand histories. Legit? Who knows. Makes you wonder though…
Matt Mazur 
If you’re worried about being able to understand why it makes a decision, you might want to look into a neurofuzzy classifier.
Interesting. I haven’t used neural networks before, but I’ve been reading about them very recently. If you had to choose a tool or framework to use for this, which would you choose?
This seems like a ripe field for innovation. Good post.
Interesting that you mentioned originally wanting to create a full ring shortstacking bot. What were your experiences with it? Did you switch to botting HU SNGs because the shortstacking bot was not profitable?
It’s nice to see you continuing to post about your botting experiences :)
Hi- posted this question on an older topic so I’m not sure you’ll see it.
I was wondering if you know of any poker sites that welcome and accept bots?
Few follow ups:
Dave, I coded the neural network from scratch; I’d recommend doing the same, just to learn how they work. They’re not too hard to do. I’d post the code here, but it’s no where to be found.
Ethan, you’re exactly right. I’ll post more about it, but it insurmountably hard writing a conditional shortstacking bot. The margins are very slim and you have to make great decisions in a few key postflop situations in order to grind out a profit. After realizing that, I set out to rewrite the whole thing but figured as long as I’m doing it, it might as well be something I was good at and enjoyed: HUSNGs.
Jon, no, no sites allow bots as far as I know of. Check with the PokerAI.org folk, they’d know better.
Hey matt. When did you do this experiment? I didn’t realize you ever got a neural net to be so accurate… that’s really awesome. Call me sometime, we should catch up; i lost my phone, and hence your number.
Hey Woody — this was early May 2007.
I’ll call you sometime later today.
Jon,
There are some poker sites that would “not mind” bots (They will both not forbid them in their Tos, nor harass with people for using them). Also, it actually depends on how you define bots. I define it as just about any software that aids your poker play, HUDs included. That might not look so resonable to you, but classificaiton of poker software is anyway quite ridicolous nowadays. There are sites e.g. that will write in their ToS that they do not allow bots, but will allow “auto-folders” – that is you can configure which hands you want to play, and it will automatically fold the rest preflop. PokerStars will no allow this, on the other hand. So everyone tries to draw the line somehow with definitions that otherwise (from non-business point of view) doesn’t make huge sense.
You can check specifically this subforum for a lot of discussions on legallity and morality of using bots:
http://www.pokerai.org/pf3/viewforum.php?f=80
Hi!
For everyone who wants to try NNs I would suggest to use MatLab’s Neural Networks Toolbox. Inputs, outputs, layout, learning rules etc can be configured. The language is quite easy also. Also, it is a wise thing to divide the data into two sets: training & testing. If you can train the NN to achieve ~100% accuracy it will probably fail to respond right to unknown inputs. I think it is called ‘overfitting’ the training data. You can avoid overfitting if train the NN on the training set and you monitor the accuracy on the training set and the (unknown) test set. To the project: since cells in NNs do (weighted) additions, I think it is unnecessary to use average, difference and such things as inputs beside the card values. On the other hand, it would be useful to give the NN the number of bets/raises before/after your position according to the shortstackink strategy I know. If someone has results in the topic I would be glad to read about it :)
I like to play poker online
This is very interesting subject
Speaking of data to teach such a NN. Does someone knows, where to get a history, in a computer-readable form, of a large number of hands, by a world class player?
Game histories used to be available from the University of Alberta AI game group.
I found reference to this when looking for game history.
Sadly, they no longer provide it unless you are a researcher ( a professor/grad student ).
By the way, you don’t need world class player information, just real money games will be be enough. If you can create a bot capable of holding its own at a PokerStars Sit-And-Go, you are doing great.
My suggestion is to use your software to help you make decisions, not make decisions for you.
:-)
I recently came across your blog and have been reading along.
I thought I would leave my first comment. I don’t know what to say except that I have enjoyed reading.
Nice blog. I will keep visiting this blog very often.online poker
It is true that poker is not just luck, you have a good memory is in your advantage for long term play, neural network could be a method.
Online gambling requires a mix of luck and knowledge, the solution is to improve the knowledge because the luck cannot be change.
It is unbelievable what a human brain can be simulate through the neural network.
It is hard to say the percentage assigned to luck and percentage assigned to tactic in poker game, what it sure is the use ‘brained’ methods to take financial advantages from online casinos.
It is amazing, you’ve got huge percentage with your application.
Nice stats! I guess you’ve to get a good ROI with your apps.
One question : with how many data did you train the neural network and with how many data did you test the neural network?
By the way, you can’t test with learning data otherwise all your results are wrong.
Make Money Online Poker Bot | CAREER-JACK
Металлопоиск и… нейросеть – CHEPA website