When I was working on the poker bot, my friends and family were fond of asking: How’s it doing?  Sure, it was probably just small talk, but it did raise an interesting question: how do you measure its success?

Net profit seems like an obvious choice, but that answer is close to meaningless unless you are very specific with the details and even if you are, it still probably won’t mean much… unless its just for small talk. For example, what period of time are you measuring? How many games did it play during that time? Once you’ve agreed on a time period and a way to count the results, how can you be sure that the results weren’t a product of luck and variance instead of skill?

Win rate seems like a decent choice, but you face the same questions as well as some new ones: do you measure $/game, big blinds/100 hands, $/hour, or something else? And once you calculate those, is that number good or not? What do you compare it to? Other bots? Other people? What’s your benchmark?

What about development time? If you spend 500 hours developing a bot that eventually makes $500, that means you made a dollar an hour… not exactly something to brag to your in-laws about. But if it made that $500 in two days, then you might be on to something.

Also, do you only count the games you know its competent at, or do you also include the shots you’ve had it take at higher limits?

You get the idea: it’s complicated.

With that in mind, I offer the following graph, which shows the poker bot’s profits at PokerStar’s $11+$0.50 No Limit Heads Up Sit-n-Go’s for all of Sept 2008, the last full month it played before PokerStars nixed my account. Midway though the month I also started mixing in a few $22+$1s to see how it would do, but I didn’t include those results here because there weren’t enough games for the results to mean anything.

For the curious non-poker geeks: “$11+$0.50” means that the buyin for the game was $11 and that the rake, which is what you pay PokerStars to play, was $0.50. So when you beat the other guy, you win his $11 minus the $0.50 rake for a total profit of +$10.50. When you lose, you’re out the rake and the buyin, or -$11.50.

Analysis like this is good in that it tells you what your results are, but it doesn’t tell you what led to those results. For example, a graph like this lets you say “Hey honey, I made $250 dollars last week” whereas with a thorough analysis you’d be able to say, if you were so inclined, something like “Hey honey, I widened my 20BB to 30BB three bet range to include low pocket pairs and Ace-x-suited hands and after 200 occurrences the bot is winning 2bb/100 more at that level than it was before.”

Now that dollar results are kind of out of the way, future posts should lean more towards that end of the spectrum.