Abstract:

We consider the problem of inferring probabilistic behavior by agents faced with decision
options m_1,m_2,...,m_n, in terms of hindsight utility values u_1,u_2,...,u_n and parameters Z
governing the aptitude of the agent.  In chess the options are the legal moves in a given
position, the utilities are values computed by strong chess programs, and the parameters
are fitted to the international chess Elo rating scale.  We show with large data that Bayesian
and maximum-likelihood methods are markedly inferior to simple frequentist methods at
this task.  We justify theoretically our contention that the former methods emphasize the
option that was actually chosen at each turn in the training sets in ways that fail to use
much of the information in the data.

The statistical model was developed with Guy Haworth (Univ. of Reading, UK) in papers at
AAAI 2011 and the 2011 Advances in Computer Games conference.  The talk will also show
how it is employed to compute "Intrinsic Ratings" based on quality of moves made rather than
the results of games, and to evaluate statistical allegations of players cheating with computer
programs during games.  Unlike many field studies of decision making the data sets have
been taken under real competition, and the talk will discuss attendant issues of inference
from large data, handling it, caveats in interpreting it, and the general practice of science.