Title: Design and Evaluation of Intelligent Reward Structures in Human Computation Games

Kristin Siu

Ph.D. Candidate

School of Interactive Computing

Georgia Institute of Technology

https://www.algorithmicallyanimated.com/

Date: Wednesday, June 2, 2021

Time: 1:00 pm to 3:00 pm (EDT)
Location: https://bluejeans.com/mriedl3

Committee:
Dr. Mark Riedl (advisor), School of Interactive Computing, Georgia Institute of Technology

Dr. Blair MacIntyre, School of Interactive Computing, Georgia Institute of Technology
Dr. Betsy DiSalvo, School of Interactive Computing, Georgia Institute of Technology

Dr. Brian Magerko, School of Literature, Media, and Communication, Georgia Institute of Technology

Dr. Seth Cooper, College of Computer and Information Science, Northeastern University

Abstract:

Despite the ubiquity of artificial intelligence, some problems and procedures— such as building commonsense knowledge understanding or generating creative works— have no or few effective algorithmic solutions, yet are considered straightforward for humans to solve. Human computation games (HCGs) are playful, game-based interfaces for tackling these problems through crowdsourcing. HCGs have been used to solve tasks that were and still are considered complex for computational algorithms such as image tagging, protein synthesis, 3D structure reconstruction, and creative artifact generation. However, despite these successes, HCGs have not seen broad adoption compared to other types of serious digital games. Among the many reasons for this lack of adoption is the reality that these games are typically not seen as engaging or compelling to play, as well as the fact that creating HCGs comes at a high development cost to task providers who are typically not game development experts. This thesis is a step towards building and establishing a more formalized design understanding of how to create HCGs that both provide a compelling player experience and complete the underlying task effectively.

In this thesis, I explore reward mechanics in HCGs. Reward mechanics are integral to HCGs due their associations with player motivation, compensation, and task validation. I first propose a framework for understanding HCG mechanics and advocate for an experimental methodology evaluating both player experience and task completion metrics to understand variations in HCG mechanics. I then use these tools to frame and design three experiments that explore small-scale variations of reward systems in HCGs: reward functions, reward distribution, and reward personalization. These studies demonstrate that even small variations in rewards (i.e., offering players the ability to choose the type of reward) may have significant positive effects on both player experience and task completion metrics. I also show that some variations (i.e., co-located, competitive reward scoring) may have both positive and negative tradeoffs across these metrics. Moreover, this work observes that existing, anecdotal design wisdom for HCGs may not always hold (i.e., allowing players to verbally collude actually predicts higher task solution accuracy). Altogether, this thesis demonstrates that certain aspects of reward systems in HCGs can be varied to improve the player experience without compromising task completion metrics, and builds more empirically-tested design knowledge for creating more engaging, effective HCGs.