Serious Gaming: Entertainment Technology Yields Results for Research Ranging from Defense to K-12 Education
Written by Rick Robinson
In a laboratory at the Georgia Institute of Technology, researchers gaze intently into a line of large flat-screen monitors. Using hand-held devices and famous-name gaming software, they guide on-screen vehicles through winding streets and around or over obstacles. Groans can be heard when a vehicle doesn’t make the grade.
No, it’s not break time in the lab. The gaming activity here is serious, aimed at investigating ways in which a robot might move through complex environments. Its ultimate purpose is to provide the U.S. military and other government agencies with tiny autonomous devices that could carry out combat or disaster-relief missions.
The term serious gaming might seem to be an oxymoron, like “static electrons.” But in today’s pragmatic research world, investigators from numerous Georgia Tech units are appropriating technologies, practices and even equipment from both digital and real-world games. Then they’re applying those gaming techniques to defense, industry, education, health care and more.
Definitions of serious games and gaming vary. Still, most definitions cite the use of gaming technologies for purposes other than entertainment.
On-screen games such as Pong go back to the 1970s, but during the past 20 years the computer- and video-game world has expanded very rapidly. This digital revolution has produced powerful game “engines” – the basic code underlying a digital game – that are now widely available to aspiring game developers.
The result is that today’s game technologies are often highly user-configurable, a process called “modding” (short for modification); that adaptability helps make them useful for research purposes. Even gaming hardware is being modified for use in applications that its designers probably never envisioned.
This article provides a look at a variety of Georgia Tech research efforts related to serious games.
Using Games for Robotics Research
At the Georgia Tech Research Institute (GTRI), a research team is using game-engine technology to support an ambitious program to develop tiny mobile robots that are both intelligent and interactive.
The overall effort is called the Micro Autonomous Systems and Technology (MAST) Collaborative Technology Alliance Program. It’s hoped that this five-year effort will result in rolling, hopping and even flying devices that could aid the military and other agencies in combat, disaster relief and other tasks.
The MAST research program, which includes Georgia Tech, 13 other universities and BAE Systems Inc., is sponsored by the U.S. Army Research Laboratory. GTRI and the College of Computing are among several Georgia Tech units involved in the program.
To date, no truly autonomous robots actually exist, much less tiny ones that can move cooperatively through unpredictable environments. To gain insight into the many challenges involved in such technology, researchers are turning to game-development techniques, said GTRI principal research engineer Lora Weiss.
“To design micro-autonomous systems, we first need to explore in a virtual way how they might behave in the real world and interact with one other,” she said. “And a good way to start exploring them is with game engines, because you can examine robotic systems using the synthetic entities found in many game worlds.”
By combining a widely available computer-game engine with open-source software called USARSim – short for urban search and rescue simulation – the GTRI engineers can simulate many challenges that robotic hardware might encounter. Modifying game parameters to suit their purposes, engineers can rapidly construct a three-dimensional world – complete with reasonably accurate 3-D physics – to test a variety of concepts.
“What we’re trying to do here is look at high level – but not high-fidelity – interactions of dynamic systems,” Weiss said. “We’re using this approach on several robotics projects.”
High-level analysis, she explains, focuses only on behavior. For example, can a robot with certain attributes move successfully across an intersection, or might a car hit it on the way?
At this level, the high-fidelity physical details of robot motion or car damage are not vital. What’s most important is understanding whether the robot can apply reason to the situation and then activate an avoidance behavior.
“Game engines and virtual worlds allow exploring these interactions in ways that, until we understand the issues, are much safer than building systems and identifying problems the hard way,” Weiss said.
However, once behavioral basics have been worked out, researchers will likely leave the game world and move to more high-fidelity simulations. Such simulations offer greater accuracy and rigor that would help investigators understand the complex physics necessary to design and build a robot prototype.
Those future hardware prototypes will have their functionality evaluated by computer-based test equipment, a process known as hardware-in-the-loop. Once a prototype seems to be working properly, final steps will involve rigorous field testing.
Getting small robots to operate collaboratively with humans will be another hurdle, Weiss acknowledges. That’s a problem being addressed not only by Weiss, but by researchers in other Georgia Tech units, including College of Computing professors Charles Isbell and Mark Riedl (see “Applying Games to Complex Environments” in this article).
“High-level modeling of humans collaborating with groups of robots will be particularly important as robots become more independent and less remote-controlled,” Weiss said. “If robots are to exist with people and operate in shared spaces, then they must cooperate.”
Pursuing Industrial Research with Game Hardware
Digital gaming equipment has become highly sophisticated – so much so that some investigators are using off-the-shelf game accessories for serious research and applications.
At GTRI, a team of engineers is developing a research device capable of analyzing physical stresses on the arms of workers in poultry plants. Critical to their approach is a Nintendo Wii game-console remote controller, known popularly as the Wiimote. This inexpensive controller’s capabilities include advanced motion-sensing.
“Poultry workers typically stand and make the same cuts eight hours a day, and they have a risk of developing repetitive-stress syndrome,” said GTRI research scientist Clayton J. Hutto. “There’s a real need to study the ergonomics of poultry work to make it less physically stressful.”
Existing motion-capture devices cost thousands of dollars, Hutto explained. In the wet environment of a poultry plant, their expensive electronics can be damaged quickly.
But motion capture based on the $30 Wiimote would be affordable even if the capture devices didn’t last long. Moreover, at such reduced cost researchers could outfit multiple workers with capture devices, greatly increasing the amount of motion data gathered.
GTRI’s work with the Wiimote goes back to 2006, the year the device was introduced. Intrigued by the Wiimote’s capabilities, GTRI research engineer Donnie Smith developed the open-source CWiid library (pronounced seaweed) as an interface to the Wiimote’s low-level message format. CWiid, programmed in the C language with bindings to an open-source language called Python, allows a Wiimote to work in a variety of platforms and uses beyond traditional games.
In 2007, supported by research scientist Josh Davis, Smith worked with fellow research scientist Nick Bollweg to develop a Wiimote-based head-tracking system. This system, pyWiiVR, consists of a Wiimote, a computer, wireless headphones and goggles that have been modified with inexpensive infrared sensors.
Connected to a computer via Bluetooth, the system provides immersive 3-D visualization and audio that changes realistically with a user’s head movements. Such technology could have several military applications including training and mission preparation, the researchers said.
Collaborating with Jonathan Holmes, a researcher with GTRI’s Ergonomic Work Assessment System program, and Jessica Pater of GTRI’s Office of Policy Analysis and Research, Hutto and his team have developed a Wiimote-enabled device that mounts on a poultry worker’s lower arm. A Wiimote detects the location of four infrared LEDs and then transmits data gathered to a computer. Software transforms this information into arm and wrist angle data, which can be visualized and stored for analysis.
“So far, we’re on track, and it’s working very well,” said Hutto. “We’re also working on a feedback capability that instantly alerts a worker when he or she exceeds the recommended physical limits. If we can change worker behavior during the performance of an actual task, then we’re really onto something.”
Creating Environments for Training and Therapy
Game developers work hard to make electronic games more engaging. In fast-moving shooting games, computer-generated adversaries increasingly behave with human-like cunning. They can even take cues from human players’ gaming styles to make the action more competitive.
In other digital amusements, such as role-playing games, programmers use artificial intelligence (AI) techniques to help create “drama managers.” Working either in the background or as actual game characters, drama managers monitor player behavior and can quickly alter events to enhance a player’s experience.
Mark Riedl, an assistant professor in the Georgia Tech School of Interactive Computing, is using the drama-manager approach to develop simulations for certain types of military training. For this application, he has developed the Automated Story Director, software that uses a drama manager as a kind of coach.
For example, he has used simulations created by the Automated Story Director to develop cultural role-playing scenarios for soldiers who may soon experience similar situations on real foreign streets. The work was sponsored by the U.S. Army Research, Development and Engineering Command.
“We create virtual characters with certain cultural qualities, and the user’s goal is to pick up on what is happening in this foreign environment and to respond accordingly,” Riedl said. “The virtual coach watches what you’re doing, and if you’re not picking up on things, it might, for example, send a character to talk to you to make it more obvious.”
Riedl crafted this simulation using the software engine of a well-known shooting game. When he was finished with modifications, the carnage was gone; in its place, a role-playing game filled with edgy socio-cultural exchanges had emerged.
Riedl – collaborating with School of Interactive Computing colleagues associate professorCharles Isbell and associate professor Ashwin Ram – is also working on AI-enhanced, scenario-generation software. This kind of software can model the learner and then automatically craft a customized interactive experience, catering to the learner’s particular needs and abilities.
Using such scenario-generation software, learners enter an interactive, multiplayer virtual world. The software assigns roles to players, coaches them on how to play their roles – and can change the scenario on the fly depending on how things go.
“This technology could be useful in almost any area in which you might want to use a game or a simulated environment to educate,” Riedl said.
In another education-related effort called Refl-ex, Riedl is developing software to aid children with autism. He is collaborating on Refl-ex with Rosa Arriaga, a senior research scientist in the Georgia Tech School of Interactive Computing, and L. Juane Heflin, an associate professor in the Department of Educational Psychology and Special Education at Georgia State University.
Refl-ex uses a game-like environment to repeat and reinforce interactions between the child and a therapist or teacher. Riedl explains that game experiences tend to be a hit with autistic youngsters, who sometimes prefer interacting with computers rather than humans.
“We have people with autism who go to therapy several times a week, and yet it’s not enough,” he said. “If they can play this game at home, it could add five or 10 hours per week of valuable interaction for them.”
Applying Games to Complex Environments
Charles Isbell sees serious gaming as a two-way street.
An associate professor in Georgia Tech’s School of Interactive Computing and an associate dean in the College of Computing, Isbell is a specialist in statistical machine learning – software that allows computers to learn and change based on incoming data.
Isbell’s research goals include using gaming elements to improve machine-learning technology. He’s also doing the reverse – employing machine-learning approaches to improve serious-gaming applications.
“I’m very interested in large, complex problems where communication is difficult, your goals change, and you need to work with other people or even other agents such as robots,” he said. “Serious gaming has much to offer here, because these kinds of problems occur in a variety of social and other games.”
Isbell’s research includes three areas related to gaming:
• Finding techniques to influence how people behave, especially in groups.
In one recently initiated project, Isbell and his team are investigating ways to influence human subjects to adopt goals without obvious coercion – what Isbell calls “computational models of influence.” This effort is related to Lora Weiss’s work with the robotics program (see “Using Games for Robotics Research”in this article).
Entertainment games, Isbell notes, typically limit players to a specific area until they’ve accomplished a given task. He wants players to pursue tasks on their own, and have a sense that they’re acting of their own free will.
• Developing tools to let non-programmers quickly build game-like environments for training simulations, modeling or other purposes.
Working with School of Interactive Computing colleague Mark Riedl and others, Isbell is building tools to allow rapid development of models or game-like programs by non-programmers. The team is developing a language called A²BL, which is an adaptation of a well-known programming language called A Behavior Language, or ABL.
“With A²BL, you as a user can describe in simple terms what you’re trying to accomplish, and with that information we can build a system for you,” Isbell said.
• Using gaming concepts to improve tools for statistical machine learning.
Isbell and his team are using gaming concepts such as narrative to help re-envision approaches to machine learning. He sees a strong connection between mathematical sequences and the process of narrative.
Traditionally, machine learning focuses on immediate tasks rather than on sequences. In other words, it doesn’t matter where you’ve been, it only matters where you are.
But narrative can help make machine learning more flexible, Isbell explains. “With a story, it matters very much where you’ve been and those of us in machine learning have something to learn from that.”
Adapting Gaming Environments for the Classroom
Can the gaming worlds that keep kids anchored to computers for hours also help them in the classroom?
Jessica Pater, a GTRI research associate, has been investigating the use of gaming in education for several years. She believes that digital gaming technology can be a serious teaching tool.
“In K-12 education, there has been hesitation about using gaming in classrooms,” she said. “But now, quite a number of schools are starting to use immersive environments to create experiences that kids otherwise wouldn’t have.”
One example: game-like environments that let students manipulate molecules in real time. This kind of technology, limited to supercomputers only a few years ago, can allow youngsters to look at the actual bonds between different atoms, offering them a better understanding of how things work at the nanoscale level.
Also useful, Pater said, are immersive online environments such as Second Life and ActiveWorlds, also known as massively multiplayer online environments. These applications support development of simulations that can dramatize scientific subjects for middle school and high school students.
“Historically, schools have lacked the bandwidth to support these applications,” Pater said. “As the price of bandwidth continues to fall, more of these opportunities are starting to take hold in schools.”
Working with Georgia’s DeKalb County schools, Pater and colleagues are developing a learning environment within the Teen Second Life application. Using a Second Life island – a secure area only students can access – the GTRI-DeKalb team is creating a project called “Small Fry To Go.”
Based on a real-world DeKalb project where students raise trout at school-based hatcheries, Small Fry To Go extends that trout-growing venture into an immersive virtual world. In this world, students can investigate the underwater environment of trout fry (hatchlings) and make critical decisions about hatchery conditions.
“The intent is to teach environmental responsibility and show the impact of our decisions on the environment,” Pater said.
One downside to using immersive spaces for online education is the unevenness of funding for classroom technology, she said. Less-affluent school districts may lack the money to install the required equipment, to buy space in online environments or to maintain the immersive domain.
Other Georgia Tech units are also investigating the educational potential of online environments. The Center for Education Integrating Science, Mathematics, and Computing(CEISMC) is sharing in a new five-year, $3 million NASA grant to Georgia Tech aimed at supporting science, technology, engineering and math (STEM) education in K-12 schools. Mike Ryan, CEISMC’s program director, will lead a course that uses Second Life to help develop online environments.
“Being online doesn’t replace the curriculum – it augments it in a way that gets kids excited,” Pater said. “It’s basically an immersive space where teachers can go and create rich content for their kids.”
Weighing the Seriousness of Games
What makes one game serious and another just entertainment?
Ian Bogost believes that distinction is in the eye of the beholder. He suggests any game or environment that affects people’s lives should be viewed as serious, regardless of its original intent.
Bogost’s research often focuses on the cultural roles played by the fast-growing gaming field. He’s interested in mainstream commercial video games, but also in games that go beyond entertainment to address politics, advertising, learning or art.
“I’m interested in upsetting the solid line between seriousness and entertainment,” said Bogost, an associate professor in the Georgia Tech School of Literature, Communication and Culture. “Can you really label a game as pure commercial entertainment if, in fact, it has tremendous influence in the cultural sense?”
He points to SimCity, a popular game that lets users design urban areas from scratch. This hit game, he said, was not conceived as a serious tool for urban planning.
“But it has clearly created a certain interest in urban planning as a concept,” he argued. “And that has probably been responsible for inspiring an entire generation of potential urban planners – or at least aware citizens who attend their local city council meetings.”
Second Life and other online environments are increasingly used formally in education, Bogost observes. But, he asks, what about more casual uses? Couldn’t such environments also be viewed as serious if users find friendships and a sense of satisfaction online?
Even real-world games involve serious elements, Bogost notes. Basketball was devised as a way to keep poor children amused and out of trouble. Football is often touted for its character-building influence.
And, he points out, the issue of serious games is on the minds of many in the game-development world. For instance, the annual Game Developers Conference has for six years included a sub-conference that focuses squarely on serious-gaming issues.
Bogost’s 2007 book, Persuasive Games: The Expressive Power of Video Games, looks at the capacity of both commercial and other games to act as an influential medium. In particular, the book examines three areas where video-game persuasion shows potential: politics, advertising and education.
Bogost has published a large number of other books, articles and games. In 2007, The New York Times took the novel step of publishing a number of Bogost-developed games online.
One example is a game dealing with the failed McCain-Kennedy immigration bill of 2005. This game, playable online, offered a fictional competition between immigration agents. It conveyed details about this complex piece of legislation and made critical points about the nature of immigration politics.
Bogost’s most recent book, Newsgames: Journalism at Play, written with students Simon Ferrari and Bobby Schweizer, examines the potential role of video games in journalism. His current research also includes a project sponsored by the Knight Foundation that examines the developing relationship between journalism and gaming.
“News is changing, and the media are going to have different roles to play,” he said. “I don’t believe for a minute that reading will go away, but I think there will be new roles for tools such as video games.”
Bogost is working on a book that focuses on the diverse uses of digital gaming.
“Today, video gaming is a hot medium, though it’s been around for 30 years already,” he said. “It could turn out to be more or less a child’s toy or a fad, or it could achieve the mainstream acceptance and influence of television and movies. What we’re going to do with gaming is an open question.”
Merging Game Worlds with the Real World
Serious games don’t have to be solemn. Some researchers are investigating the potential for online games to be constructive and social and also appealing to a broad range of people.
Celia Pearce, an assistant professor in the Georgia Tech School of Literature, Communication and Culture, is interested in games that push the boundaries of online play. Working with her Emergent Game Group, she develops large-scale multiplayer game worlds that offer a viewpoint on the real world.
“I’m not sure I like the term ‘serious gaming’ – it could imply that a game is belaboring a point for is didactic,” she said. “Other terms that might work better are ‘activist games’ or ‘games for change.’”
Among the projects that Pearce and her team are working on is “Mermaids,” an experimental multiplayer game set underwater among the ruins of an extinct mermaid culture. Players must rebuild this lost culture while trying to avoid their ancestors’ fatal mistakes, which are unknown.
“This isn’t a blatantly activist game, but it has a very green subtext,” Pearce said. “Teachers have told us it could be a good way to reach about environmentalism.”
Project Passage is a suite of historical digital and board games. The digital game of the group, called “Five Boroughs,” takes place in the New York City of 1928, where players assume the roles of new immigrants.
Unlike many commercial games, where players lock themselves into a fully fledged character from the start, Project Passage players must familiarize themselves with their new world before they make important choices, Pearce said. Moreover, they even get opportunities to back out of those decisions if they want.
“If you look at the narrative of many games, everything is oversimplified – you’re either good or bad, a criminal or a cop,” she said. “Our game introduces nuances and ambiguities that often don’t get included in games, and are more similar to how the real world operates.”
These experimental games are developed under Pearce’s direction by students who get credit for each project studio they participate in.
“It’s kind of a market-driven program,” Pearce said. “We announce each semester what projects we’re doing, and the students work on the projects they’re interested in.”
Pearce – author of The Interactive Book (MacMillan 1997) and Communities of Play: Emergent Cultures in Multiplayer Games and Virtual Worlds (MIT 2009) – has also created games that blend the digital world with real-world activity.
In one game created by students in her group, participants are contacted by an international cast of characters from the future who report on the damage caused by current environmental practices. Players help one another complete real-world missions aimed at cleaning up today’s biosphere.
Pearce, who is also involved in independent game development, laments that the commercial gaming industry generally isn’t interested in funding projects that push the envelope.
“But who knows,” she adds. “If some of the more experimental games found a substantial following, that could change.”
Empowering Players to Create Their Own Games
Atlanta-based Kaneva, which has created a unique online environment, wants participants to generate their own 3-D casual games and other content for the Kaneva community.
Kaneva was founded by Georgia Tech alumnus Chris W. Klaus, who sold his Internet Security Systems company to IBM for $1.3 billion in 2006. Kaneva is definitely MMO – massively multiplayer and online – but it views itself as more of an environment, an enabling tool for its users.
“Kaneva is Latin for canvas, and our mission has always been to provide a digital canvas for our online community with 3-D game graphics and video technology,” Klaus said. “Now we’re entering into the phase of empowering these environments with gaming mechanics, so that ultimately our community will create its own 3-D games, serious and otherwise.”
Klaus said he’s already seeing users exploit the Kaneva space to achieve their own goals. When singer Michael Jackson died, site members immediately used the website’s tools to create a variety of 3-D graphical memorial spaces.
And Klaus points to the more than 20,000 churches created by Kaneva users as another means of self-expression.
“If you go into one of these online churches, you will see immediately that this is definitely not a game,” he said.
One area where Klaus particularly expects game-like environments to develop involves Kaneva’s home decorator application. The website lets users upload their own photos and other graphics and then use Kaneva’s 3-D tools to produce a multitude of design variations.
“If you then add some game mechanics such as points, prizes and levels,” Klaus said, “you’re well on your way to creating a serious online game that many users would also find fun and satisfying.”
This article was originally published in the Spring 2010 issue of Georgia Tech’s Research Horizons magazine.