The film industry has explored a future in which characters interact with large, projected wall displays through speech, gaze, and gesture. Characters like Tony Stark from the Iron Man franchise or those in Minority Report can perform data exploration and analysis activities using various visualizations by non-haptic means.

Today’s systems for data visualization, which utilize desktop and laptop computers to interact via mouse-driven direct manipulation interfaces following the window-icon-menu-pointer (WIMP) paradigm, pale in comparison to the natural, fluid interactions presented in those futuristic film sequences.

Through a new grant provided by the National Science Foundation (NSF) Information Integration and Informatics (III) program, School of Interactive Computing Professor John Stasko aims to explore, design, develop, and evaluate post-WIMP interfaces for data visualization and data analytics.

The project, titled Creating Natural Data Visualization and Analysis Environments, has received three years of funding worth a total of $493,752.

To move beyond WIMP interfaces, Stasko said, new forms of natural user interfaces (NUIs) employing multimodal interactions such as speech, pen, touch, gestures, gaze, and head and body movements must be developed.

“While no one interaction modality may provide all desired capabilities, combinations of modalities – speech, gaze, and pen, for example – could provide a more natural, intuitive, and integrated interface experience,” Stasko said.

In this scenario, the system could assist individuals who know the information they want to extract from their data but not the specific commands or interface actions to take in a visualization system to produce the proper charts.

“If the objects to be acted upon are not clear from speech commands, then gaze, gesture, and touch can clarify a person’s intent,” Stasko said. “Furthermore, these input modalities may excel when a conventional mouse and keyboard are not available.”

Stasko is assisted on the project by Ph.D. student Arjun Srinivasan.