“This was particularly exciting to figure out how we could broaden accessibility to data sets because just like so much other scientific data, it’s out there and available, but when it’s presented to the public, it’s usually in visual form,” she said. “There are many open questions about how to do this well within a museum with complex scientific data. We’re moving the needle on that, but there’s a long way to go.”

Right Combination

Amspoker and Roberts created three different versions of the prototype. One was sound-only, one was texture-only, and the other was a combination of sound and texture.

“We expected the multimodal version would work best,” Amspoker said. “We found people used sound and texture in different ways when interacting with it. In cases where people relied on texture, it was still difficult to tell when they crossed the barrier from one texture to another. Sound was very useful in that case.”

Amspoker said computer vision and an app she designed allow the technology to be deployed on any surface, whether a mobile device, a wooden board, or even a classroom floor. A camera set up above the display tracks the user’s hand movements.

“It figures out where you are on the board, and then our code uses the location of your finger to decide what sound should play from the computer,” she said. “What’s nice about our system is it only needs a computer and a webcam, and you can use whatever materials you have on hand for the map.”

Building on a Legacy

Roberts said she is building on the work of a previous NSF-funded collaboration with Dr. Amy Bower, a senior scientist at the Woods Hole Oceanographic Institute in Massachusetts who is blind.

Bower lost her vision in graduate school, but because of her lifelong interest in oceanography, she set out to create ways to learn about ocean data through sound. 

In 2021, she launched the Accessible Oceans project through the National Science Foundation’s Advancing Informal STEM Learning program. The interdisciplinary team, including Roberts and collaborators Leslie Smith of Your Ocean Consulting and Jon Bellona of the University of Oregon, created auditory displays of sonified data for museums.

In 2023, the team published an article in Oceanography, the official magazine of the Oeanography Society.

“Informal learning environments are increasingly recognizing the importance of employing multiple modalities to engage all learners and are leveraging sound to enhance visitor experience,” the authors wrote.

“While sonic additions of music, soundscapes, and field recordings add qualitative value, there is a need to explore the potential of sound to facilitate engagement with quantitative information. Data sonification is a promising avenue for increasing accessibility to data within the museum context.”