Novel computer program creates "sound vision" for blinds

A man blind since birth is taking up a surprising new hobby: photography. His newfound passion is thanks to a system that turns images into sequences of sound. The technology not only gives “sight” to the blind, but also challenges the way neurologists think the brain is organized.

In 1992, Dutch engineer Peter Meijer created vOICe, an algorithm that converts simple grayscale images into musical soundscapes. (The capitalized middle letters sound out “Oh, I see!”). The system scans images from left to right, converting shapes in the image into sound as it sweeps, with higher positions in the image corresponding to higher sound frequencies. For instance, a diagonal line stretching upward from left to right becomes a series of ascending musical notes. While more complicated images, such as a person sitting on a lawn chair, at first seem like garbled noise, with enough training users can learn to “hear” everyday scenes.
In 2007, neuroscientist Amir Amedi and his colleagues at the Hebrew University of Jerusalem began training subjects who were born blind to use vOICe. Despite having no visual reference points, after just 70 hours of training, the individuals went from “hearing” simple dots and lines to “seeing” whole images such as faces and street corners composed of 4,500 pixels. (For comparison, Nintendo’s Mario was made up of just 192 pixels in his first video game appearance.) By attaching a head-mounted camera to a computer and headphones, the blind users were even able to navigate around a room by the sound cues alone. Every few steps the system snaps a photo and converts it into sound, giving the users their bearings as they traverse tables and trashcans. One patient even took up photography, using the head-mounted system to frame his snapshots.

The training program also devoted 10 hours to recognizing human silhouettes represented by sound. By the end of the training, participants could detect the exact posture of the person represented by the soundscape and replicate the pose, the team reported online Thursday in Current Biology.

When the researchers mapped the brain activity of the participants, they found something astonishing. The generally accepted model of the brain contains regions devoted to each sense, such as the sight-centric visual cortex. Researchers had long believed that if those regions aren’t used for their intended sense, they are repurposed for other uses; for example, the visual cortex of someone blind from birth could be used to help boost her hearing. But Amedi and his colleagues found that the area of the visual cortex responsible for recognizing body shapes in sighted people — called the extrastriate body area — lit up with activity in the study participants when they were interpreting the human silhouettes.