AI brings object-level vision prosthetics closer to reality

Image credit: By Jeff Dahl – Own work by uploader, Based on the public domain document: [1], CC BY-SA 3.0, Link

By Stephanie Parker

This research from the NeuroAI Lab of Martin Schrimpf, part of EPFL’s Schools of Computer and Communication Sciences and Life Sciences, uses AI models to predict exactly where to stimulate the brain to evoke images of faces and specific objects in the users instead of simply evoking spots of light. The models developed at EPFL were used by Dutch researchers for live trials on sighted monkeys. The preliminary results, presented in April at the International Conference on Learning Representations, show very promising implications for vision in humans as well.

“The motivation for this project is that there are many people with visual deficits that are irreparable, in the sense that somewhere along the visual processing stream, starting with the retina, there is a deficit which cannot be repaired,” says Johannes Mehrer, a scientist in the NeuroAI lab who led the research. “One way of tackling this problem is to develop a visual prosthesis.”

There are multiple kinds of visual prosthetics including retinal, optical nerve, and cortical. Retinal prosthetics are placed somewhere on the retina, and optical nerve prosthetics are used when the retina is too damaged for an implant and the optical nerve can be stimulated instead. When neither the retina nor optical nerve can be implanted, cortical prosthetics are used. These bypass the retina and optical nerve entirely and work instead by stimulating the visual cortex, using electrodes to “draw” images onto it. However, thus far, this approach is limited in that it targets lower-level regions of the brain where it is only possible to project light flashes and simple shapes. There are also hardware constraints because multiple electrodes are needed to stimulate different areas at the same time, but only a certain number of electrodes can be used in one area.

“The images they can elicit, in this case simple symbols, are really limited in their complexity,” Mehrer explains. “At the moment, existing approaches to visual prostheses couldn’t elicit the percept of a more complex visual object such as a house or a car.”

Higher-level visual regions of the brain underlie the processing of more complex objects and could thus serve as a target for a new generation of visual prostheses allowing for eliciting images of faces, houses, and other objects. However, these higher-level regions are less accessible, because it is not known exactly where and how to stimulate them. This is where the AI model comes in.

“We had the idea to use an artificial neural network, in this case a specific type called a topographic neural network, to test various patterns of brain stimulations in these higher-level regions of the brain and simulate their outcomes,” Mehrer says. “We can then run all sorts of simulations using different combinations of different parameters that would otherwise take up a lot of experimental time and would cost a lot of money.”

The EPFL researchers, working entirely on computers, set up a model to select the best combination of images with the specific pattern of stimulation. Following their results, a team of resarchers in Amsterdam decided to test the model’s prediction on two of their monkeys who already had implants for other experiments not involving EPFL.

“Our model turned out to be quite efficient in predicting which stimulation pattern would yield a strong effect on the monkeys’ behavior with respect to visual object recognition,” says Martin Schrimpf, head of the NeuroAI Lab. “Our models can do the image selection, but the more crucial part is that given an image, it can tell us what the optimal stimulation pattern for a particular desired behavior is.”

What the researchers have been able to show so far with this work is that they can shape object perception, meaning that if a visual stimulus is presented, they can bias its representation in the brain. However, they cannot yet create object perception out of nothing. Stimulating the cortex while there is no visual stimulus presented would be their next step towards restoring meaningful sight to the blind.

“The monkey saw an image already, and then we were able to basically distort it to change the perception in somewhat predictable ways,” says Schrimpf. “The bigger goal will be to evoke a percept from scratch: to make someone see something meaningful even when their eyes aren’t delivering a usable image.”

This work showing that model-guided brain stimulation could lead to more advanced visual prosthetics could also be applied to hearing prosthetics. Through a grant from the Horton Health Foundation, Schrimpf and his team will next investigate if this kind of modeling works for auditory stimulation.

“Cochlear implants are great, but they are also not perfect in many ways, and they don’t really fully restore auditory processing,” says Schrimpf. “Our idea is to develop these kinds of topographic models that can predict what stimulation does to neural activity for auditory processing as well.”