A new artificial intelligence tool can infer what a person is looking at simply by analyzing their brain scans, recreating the image with striking accuracy. It can also work in reverse, predicting a person’s brain activity based on what they are seeing.

Michal Irani, who developed the tool with colleagues at the Weizmann Institute of Science in Rehovot, Israel, hopes it will reveal more about how the brain works and might one day help people with locked-in syndrome communicate, or allow scientists to recreate the content of dreams. Judy Illes, a neuroethics expert at the University of British Columbia, calls the work “magnificent”; other scientists, however, warn that a similar approach could expose people’s thoughts and mental images without their consent.

Peering into the brain with greater precision

For years, neuroscientists have tried to use functional magnetic resonance imaging (fMRI) to reconstruct what people see. Early attempts produced blurry images that were hard to interpret. fMRI tracks the flow of oxygenated blood through the brain, but in conventional scanners each “voxel” corresponds to roughly three cubic millimeters of tissue, containing tens of thousands of neurons.

Irani’s team used newer datasets collected with higher-resolution scanners, in which each voxel corresponds to about one cubic millimeter. Earlier models could already generate, say, an image of a banana a participant had seen, but with a different structure and position. The goal was to recreate the images far more faithfully.

A better decoder and a “universal” encoder

The model was trained on data from eight people who had each viewed around 9,000 images inside an fMRI scanner. The “brain decoder” has two components: one predicts an image’s structure, such as where colors are located, and the other predicts its content. These predictions guide a diffusion model that produces a much more accurate rendition.

To overcome the shortage of data, the researchers also built an encoder that predicts brain activity from an image, training the two tools together iteratively. As a result, about 70% of the training data comes from images never shown to a person in a scanner. The resulting encoder can adapt to a new individual with just one hour of data, compared with the roughly 40 hours required by other approaches—a significant advantage, given that an hour of scanning costs between $600 and $1,000.

The state of the art, and the ethical questions

The tool isn’t perfect: a cake was reconstructed as a stack of sandwiches, and a dog in a bathtub as a similarly colored goat. Still, in benchmark tests it clearly outperformed previous systems. Irani now plans to move on to video and audio and, eventually, to reconstruct thoughts, imagination, and dreams, with possible applications in studying the flashbacks associated with post-traumatic stress disorder.

Advances like this inevitably raise questions about mental privacy. Similar approaches are being explored with EEG, which can be recorded through electrode caps or even earbuds. According to Marcello Ienca of the Technical University of Munich, that would be a “game changer”: companies could extract information from users’ brains without consent, and some courts might accept reconstructions of mental images as evidence. Irani acknowledges the potential for misuse, but for now she prefers to “think only about good things.”