Scientists reconstruct brain images into video

Jack Gallant and Shinji Nishimoto in Gallant's lab, were able to "read the mind" by deciphering and rebuilding the human visual experience.

Researchers at the University of California, Berkeley have developed an algorithm that can be applied to functional magnetic resonance imaging (fMRI) to reconstruct YouTube videos from viewers' brain activity, World Bulletin has reported.

Jack Gallant, study leader and a UC Berkeley neuroscientist, and Shinji Nishimoto, a post-doctoral researcher in Gallant's lab, were able to "read the mind" by deciphering and rebuilding the human visual experience.

However, they were also careful to point out that any technology that allows us to read each other's thoughts and intentions are at least decades away.

The study, which appears in Current Biology this week, marks the first time that anyone has used brain imaging to determine what moving images a person is seeing. It could help researchers model the human visual system on a computer, and it raises the tantalizing prospect of one day being able to use the model to reconstruct other types of dynamic imagery, such as dreams and memories.

Jack Gallant, a UC Berkeley neuroscientist and coauthor of the study, described the results as a "major leap toward reconstructing internal imagery," adding that "we are opening a window into the movies in our minds," the website reported.

"Once we had this model built, we could read brain activity for that subject and run it backwards through the model to try to uncover what the viewer saw," added Gallant.

"If you can decode movies people saw, you might be able to decode things in the brain that are movie-like but have no real-world analog, like dreams," Gallant said.

"The brain isn't just one big blob of tissue. It actually consists of dozens, even hundreds of modules, each of which does a different thing," noting, Gallant hope to look at more visual modules, and try to build models for every single part of visual system.

Whether the technology could also be used to watch people's dreams or memories -- even intentions -- depends on how close those abstract visual experiences are to the real thing. "We simply don't know at this point. But it's our next line of research," noted Gallant.

Shinji Nishimoto, a neuroscientist in Gallant's lab and the study's lead author, said the results shed light on how the brain understands and processes visual experiences.

"We need to know how the brain works in naturalistic conditions," Nishimoto said in a statement. "For that, we need to first understand how the brain works while we are watching movies."

"Our natural visual experience is like watching a movie," explained Shinji Nishimoto. "In order for this technology to have wide applicability, we must understand how the brain processes these dynamic visual experiences," he added.

Ultimately, Nishimoto said, scientists need to understand how the brain processes dynamic visual events that we experience in everyday life.