Watching a film reveals how the brain balances eyes and ears

by NYU Tandon School of Engineering

edited by Gaby Clark, reviewed by Robert Egan

Gaby Clark

Scientific Editor

Meet our editorial team
Behind our editorial process

Robert Egan

Associate Editor

Meet our editorial team
Behind our editorial process Editors' notes

This article has been reviewed according to Science X's editorial process and policies. Editors have highlighted the following attributes while ensuring the content's credibility:

fact-checked

peer-reviewed publication

trusted source

proofread

The GIST Add as preferred source

For most of us, watching a movie feels effortless. We follow dialogue, read facial expressions, notice music cues and shifting scenery, and somehow fuse it all into a coherent story. But beneath that smooth experience, the brain is constantly deciding which sensory stream matters most in each moment.

A new study published in the journal Nature Communications suggests that the frontal cortex, a region associated with planning and higher cognition, may act as a kind of traffic controller for this process—dynamically shifting attention between what we hear and what we see as a story unfolds.

Inside the movie experiment

To investigate, neuroscientists recorded brain activity directly from 19 epilepsy patients who had temporarily implanted electrodes for clinical monitoring. While in the hospital, participants watched a 12-minute multilingual short film containing scenes in English, Greek, German and French. Some foreign-language scenes included English subtitles, creating a natural test of how the brain handles changing audiovisual demands. Because the electrodes sat on or inside the brain, the researchers could track neural responses with millisecond precision, far faster than MRI scans allow.

They found that the frontal cortex was not processing all sensory information equally. Instead, it showed a striking internal division. Ventral, or lower, frontal regions responded more strongly to auditory information, while dorsal, or upper, frontal regions were more tuned to visual input.

"This suggests the frontal cortex has an organized map for handling different kinds of information during real-world experiences," said first author Faxin Zhou, a Ph.D. candidate in the NYU Tandon Biomedical Engineering Department. "It is not just a general control center, it appears to separate sound and sight in a structured way."

Language shifts the balance

The pattern became even more interesting when the language changed. During English-language scenes, when listeners could understand speech directly, frontal brain areas leaned more heavily on auditory processing. But during scenes in unfamiliar languages, activity shifted toward visual regions, suggesting viewers relied more on facial expressions, gestures and subtitles to follow the plot.

To confirm that interpretation, the team recruited online volunteers to rate short clips from the film. Participants judged which moments were most important to understanding the story and whether audio or visual cues were more useful in each scene. Those ratings closely matched the neural data: Spoken English favored sound, while foreign-language scenes favored visual cues. In other words, the brain appears to reweight its sensory priorities on the fly.

"When comprehension through speech becomes harder, the brain flexibly reallocates resources toward visual signals," said senior author Adeen Flinker, associate professor of biomedical engineering at NYU Tandon and neurology at NYU Grossman School of Medicine. "That adaptability may be essential for navigating everyday environments filled with competing information."

A more active role

The findings help illuminate a longstanding question in neuroscience: how the brain merges multiple senses in realistic settings. Much prior research has relied on simplified laboratory tasks. Movies, by contrast, more closely resemble real life, where sensory cues arrive continuously and unpredictably.

The study also hints that the frontal cortex may do more than merge information after the fact. It may actively decide which stream—sound or sight—deserves priority before conscious understanding emerges.

Limits and possible applications

That insight could have practical implications. Better understanding of how the brain reallocates sensory attention may help researchers design therapies for people with language disorders, autism, attention deficits or hearing loss. It could also inspire more adaptive artificial intelligence systems that shift between audio and visual inputs depending on context.

The work has limitations. Because the participants were hospital patients with epilepsy, they may not perfectly represent the general population. Electrode placement was determined by medical need, not experimental design, leaving some brain areas less sampled than others. Still, the precision of direct neural recording offers a rare glimpse into how the living human brain manages everyday perception.

Publication details

Faxin Zhou et al, Frontal cortex organization supporting audiovisual processing during naturalistic viewing, Nature Communications (2026). DOI: 10.1038/s41467-026-73947-8

Journal information: Nature Communications

Key medical concepts

Frontal LobeElectrocorticographyLanguage DisordersHearing LossEpilepsy

Clinical categories

Neurology Provided by NYU Tandon School of Engineering Who's behind this story?

Gaby Clark

MA in English, copy editor since 2021 with experience in higher education and health content. Dedicated to trustworthy science news. Full profile →

Robert Egan

Bachelor's in mathematical biology, Master's in creative writing. Well-traveled with unique perspectives on science and language. Full profile →

Citation: Watching a film reveals how the brain balances eyes and ears (2026, June 22) retrieved 22 June 2026 from https://medicalxpress.com/news/2026-06-reveals-brain-eyes-ears.html This document is subject to copyright. Apart from any fair dealing for the purpose of private study or research, no part may be reproduced without the written permission. The content is provided for information purposes only.