Brain-computer interface enables avatar speech and gestures for people with paralysis
· Medical Xpressby National Institutes of Health
edited by Gaby Clark, reviewed by Robert Egan
Gaby Clark
Scientific Editor
Meet our editorial team
Behind our editorial process
Robert Egan
Senior Editor
Meet our editorial team
Behind our editorial process Editors' notes
This article has been reviewed according to Science X's editorial process and policies. Editors have highlighted the following attributes while ensuring the content's credibility:
fact-checked
peer-reviewed publication
trusted source
proofread
The GIST Add as preferred source
A scientific team has shown that a new brain-computer interface (BCI) can allow people with vocal tract and bodily paralysis to convey both speech and upper-body gestures simultaneously. While BCIs have facilitated one or the other in the past, this new system is the first to enable both modes of communication at once, more closely replicating natural expression.
Scientists at the University of California, San Francisco, used machine learning to decipher the unique brain activity that underpins concurrent speech and physical gestures in three patients. Their BCI successfully translated the thoughts of two participants into commands that dictated the expressions of a full-body virtual avatar. The research is published in the journal Nature Neuroscience.
"Conversation is about much more than the words being spoken. It's a multilayered, dynamic process involving the whole motor cortex," said Dr. Edward Chang, a professor of neurological surgery at UCSF. "This proof-of-concept shows us it's possible for a BCI to restore some of this freedom and flexibility."
Limits of current communication aids
People with amyotrophic lateral sclerosis (ALS) or who have experienced brainstem strokes often become severely paralyzed, which can hinder or eliminate their verbal and nonverbal capabilities. Eye-tracking technologies, the current standard of care, enable text-to-speech for patients but are slow, offer limited forms of expression and can be physically exhausting to use.
Chang and his colleagues have been working to provide a more natural solution. They previously implanted a thin strip of sensors called an electrocorticography (ECoG) array onto the motor cortex of several patients. Computer models, called decoders, then translated their brain signals into computer commands that controlled a digital head and face.
Training for mixed expressions
For the new study, the authors deployed ECoG arrays in patients with varying levels of vocal tract and bodily paralysis, but this time aimed to permit upper-limb expressions by connecting participants to an avatar with a complete body. The researchers acquired data while participants attempted to verbalize specific phrases or perform common gestures such as a hand wave or thumbs-up sign, both separately and concurrently.
Verbal and nonverbal communication have each been facilitated by BCIs in the past, but attempting both at once seemingly diminishes speech capabilities. Scientists have suspected that the brain signals associated with simultaneous speech and gestures would simply be an aggregate of the two types of signals, but Chang and his co-authors learned that multimodal communication is more than the sum of its parts.
Despite some overlap, the picture painted by data from simultaneous expressions was very different from the kinds created by speech or gestures separately. They found that decoders were more successful at deciphering signals from mixed expressions if they had previously been trained on data acquired while participants performed simultaneous rather than isolated speech and gestures.
A step toward wireless use
With this strategy, researchers and participants proved that a BCI device can enable multifaceted, lifelike expressions.
"These promising results give me hope that in the future, patients with severe paralysis will be able to recapture the holistic nature of human communication," said Dr. Debara Tucci, director of NIH's National Institute on Deafness and Other Communication Disorders (NIDCD).
While the BCI used in this study entailed a wired system connecting implanted sensors to external processing units, Chang explained that his team will soon test a fully implantable, wireless version with better prospects for long-term application.
Publication details
Edward Chang, Simultaneous speech and gesture decoding for multimodal communication in paralysis, Nature Neuroscience (2026). DOI: 10.1038/s41593-026-02446-2. www.nature.com/articles/s41593-026-02446-2
Journal information: Nature Neuroscience
Key medical concepts
Clinical categories
NeurologyPhysiatry Provided by National Institutes of Health Who's behind this story?
Gaby Clark
MA in English, copy editor since 2021 with experience in higher education and health content. Dedicated to trustworthy science news. Full profile →
Robert Egan
Bachelor's in mathematical biology, Master's in creative writing. Well-traveled with unique perspectives on science and language. Full profile →
Citation: Brain-computer interface enables avatar speech and gestures for people with paralysis (2026, September 14) retrieved 15 September 2026 from https://medicalxpress.com/news/2026-09-brain-interface-enables-avatar-speech.html This document is subject to copyright. Apart from any fair dealing for the purpose of private study or research, no part may be reproduced without the written permission. The content is provided for information purposes only.