“What do you think of my artificial voice?” asks a woman on the computer screen, her green eyes widening slightly. The image is clearly computerized and the voice is hesitant, but it’s still a memorable moment. She is a digital avatar of a person who lost the ability to speak after suffering a stroke 18 years ago. Now, as part of an experiment involving a brain implant and Artificial Intelligence (AI) algorithms, the patient can communicate with a reproduction of her own voice and even express a limited set of facial expressions through her avatar.
Two articles published at the end of August in Nature and produced by two independent research teams show how quickly this area is advancing, although these prototypes are still far from being accessible to the public. Each study involved the participation of a woman who lost the ability to express herself clearly, one after a brain stem stroke and the other due to Amyotrophic Lateral Sclerosis (ALS), a progressive neurodegenerative disease.
Each participant had a different type of recording device implanted in their brain, and both were able to speak at a rate of about 60 to 70 words per minute. This is approximately half the speed of normal speech, but more than four times faster than has previously been reported. A team led by Edward Chang, a neurosurgeon at the University of California, San Francisco (USA), also captured the brain signals that control the small movements that make facial expressions possible, allowing them to create an avatar that represented the participant’s speech. of the study almost in real time.
The papers “are an example of really sophisticated and methodical science and engineering for the brain,” says Judy Illes, a neuroethicist at the University of British Columbia in Vancouver, Canada, who was not involved in either study. Illes especially enjoyed creating an expressive avatar. “Communication does not just involve the exchange of words between people. These are words and messages communicated through intonation, expression, accent, context”, he states. “I think it was creative and extremely well thought out to try to incorporate this component of individuality into fields like fundamental science, engineering and neurotechnology.”
Chang and his team have been working on this issue for more than a decade. In 2021, they demonstrated that they could record the brain activity of a person who had suffered a brainstem stroke and convert these signals into written words and sentences, albeit slowly. In the latest paper, the team used a larger implant (a device the size of a credit card), with twice the number of electrodes, to capture signals from the brain of another patient, named Ann, who lost the ability to speak after a spill almost two decades ago.
The implant does not record thoughts. Instead, it captures the electrical signals that control the muscular movements of the lips, tongue, jaw and vocal cords, all of which enable speech. For example, “if you make a P or B sound, this involves bringing your lips closer together. This, in turn, would activate a certain portion of the electrodes involved in controlling the lips,” says Alexander Silva, study author and graduate student in Chang’s lab. A connector, positioned on the patient’s scalp, allows staff to transfer these signals to a computer, where AI algorithms decode them and a language model helps provide self-correction capabilities to improve conversion accuracy. Using this technology, the team translated Ann’s brain activity into written words at a rate of 78 words per minute, using a vocabulary of 1,024 words, with a 23% error rate.
Chang’s group was also able to decode brain signals directly from spoken words, a first for any group. And the captured muscle signals allowed the participant to express, through the avatar, three different emotions (happy, sad and surprise) at three different levels of intensity. “Speech is not just a simple communication of words, but it is also part of who we are. Our voice and expressions are part of our identity,” says Chang. The study participant hopes to become a counselor. “It’s my big goal,” he told researchers. She believes that, by using this type of avatar, she could make her customers feel more comfortable. The team used a video recording of her wedding to replicate her voice, so the avatar even sounded like herself.
The second team, led by researchers from Stanford (USA), first published their results as a preprint in January. The researchers inserted four much smaller implants (each the size of an aspirin), capable of recording signals from nerve cells, into a participant with ALS named Pat Bennett, who trained the system by reading syllables, words and phrases over 25 sessions.
The researchers then tested the technology by having it read sentences that had not been used during training. When they were extracted from a set of 50 words, the error rate was about 9%. When the team expanded the vocabulary to 125,000 words, which covers much of the English language, the error rate rose to about 24%.
Speech using these interfaces is not perfect. It’s still slower than normal, and while a 23% or 24% error rate is much better than previous results, it’s still not ideal. In some cases, the system replicated sentences perfectly. In others, “How is your cold?” (How is your cold?) came out as “How is your old?” (How is your old man?).
But scientists are convinced they can do better. “The interesting thing is that as you add more electrodes the performance of the decoder continues to increase,” says Francis Willett, a Stanford neuroscientist and author of the paper. “If we can get more electrodes to decode even more neurons, we can be even more accurate.”
Current systems are not practical in everyday situations. Because they rely on wired connections and a computer system too large to handle the processing, the women cannot use the brain implants to communicate outside the context of the experiment. “There is still a lot of work to be done to transform this knowledge into something useful for people with unmet needs,” says Nick Ramsey, neuroscientist at the UMC Utrecht Brain Center, in Amsterdam (Netherlands), and author of an analysis published alongside the articles.
Illes also draws attention to the fact that each team reports results from a single individual, and they may be invalid for other people, even those with similar neurological conditions. “This validates the initial idea,” she says. “We know that brain injuries are very complex and highly variable. Generalization of results, even within the population of patients with stroke or ALS, is possible, but not certain.”
However, it creates the possibility of a technological solution for people who lose the ability to communicate. “What we did was prove that it is executable and that there is a path to accomplish this,” says Chang.
Being able to speak is crucial. The participant in Chang’s study often used a letter board to communicate. “My husband was tired of having to get up and decipher the board for me,” she told researchers. “We didn’t argue, because he didn’t give me the chance to argue back. As you can imagine, this frustrated me greatly!”
According to the medical manager of Rehabilitation at Hospital Israelita Albert Einstein, Milene Ferreira, neurological disorders account for the second biggest cause of disability in the world population.
“The loss of communication capacity is one of the main factors in reducing autonomy, productivity and satisfaction. Not being able to express what you think or want is described as much more distressing or limiting than motor difficulties by many patients. This study context not only sheds light on this type of deficit, but also paves the way for other experiments aimed at capturing and translating brain activity applied to rehabilitation and inclusion”, observes Milene.
The view of those who work directly with patient rehabilitation is that brain-machine interface research depends on the integration of several areas of knowledge. For the specialist, there is still a lot of progress needed to make applicable solutions available to people.
(source: MIT Technology Review)
