Above – An early research kit of a wearable brain-computer interface device, built by Facebook Reality Labs. The team has been testing the ability to decode single imagined words with non-invasive technologies, and they expect to get their first results from our wearable prototypes in the near term. Being able to decode even just a handful of imagined words — like “select” or “delete” — would provide entirely new ways of interacting with today’s VR systems and tomorrow’s AR glasses.
FRL (Facebook Reality Labs) is continuing to explore non-invasive BCI methods with other partners, including the Mallinckrodt Institute of Radiology at Washington University School of Medicine and APL at Johns Hopkins. They are starting with a system using near-infrared light.
Like other cells in your body, neurons consume oxygen when they’re active. So if we can detect shifts in oxygen levels within the brain, we can indirectly measure brain activity. Think of a pulse oximeter — the clip-like sensor with a glowing red light you’ve probably had attached to your index finger at the doctor’s office. Just as it’s able to measure the oxygen saturation level of your blood through your finger, we can also use near-infrared light to measure blood oxygenation in the brain from outside of the body in a safe, non-invasive way. This is similar to the signals measured today in functional magnetic resonance imaging (fMRI) — but using a portable, wearable device made from consumer-grade parts.
Facebook funded University of California San Francisco scientists. UCSF researchers asked patients to answer out loud a list of simple multiple-choice questions ordered randomly.
Decoding what patients with speech impairments are trying to say is improved by taking into account the full context in which they are trying to communicate.
The lateral surface of the human cortex contains neural populations that encode key representations of both perceived and produced speech Recent investigations of the underlying mechanisms of these speech representations have shown that acoustic and phonemic speech content can be decoded directly from neural activity in superior temporal gyrus (STG) and surrounding secondary auditory regions during listening. Similarly, activity in ventral sensorimotor cortex (vSMC) can be used to decode characteristics of produced speech based primarily on kinematic representations of the supralaryngeal articulators and the larynx for voicing and pitch17. A major challenge for these approaches is achieving high single-trial accuracy rates, which is essential for a clinically relevant implementation to aid individuals who are unable to communicate due to injury or neurodegenerative disorders.
Recently, speech decoding paradigms have been implemented in real-time applications, including the ability to map speech-evoked sensorimotor activations, generate neural encoding models of perceived phonemes, decode produced isolated phonemes, detect voice activity, and classify perceived sentences. These demonstrations are important steps toward the development of a functional neuroprosthesis for communication that decodes speech directly from recorded neural signals. However, to the best of our knowledge there have not been attempts to decode both perceived and produced speech from human participants in a real-time setting that resembles natural communication. Multimodal decoding of natural speech may have important practical implications for individuals who are unable to communicate due to stroke, neurodegenerative disease, or other causes. Despite advances in the development of assistive communication interfaces that restore some communicative capabilities to impaired patients via non-invasive scalp electroencephalography, invasive microelectrode recordings, electrocorticography (ECoG), and eye tracking methodologies, to date there is no speech prosthetic system that allows users to have interactions on the rapid timescale of human conversation.
Natural communication often occurs in dialogue, differentially engaging auditory and sensorimotor brain regions during listening and speaking. However, previous attempts to decode speech directly from the human brain typically consider listening or speaking tasks in isolation. Here, human participants listened to questions and responded aloud with answers while we used high-density electrocorticography (ECoG) recordings to detect when they heard or said an utterance and to then decode the utterance’s identity. Because certain answers were only plausible responses to certain questions, we could dynamically update the prior probabilities of each answer using the decoded question likelihoods as context. We decode produced and perceived utterances with accuracy rates as high as 61% and 76%, respectively (chance is 7% and 20%). Contextual integration of decoded question likelihoods significantly improves answer decoding. These results demonstrate real-time decoding of speech in an interactive, conversational setting, which has important implications for patients who are unable to communicate.
SOURCES- Facebook, Nature Communications
Written By Brian Wang, Nextbigfuture.com