While the system currently works on neural patterns detected while someone is speaking aloud, experts say the system could eventually aid communication for patients who are unable to speak or type, such as those with locked in syndrome.
“We are not there yet but we think this could be the basis of a speech prosthesis,” said Dr Joseph Makin, co-author of the research from the University of California, San Francisco.
Writing in the journal Nature Neuroscience, Makin and colleagues reveal how they developed their system by recruiting four participants who had electrode arrays implanted in their brain to monitor epileptic seizures.
These participants were asked to read aloud from 50 set sentences multiple times, including “Tina Turner is a pop singer”, and “Those thieves stole 30 jewels”. The team tracked their neural activity while they were speaking.
This data was then fed into a machine-learning algorithm – a type of artificial intelligence system – that converted the brain activity data for each spoken sentence into an abstract string of numbers.
To make sure the string of numbers related only to aspects of speech, the system compared sounds predicted from small chunks of the brain activity data with actual recorded audio. The string of numbers was then fed into a second part of the system which converted it into a sequence of words.
At first the system spat out nonsense sentences. But as the system compared each sequence of words with the sentences that were actually read aloud it improved, learning how the string of numbers related to words, and which words tend to follow each other.
The team then tested the system, generating written text just from brain activity during speech.
The system was not perfect. Among its mistakes, “Those musicians harmonise marvellously” was decoded as “The spinach was a famous singer”, and “A roll of wire lay near the wall” became “Will robin wear a yellow lily”.
However the team found the accuracy of the new system was far higher than previous approaches. While accuracy varied from person to person, for one participant just 3% of each sentence on average needed correcting – higher than the word error rate of 5% for professional human transcribers. But, the team stress, unlike the latter, the algorithm only handles a small number of sentences.
“If you try to go outside the [50 sentences used] the decoding gets much worse,” said Makin, adding that the system is likely relying on a combination of learning particular sentences, identifying words from brain activity, and recognising general patterns in English.
The team also found that training the algorithm on one participant’s data meant less training data was needed from the final user – something that could make training less onerous for patients.
Dr Christian Herff an expert in the field from Maastricht University who was not involved in the study, said the research is exciting because the system used less than 40 minutes of training data for each participant, and a limited collection of sentences, rather than millions of hours typically needed.
“By doing so they achieve levels of accuracy that haven’t been achieved so far,” he said.
However he noted the system is not yet usable in severely disabled patients, as it relies on the brain activity recorded from people speaking a sentence out loud.
“Of course this is fantastic research but those people could just use ‘Ok Google’ as well,” he said. “This is not translation of thought [but of brain activity involved in speech].”
Herff added that people should not worry about others reading their thoughts just yet – the brain electrodes must be implanted, while imagined speech is very different to inner voice.
But Dr Mahnaz Arvaneh, an expert in brain machine interfaces at Sheffield University, said it was important to consider ethical issues now.
“We [are still] very, very far away from the point that machines can read our minds,” she said. “But it doesn’t mean that we should not think about it and we should not plan about it.”
More about: #AI
