Mapping Acoustics to Articulatory Gestures in Dutch: Relating Speech Gestures, Acoustics and Neural Data. 2022

Paolo Favero, and Julia Berezutskaya, and Nick F Ramsey, and Aleksei Nazarov, and Zachary V Freudenburg

Completely locked-in patients suffer from paralysis affecting every muscle in their body, reducing their communication means to brain-computer interfaces (BCIs). State-of-the-art BCIs have a slow spelling rate, which inevitably places a burden on patients' quality of life. Novel techniques address this problem by following a bio-mimetic approach, which consists of decoding sensory-motor cortex (SMC) activity that underlies the movements of the vocal tract's articulators. As recording articulatory data in combination with neural recordings is often unfeasible, the goal of this study was to develop an acoustic-to-articulatory inversion (AAI) model, i.e. an algorithm that generates articulatory data (speech gestures) from acoustics. A fully convolutional neural network was trained to solve the AAI mapping, and was tested on an unseen acoustic set, recorded simultaneously with neural data. Representational similarity analysis was then used to assess the relationship between predicted gestures and neural responses. The network's predictions and targets were significantly correlated. Moreover, SMC neural activity was correlated to the vocal tract gestural dynamics. The present AAI model has the potential to further our understanding of the relationship between neural, gestural and acoustic signals and lay the foundations for the development of a bio-mimetic speech BCI. Clinical Relevance- This study investigates the relationship between articulatory gestures during speech and the underlying neural activity. The topic is central for development of brain-computer interfaces for severely paralysed individuals.

UI MeSH Term Description Entries
D007446 Chromosome Inversion An aberration in which a chromosomal segment is deleted and reinserted in the same place but turned 180 degrees from its original orientation, so that the gene sequence for the segment is reversed with respect to that of the rest of the chromosome. Inversion, Chromosome,Inversion, Chromosomal,Chromosomal Inversion,Chromosomal Inversions,Chromosome Inversions,Inversions, Chromosomal,Inversions, Chromosome
D007802 Language A verbal or nonverbal means of communicating ideas or feelings. Dialect,Dialects,Languages
D010243 Paralysis A general term most often used to describe severe or complete loss of muscle strength due to motor system disease from the level of the cerebral cortex to the muscle fiber. This term may also occasionally refer to a loss of sensory function. (From Adams et al., Principles of Neurology, 6th ed, p45) Palsy,Plegia,Todd Paralysis,Todd's Paralysis,Palsies,Paralyses,Paralysis, Todd,Paralysis, Todd's,Plegias,Todds Paralysis
D011788 Quality of Life A generic concept reflecting concern with the modification and enhancement of life attributes, e.g., physical, political, moral, social environment as well as health and disease. HRQOL,Health-Related Quality Of Life,Life Quality,Health Related Quality Of Life
D005868 Gestures Movement of a part of the body for the purpose of communication. Gesture
D006801 Humans Members of the species Homo sapiens. Homo sapiens,Man (Taxonomy),Human,Man, Modern,Modern Man
D000162 Acoustics The branch of physics that deals with sound and sound waves. In medicine it is often applied in procedures in speech and hearing studies. With regard to the environment, it refers to the characteristics of a room, auditorium, theatre, building, etc. that determines the audibility or fidelity of sounds in it. (From Random House Unabridged Dictionary, 2d ed) Acoustic
D013060 Speech Communication through a system of conventional vocal symbols. Public Speaking,Speaking, Public

Related Publications

Paolo Favero, and Julia Berezutskaya, and Nick F Ramsey, and Aleksei Nazarov, and Zachary V Freudenburg
March 2012, The Journal of the Acoustical Society of America,
Paolo Favero, and Julia Berezutskaya, and Nick F Ramsey, and Aleksei Nazarov, and Zachary V Freudenburg
October 2014, The Journal of the Acoustical Society of America,
Paolo Favero, and Julia Berezutskaya, and Nick F Ramsey, and Aleksei Nazarov, and Zachary V Freudenburg
January 2023, The Journal of the Acoustical Society of America,
Paolo Favero, and Julia Berezutskaya, and Nick F Ramsey, and Aleksei Nazarov, and Zachary V Freudenburg
July 2018, The Journal of the Acoustical Society of America,
Paolo Favero, and Julia Berezutskaya, and Nick F Ramsey, and Aleksei Nazarov, and Zachary V Freudenburg
March 2016, Computer speech & language,
Paolo Favero, and Julia Berezutskaya, and Nick F Ramsey, and Aleksei Nazarov, and Zachary V Freudenburg
August 1992, The Journal of the Acoustical Society of America,
Paolo Favero, and Julia Berezutskaya, and Nick F Ramsey, and Aleksei Nazarov, and Zachary V Freudenburg
October 2022, Journal of experimental psychology. Learning, memory, and cognition,
Paolo Favero, and Julia Berezutskaya, and Nick F Ramsey, and Aleksei Nazarov, and Zachary V Freudenburg
August 2009, NeuroImage,
Paolo Favero, and Julia Berezutskaya, and Nick F Ramsey, and Aleksei Nazarov, and Zachary V Freudenburg
January 2015, Annual International Conference of the IEEE Engineering in Medicine and Biology Society. IEEE Engineering in Medicine and Biology Society. Annual International Conference,
Paolo Favero, and Julia Berezutskaya, and Nick F Ramsey, and Aleksei Nazarov, and Zachary V Freudenburg
June 2011, The Journal of the Acoustical Society of America,
Copied contents to your clipboard!