The McGurk Effect: Can We Hear Through Our Eyes?

Our brain is an organ full of complexity and capable of incredible things. There is no doubt that its operation and efficiency never cease to amaze us, although sometimes it can also make mistakes. An example of this is observed in situations in which we receive stimuli in an ambiguous way and understanding becomes difficult. Faced with these types of scenarios, the brain implements strategies that help it fill in the information.

At present we live in many situations in which sound interferences appear, as we use new technologies on a daily basis. Video calls, online meetings and a long etcetera are part of our routines. Therefore, many people experience this curious phenomenon in everyday life.

Let's imagine that we are in a very important conference with our boss and just when he asks us a question there is some interference in the sound. Our brain will try to understand the message relying on the information that reaches us at a visual level, especially from the face and lips of the interlocutor. Although this "trap" can help us at times, it is not without errors, resulting in the McGurk effect..

What is the McGurk effect?

As we have been saying, the McGurk effect is a perceptual phenomenon whereby visual and auditory information are intermingled in situations where the message is difficult to understand. A characteristic of our brain is that it works in an integrated way, in such a way that our senses are not independent, but are connected. Although vision is the most primary sense for humans, the perception of speech is clearly multimodal. That is, to understand our interlocutor we require information from various sensory modalities, mainly vision and hearing.

Although it is believed that only those with hearing impairment rely on vision as a compensatory strategy, this tactic occurs in everyone. In fact, our perception of the volume of the message changes when the interlocutor is visible to us. When we see the person who is speaking to us, we have the sensation of hearing their voice at a higher volume.

At the brain level, both hemispheres are involved in this effect, as they work together to integrate the speech information that is received at the visual and auditory levels. What's more, In our brain there is an area, the superior temporal sulcus, which is especially involved in the task of integrating information from various perceptual channels.

Another curiosity about the McGurk effect is that it is maintained regardless of whether the person knows the phenomenon or not. This is different from what happens, for example, in optical illusions, where once the illusion is detected it can be decomposed.

Study of the McGurk phenomenon

The McGurk effect was first studied in the 1970s by two cognitive psychologists named Harry McGurk and John McDonald.. This phenomenon was described by pure chance, when McGurk and his partner, McDonald, asked a technician to record a video articulating a different phoneme than the one he was actually emitting with his voice. The video was intended for research on language perception in babies. However, when it was reproduced, both researchers were surprised to hear a third phoneme different from the one their technician had emitted and articulated.

After this finding, they decided to formalize an experiment to test the phenomenon empirically. In this it was found that when a person moves their lips pronouncing the syllable “ga” while actually saying “ba” aloud, the brain will receive the message “da”. This means that the auditory and visual information do not always coincide. This effect also occurs with other syllable combinations.

For example, it can be achieved with the combination “ka” (visual) and “pa” (auditory), which gives rise to the perception of “ta”. Besides, also this phenomenon can be observed not only with isolated syllables, but also with complete sentences. As a result of this work, the authors produced the article "Listening to lips and seeing voices", which was published in the prestigious journal Nature in 1976.

The McGurk effect in different population groups: what differences are there?

Over time, this phenomenon has been studied in special populations in order to see if it occurred to the same extent as in the general population. These have been, schematically, the main results of the studies.

In people with dyslexia, the effect has been shown to be less with respect to people of the same chronological age. A lower McGurk effect has also been detected in Alzheimer's patients. It has been suggested that these patients suffer a worse interhemispheric connection, which makes it difficult to integrate information and therefore reduces the intensity of the phenomenon compared to people without Alzheimer's.

In children with specific language impairment this effect also seems to occur to a lesser extent. It is believed that this can be explained by the fact that these children pay less attention to visual information than to auditory information when they perceive speech. A reduced effect also appears to be observed in children with Autism Spectrum Disorder (ASD). Interestingly, if this same experiment is performed using non-human stimuli (for example, instead of using the human voice using sounds of objects), the results are equal to those obtained in children without ASD.

In aphasic people it has also been proven that the McGurk effect is impaired. When the perception of language is affected in an aphasic patient, it does so at all levels (visual and auditory), so it is expected that minimal results will be obtained in the experiment. In patients suffering from schizophrenia, a more subtle effect has also been observed, although it is not diminished as it occurs in other pathologies. It has been observed that the audiovisual integration of these people is somewhat slower than in the general population. In addition, they show greater sensitivity to auditory information than to visual information.

In people undergoing a callosotomy (surgical section of the corpus callosum for medical reasons), the McGurk effect does not go away, although it is more subtle. This result is to be expected, since the corpus callosum constitutes a key structure for the interhemispheric connection. If this is reduced, the integration of the information will also reduce it, thus minimizing the intensity of the effect. Those who suffer some kind of damage to the left hemisphere show an above-average McGurk effect. This is because these types of patients rely much more than the control group on visual signals as a form of compensation.

Instead, those who suffer damage to the right hemisphere will show a lesser effect, since both the audiovisual and visual integration will be damaged. Similarly, it has been observed that right-handed people are more likely to show this perceptual effect.

In addition, it has been proven that there are certain strategies that serve to reduce this effect on purpose. For example, if the person shifts their attention to a tactile task, the phenomenon becomes more subtle. Touch is a sensory perception, just like hearing and vision, so increasing attention to this modality reduces the amount we dedicate to vision and hearing.

In addition to all of the above, the relationship between the McGurk effect and the language spoken has also been studied. It seems that speakers from western countries, such as Germany, Spain or Italy, show a much more pronounced effect than those from eastern countries. The structure of Asian languages, such as Chinese or Japanese, is believed to make it easier for their speakers to detect syllables without errors. It has also been hypothesized that in these cultures the effect is more subtle due to their lower tendency to eye contact.

Conclusions.

Although it was discovered by chance, the McGurk effect is more than just a fun curiosity. As we have seen, its study in clinical populations has given us a lot of information not only about normal speech processing, but also about how such processing is in people with some pathology.

On the other hand, these findings have served to confirm that, indeed, the perception of speech involves both the visual and auditory modalities. This system has been optimized over time, with the ultimate goal of improving our communication processes. Although they have always been associated compensation strategies with disability, the McGurk effect is proof that this idea is clearly wrong. Our brain works as a very complex network, where everything is connected and related. This allows us to face adverse situations in the best possible way.

As we mentioned at the beginning of the article, our brain never ceases to amaze us, but neither does science. Although on many occasions researchers plan their work in a meticulous way with the aim of investigating certain topics, sometimes something as banal as chance leads research in other directions. Thanks to a mistake, today we know that perceiving speech is more than listening, but also that communication for humans is almost as important as water. Although in certain serious pathologies it can be damaged, our brain always does the impossible to keep a trace of it.