McGurk Effect Explained

The human brain is a work of art and the illusions it can create are mind-boggling. If you trust that "seeing is believing", you may actually be right. The McGurk effect proves that what you see is what you hear. The effect centers around an observation that when you hear a sound and see the lip movements making that sound, you hear it correctly, but if you see the lips making a sound but hear a different sound, simultaneously, your brain would still prefer to process what you see, irrespective of what you hear. Sounds complicated? Let's see!

An Experiment to Demonstrate the McGurk Effect

As shown in the figure below, you will need a minimum of three people to try this experiment. Let us call one as an Observer (O in the figure), another one as the Speaker (S in the figure) and the third one as the Voice-over (V in the figure).

Let "O" stand facing towards "S" so that the 'Observer' can view the 'Speaker' directly. Place the 'Voice-over' behind "O" so that "V" is hidden.

Step 1: Take a monosyllabic pair of words, such as "Face" and "Base". Ask "S" to continuously but silently mouth the word "Face". In tandem with the lip movements of "S", let "V" loudly utter the word "Base". After a few repetitions, ask the 'Observer' what was heard. As per the McGurk effect, the observer should hear "Face" although the word he actually hears is "Base" repeated by the hidden voice-over.

Step 2: Repeat step 1 but this time, let the Observer remain standing with the eyes closed, so he does not see "S" but only hears "V". Now, of course he will confirm as having heard "Base".

You may repeat the experiment with more such monosyllabic sounds and you will understand the effect better. Sometimes, a third sound is heard by the observer depending on the choice of the monosyllable pair. For example, if V says "ba-ba" over the lip movements "ga-ga" by the speaker S, O will perceive "da-da", which is essentially a fusion of the two sounds! You can try with some more pair of words such as "Been, Beep", "Pram, Cram", "Vet, Debt", "Bait, Gate", etc.

The McGurk effect proves how much we rely more on visual cues, while almost ignoring what we hear. This is referred to as visual dominance. This playoff between our auditory and visual senses depends on a lot of factors. If your native language or mother tongue is a language other than English and if you have watched a Hollywood movie dubbed in that language, you must have experienced this perception clash at some point. The more the variations between the lip synchronization and the dubbed sound, the less enjoyable the movie becomes. This is because the bad lip synchronization irritates the two senses in our brain and we have to "guess" more of what actually is being said. This proves that speech is more than something we hear. Our amazing brain processes speech based on the body language (hand gestures, face movements, etc.) of the speaker and to a great extent our experience with what is being spoken about. For example, if you're familiar with the sequence of words (like in a song or a poem), irrespective of the lip movements and audio mismatch, you will inherently "hear" correctly.

Research and Applications of The McGurk Effect

The McGurk effect explained above and related research finds application in a variety of fields, some of which are enlisted below.

• Movie dubbing and forensic lip reading
• Speech recognition programs
• Diagnostics and study of speech disorders and language impairment
• Diagnostics and study of psychological disorders like Schizophrenia
• Diagnostics and study of brain-related disorders where brain damage has occurred due to physical injury or epileptic seizures
• Study of infants' sensitivity to audiovisual correspondence

As is the case with the other similar perceptual effects, such as the Stroop effect and the Ganong effect, the McGurk effect too has become the subject of extensive research in fields such as psycholinguistics and neurolinguistics, around the world. Hope we have stirred your curiosity with the overview and demonstration of this particular behavior of the human brain relating to audiovisual perception. So, try to see what you can hear and hear what you see, and let us know how it goes!