Audiovisual semantic interactions between linguistic and nonlinguistic stimuli: The time-courses and categorical specificity

Journal article

We examined the time-courses and categorical specificity of the crossmodal semantic congruency effects elicited by naturalistic sounds and spoken words on the processing of visual pictures (Experiment 1) and printed words (Experiment 2). Auditory cues were presented at 7 different stimulus onset asynchronies (SOAs) with respect to the visual targets, and participants made speeded categorization judgments (living vs. nonliving). Three common effects were observed across 2 experiments: Both naturalistic sounds and spoken words induced a slowly emerging congruency effect when leading by 250 ms or more in the congruent compared with the incongruent condition, and a rapidly emerging inhibitory effect when leading by 250 ms or less in the incongruent condition as opposed to the noise condition. Only spoken words that did not match the visual targets elicited an additional inhibitory effect when leading by 100 ms or when presented simultaneously. Compared with nonlinguistic stimuli, the crossmodal congruency effects associated with linguistic stimuli occurred over a wider range of SOAs and occurred at a more specific level of the category hierarchy (i.e., the basic level) than was required by the task. A comprehensive framework is proposed to provide a dynamic view regarding how meaning is extracted during the processing of visual or auditory linguistic and nonlinguistic stimuli, therefore contributing to our understanding of multisensory semantic processing in humans.

Authors: Chen, Y; Spence, C

• Publication status: Published
• Peer review status: Peer reviewed
• Publisher: American Psychological Association
• Journal: Journal of Experimental Psychology: Human Perception and Performance
• Volume: 44
• Issue: 10
• Pages: 1488–1507
• Publication date: 2018-04-30
• Acceptance date: 2018-02-21
• DOI: 10.1037/xhp0000545
• EISSN: 1939-1277
• ISSN: 0096-1523
• Keywords: categorization; meaning; sound; crossmodal; word; lexical; picture