Comparing audiovisual semantic interactions between linguistic and non-linguistic stimuli

We examined the time-courses of the crossmodal semantic congruency effects elicited by naturalistic sounds or spoken words on the processing of visual pictures and printed words. Auditory primes were presented at seven stimulus onset asynchronies (SOAs) with respect to the visual targets, ranging from auditory leading by 1000 ms to auditory lagging by 250 ms. Participants made speeded categorization judgments (living vs. non-living) regarding the visual targets. Three common effects were observed for both types of visual targets: Both naturalistic sounds and spoken words induced a slow facilitatory effect when leading by 250 ms or more in the congruent condition, but induced a rapid inhibitory effect when leading by 250 ms or less in the incongruent condition. Only spoken words that did not match the visual targets elicited an additional inhibitory effect when leading by 100 ms or when presented simultaneously. As compared to non-linguistic stimuli, the priming effects associated with linguistic stimuli occurred over a wider range of SOAs, and the interactions occurred at a more specific level of the category hierarchy (i.e., the basic level) than required by the task. These results suggest different routes to semantic access: Linguistic stimuli accessed meaning mediated by the lexical representation that activated semantic representations at the corresponding level of the category hierarchy precisely, whereas non-linguistic stimuli accessed their meaning rapidly and directly, but first accessed meaning at a higher-level of semantic hierarchy. We propose a comprehensive framework to provide a dynamic view regarding how meaning is extracted during the processing of linguistic and non-linguistic stimuli presented in the visual and auditory modalities, therefore contributing to our understanding of the human semantic system.