Yuki Kamide1, Christoph Scheepers2, Gerry Altmann1 & Matthew Crocker2
1 University of York, 2 Saarland University
The present study investigates predictive processing during the incremental integration of visual scenes and concurrent auditory input, using the "visual-world" eye-tracking paradigm (e.g., Cooper, 1974; Tanenhaus et al., 1995). Altmann & Kamide (1999) found that selectional restrictions associated with verbs (e.g., 'eat') can be used to predict the semantic class of forthcoming Themes (e.g., as edible). The prediction was evidenced by anticipatory eye movements to the appropriate Theme object (a cake) prior to the onset of the post-verbal referring expression ('the cake'). However, the English study did not fully demonstrate that syntax plays a role in the predictive process --- one could argue that the anticipatory eye-movements merely reflected semantic integration of the visual context with the (semantic interpretation of the) sentence fragment heard thus far.
German provides an opportunity to explore whether there is a purely syntactic component to the predictive process by virtue of its case marking system. In principle, case-marking on the sentence-initial noun phrase, coupled with information about the verb, could be used to predict which object in a concurrent visual scene might most plausibly be referred to post-verbally.
In the main study we shall report, the following sentential conditions were presented concurrently with a visual scene depicting a hare, a cabbage, a fox, and a tree (distractor):
| (1) |
Der Hase frißt gleich den Kohl. |
| (2) |
Den Hasen frißt gleich der Fuchs. The [acc] hare eats in-a-moment the [nom] fox. |
The fact that the hare is nominative (most likely the Agent) in (1) and accusative (most likely the Patient) in (2) would suggest, in combination with the verb 'frißt' (eats), that the cabbage in (1) and the fox in (2) are the objects which will most plausibly be referred to post-verbally. Thus, if syntactic (case-marking) information and the semantic information associated with the verb interact during the prediction of the post-verbal referring expression, the plausible objects (cabbage and fox in (1) and (2), respectively) will be looked at more often than the implausible objects (fox and cabbage in 1 and 2 respectively). Moreover, we should observe such a pattern before the onset of the post-verbal noun phrase. This was exactly what the experiment revealed (the relevant effects were reliable before the onset of the determiner of NP2). We conclude that anticipation during sentence processing does not rely solely on semantic processing, but on syntactic processes also. Overall, the results indicate the rapid integration of the two different types of information (syntactic and semantic) during the incremental prediction process.
References
Altmann, G. T. M., & Kamide, Y. (1999). Incremental interpretation at verbs: Restricting the domain of subsequent reference. Cognition, 73, 247-264.
Cooper, R. M. (1974). The control of eye fixation by the meaning of spoken language: A new methodology for the real-time investigation of speech perception, memory, and language processing. Cognitive Psychology, 6, 84-107.
Tanenhaus, M. K., Spivey-Knowlton, M. J., Eberhard, K. M., & Sedivy, J. C. (1995). Integration of visual and linguistic information in spoken language comprehension. Science, 268(5217), 1632-1634.