Reassessing the ability of simple recurrent networks (SRNs) to account for verbal working memory performance

Nicolas Ruh,¹ Kerstin Klöckner² & Lars Konieczny^1

1 University of Freiburg, ² Saarland University

We will present results from extensive simulation studies and a series of on-line reading experiments on Subject vs. Object RCs such as (1) to (4).

(1)	Der Wärter, der den Häftling beleidigt hatte, entdeckte den Tunnel.
	The guard, who[nom] the[acc] prisoner insulted had, discovered the tunnel.
(2)	The guard who had insulted the prisoner discovered the tunnel.
(3)	Der Wärter, den der Häftling beleidigt hatte, entdeckte den Tunnel.
	The guard, who[acc] the[nom] prisoner insulted had, discovered the tunnel.
(4)	The guard who the prisoner had insulted discovered the tunnel.

In a recent proposal, MacDonald & Christiansen (2002) claimed that the well established processing difference of English Subject vs. Object RCs (King & Just, 1991; Gibson, 1998) can be explained best by sentences' differential degree of word order regularity, as captured by Simple Recurrent Networks: Whereas word order in English Subject RCs (2) resembles that of the most frequent main clause (S-verb-O), Object RCs (4) suffer from irregular (i.e., much less frequent) word order (O-S-verb). Furthermore, the fact that low-reading span participants suffer from object-extractions more than high-span readers has been attributed to low-spans' lower degree of reading experience, as simulated by the number of SRN training epochs.

Our results cast serious doubt on their interpretations. In German, in contrast to English, word order in SRCs (1), S-O-verb, differs from main clause word order (S-verb-O). Nevertheless, reading time data still exhibit a clear advantage of SRCs over ORCs. We ran a series of simulations on SRNs, keeping architecture and training parameters as close as possible to MacDonald & Christiansen’s. While we succeeded in replicating the original data on English input, the advantage of SRCs disappeared and, in early epochs, even reversed after switching to German. A careful reanalysis of both the English and German network behaviour indicated that:

the performance of the networks was mainly determined by very local (2-3 words) regularities;
the network failed to capture the subject/main verb number agreement across the distance of the embedded relative clause, even after ample training (12 epochs);
the network did not acquire a notion of RC-embedding, thus misinterpreting parts of the embedded sentence as inaccurate main clause input.

The findings suggest that MacDonald & Christiansen’s main results may be simulation artefacts. It remains an open question though whether a more appropriate simulation setting or network architecture will be able to account for the data.

References

Gibson, E. (1998). Linguistic complexity: locality of syntactic dependencies. Cognition, 68,1, 1-76.

King, J., & Just, M. A. (1991). Individual differences in syntactic processing: The role of working memory. Journal of Memory and Language, 30, 580-602.

MacDonald, M. C., & Christiansen, M. H. (2002). Reassessing working memory: A comment on Just and Carpenter (1992) and Waters & Caplan (1996). Psychological Review.