New paper on localizing compositional processing in the brain with RNNGs

As part of our NSF-funded work, we have a new paper out drawing on efforts from John Hale (DeepMind, UGA), Chris Dyer (DeepMind), and Adhi Kuncoro (DeepMind). Here we combine the deep-learning based Recurrent Neural Network Grammars developed by Chris, Adhi and others, along-with fMRI data collected by John and Jon to localize brain activity that is sensitive to explicit composition of hierarchical structure, above-and-beyond sequence-based accounts such as LSTMs. (Compared to previous fMRI work in this domain, this is the first effort to our knowledge that models structure-building while accounting for structural ambiguity.) Take-away: models with explicit composition provide a superior fit to data from the left temporal lobe (anterior and posterior) as well as the left IFG.

Brennan, J. R., Kuncoro, A., Dyer, C., & Hale, J. T. (2020) Localizing syntactic predictions using recurrent neural network grammars. Neuropsychologia 146: 1074–1079 doi: 10.1016/j.neuropsychologia.2020.107479

Abstract:

Brain activity in numerous perisylvian brain regions is modulated by the expectedness of linguistic stimuli. We leverage recent advances in computational parsing models to test what representations guide the processes reflected by this activity. Recurrent Neural Network Grammars (RNNGs) are generative models of (tree, string) pairs that use neural networks to drive derivational choices. Parsing with them yields a variety of incremental complexity metrics that we evaluate against a publicly available fMRI data-set recorded while participants simply listen to an audiobook story. Surprisal, which captures a word’s un-expectedness, correlates with a wide range of temporal and frontal regions when it is calculated based on word-sequence information using a top-performing LSTM neural network language model. The explicit encoding of hierarchy afforded by the RNNG additionally captures activity in left posterior temporal areas. A separate metric tracking the number of derivational steps taken between words correlates with activity in the left temporal lobe and inferior frontal gyrus. This pattern of results narrows down the kinds of linguistic representations at play during predictive processing across the brain’s language network.