NSF grant to develop computational tools for modeling neural data during natural listening

We’re very pleased to have received an NSF CRCNS grant to develop computational models of brain systems involved in sentence comprehension. This is part of a collaboration that includes John Hale (Cornell), funded under a separate award, and Christophe Pallier and colleagues (Paris) funded under an award from the ANR.

The basic idea of the project is this: Computational neuroscience has matured to the point where ideas from computational linguistics may now be applied to the analysis of neural signals. This offers a new way of studying human language in the brain. While dominant models assign intuitive verbal labels to nodes of the “language network” (for example, see the articles collected in the mega-volume edited by Hickok and Small), new investigations of information flowing through this network use explicit language models based on n-grams, dependency relations and even phrase structure (e.g. Wehbe et al., 2014; Willems et al., 2015, Brennan et al., 2016). Applying increasingly realistic conceptualizations of sentence structure, this new approach goes beyond intuitive verbal labels by matching particular mechanisms to particular nodes of the brain’s language network. Our project for the first time rigorously compares alternative candidate mechanisms of language comprehension.

Figure 1: Modeling approach: derive expected neural signals for sentence comprehension from alternative conceptions of language structure. (1) Alternative language models describe the cog- nitive states of an individual listening to a story word-by-word. (2) These states are summarized via a complexity metric, and matched with neural signals via a neural response function. The hemodynamic response function used in fMRI research is illustrated here. Potentially confound- ing covariates may be statistically removed. (3-4) Predicted neural time-courses from step 2 are tested against neural signals recorded from individuals who passively listened to a story.
Figure 1: Modeling approach: derive expected neural signals for sentence comprehension from alternative conceptions of language structure. (1) Alternative language models describe the cog- nitive states of an individual listening to a story word-by-word. (2) These states are summarized via a complexity metric, and matched with neural signals via a neural response function. The hemodynamic response function used in fMRI research is illustrated here. Potentially confound- ing covariates may be statistically removed. (3-4) Predicted neural time-courses from step 2 are tested against neural signals recorded from individuals who passively listened to a story.

Read on for more details!

We develop models that integrate parsing algorithms with information-theoretical complexity metrics which have been widely applied at the behavioral level in psycholinguistics. These neuro-computational models yield time-locked predictions about stimulus texts; for instance that a word should be particularly difficult because it is unexpected in context. We apply the models to naturalistic narratives such as The Little Prince and its translation from French into English. We test model-derived theoretical predictions against data that is collected using electroencephalography (EEG) and functional neuroimaging (fMRI), across both languages. The fit or lack thereof between the predictions and the neural signals drives the development of more realistic models. The innovative idea is that the same notion of parsing algorithm, as studied in computer science, may also serve as a theoretical model of brain function.

Figure 2: Alternative predictions concerning neural signals that reflect grammatical expectation from language models with different domains of locality. A Markov model reflects a lin- ear domain while a phrase-structure grammar reflects a hierarchical domain. Expectations are modeled as probability distributions over sentence-remainders (red question-marks at right). The predictions come from language models which were trained on 6.5 million words of French liter- ature. Both models use surprisal to link expectations with predicted neural signals
Figure 2: Alternative predictions concerning neural signals that reflect grammatical expectation from language models with different domains of locality. A Markov model reflects a lin- ear domain while a phrase-structure grammar reflects a hierarchical domain. Expectations are modeled as probability distributions over sentence-remainders (red question-marks at right). The predictions come from language models which were trained on 6.5 million words of French liter- ature. Both models use surprisal to link expectations with predicted neural signals

With these models, we pursue two specific questions regarding language comprehension in the brain:

  1. What aspects of sentence structure determine our expectations for upcoming words?
  2. What is the detailed balance between memorization and composition in natural language?

For the first question, we computing alternative predictions from neuro-computational models that differ in the “domain of locality”, or size of the recombinable units of analysis. Doing so allows us to test which theories of sentence structure align best with neural signals collected under “everyday listening” conditions.

Figure 3: (a) Multi-word expressions defined by their Degree of Cohesion (Dice’s coefficient) differ between French and English. (b) Predictions for neural signals associated with the number of syntactic composition steps when MWEs are ignored are shown in solid. The dotted line indicates how a parsing model that incorporates MWEs as stored chunks (blue) produces quantitatively different predictions for a neural signal associated with composition operations.
Figure 3: (a) Multi-word expressions defined by their Degree of Cohesion (Dice’s coefficient) differ between French and English. (b) Predictions for neural signals associated with the number of syntactic composition steps when MWEs are ignored are shown in solid. The dotted line indicates how a parsing model that incorporates MWEs as stored chunks (blue) produces quantitatively different predictions for a neural signal associated with composition operations.

The team’s experience with “multi-word expressions” facilitates the investigation of question (2). These are strings of words that are likely to be retrieved from memory as opposed to being composed on the fly. They operationalize a central distinction between two types of comprehension that engage different neural circuits.

 

 

 

The Team

John T. Hale Linguistics and Cognitive Science Cornell University
Wen-Ming Luh Cornell MRI Facility Cornell University
Jonathan Brennan Linguistics and Psychology University of Michigan
Christophe Pallier Cognitive Neuroimaging Université Paris Saclay
Asaf Bachrach Structures Formelles du Langage Université Paris 8
Éric de la Clegerie INRIA Université Paris Diderot
Mattheiu Constant INRIA Université Paris Diderot
Benoît Crabbé INRIA Université Paris Diderot
Benoît Sagot INRIA Université Paris Diderot