Semantic Prediction in Brain and Mind

Abstract

We highlight a novel brain correlate of prediction, the prediction potential (or PP), a slow negative-going potential shift preceding visual, acoustic, and spoken or written verbal stimuli that can be predicted from their context. The cortical sources underlying the prediction potential reflect perceptual and semantic features of anticipated stimuli before these appear. We highlight a novel brain correlate of prediction, the prediction potential (or PP), a slow negative-going potential shift preceding visual, acoustic, and spoken or written verbal stimuli that can be predicted from their context. The cortical sources underlying the prediction potential reflect perceptual and semantic features of anticipated stimuli before these appear. Much current research in the neural, cognitive, and social sciences focuses on prediction in perception and action. When perceiving a series of events, the item occurring next can frequently be anticipated some time before it occurs, and, similarly, in performing a series of motor acts, the next-following one is typically processed before its onset. This can be illustrated using language, where the sentence fragment ‘She takes the ice cream and she …’ predicts the target word ‘licks …’, whereas ‘ticks …’ would represent a violation of typical expectations. In spite of the great interest in prediction mechanisms, it is difficult to find reliable indexes of prediction in mind and brain. This brief note will give a recent update (see the supplemental information online). The N400 event-related potential [1.Kutas M. Federmeier K.D. Thirty years and counting: finding meaning in the N400 component of the event-related brain potential (ERP).Annu. Rev. Psychol. 2011; 62: 621-647Crossref PubMed Scopus (2441) Google Scholar] following semantically unpredictable words has been interpreted as a neurophysiological index of prediction, but a recent review argues that responses following a stimulus cannot, by definition, demonstrate prediction, as predictions always precede the critical predicted item [2.Pickering M.J. Gambi C. Predicting while comprehending language: a theory and review.Psychol. Bull. 2018; 144: 1002-1044Crossref PubMed Scopus (145) Google Scholar]. If, 300–500 ms after its onset, a word or other meaningful item elicits an enhanced N400 brain response, this response dynamic can be explained at the level of access or integration processes, rather than as a direct index of prediction [2.Pickering M.J. Gambi C. Predicting while comprehending language: a theory and review.Psychol. Bull. 2018; 144: 1002-1044Crossref PubMed Scopus (145) Google Scholar]. An exciting perspective on using the N400 for monitoring prediction is opened by examples, where a sentential context (e.g., ‘The day was breezy so the boy went outside to fly …’) predicts a phrase including at least two subsequent but interdependent words (‘a kite’) so that an N400 elicited by a first unpredicted item (e.g., ‘an …’) can therefore be taken as a prediction-related indicator of the still upcoming second item (‘… airplane’), which requires (in this case phonological) agreement with the determiner and therefore motivates the choice of the N400-eliciting word form ('an') [3.DeLong K.A. et al.Probabilistic word pre-activation during language comprehension inferred from electrical brain activity.Nat. Neurosci. 2005; 8: 1117-1121Crossref PubMed Scopus (716) Google Scholar]. Unfortunately, this sophisticated argument has recently been criticized based on failed replication attempts [4.Nieuwland M.S. et al.Large-scale replication study reveals a limit on probabilistic prediction in language comprehension.eLife. 2018; 7e33468Crossref PubMed Scopus (136) Google Scholar]. We argue here that brain responses preceding the predicted item, which provide a direct measure of the physiological correlates of prediction, are now available for a broad range of conditions where words, signs, and more elementary stimuli are strongly expected. A range of recent studies reported prestimulus predictive brain activity [5.Dikker S. Pylkkanen L. Predicting language: MEG evidence for lexical preactivation.Brain Lang. 2013; 127: 55-64Crossref PubMed Scopus (91) Google Scholar, 6.Leon-Cabrera P. et al.Ahead of time: early sentence slow cortical modulations associated to semantic prediction.Neuroimage. 2019; 189: 192-201Crossref PubMed Scopus (14) Google Scholar, 7.Leon-Cabrera P. et al.Electrophysiological correlates of semantic anticipation during speech comprehension.Neuropsychologia. 2017; 99: 326-334Crossref PubMed Scopus (23) Google Scholar, 8.Grisoni L. et al.Somatotopic semantic priming and prediction in the motor system.Cereb. Cortex. 2016; 26: 2353-2366Crossref PubMed Scopus (43) Google Scholar, 9.Grisoni L. et al.Neural correlates of semantic prediction and resolution in sentence processing.J. Neurosci. 2017; 37: 4848-4858Crossref PubMed Scopus (40) Google Scholar, 10.Grisoni L. et al.Prediction mechanisms in motor and auditory areas and their role in sound perception and language understanding.Neuroimage. 2019; 199: 206-216Crossref PubMed Scopus (13) Google Scholar] (see the supplemental information online). Dependent measures range from evoked potentials closely following the predicting stimulus, high-frequency oscillations and synchrony, to slow potential shifts preceding the predicted item and multivoxel patterns of neurometabolic activity. Several of these studies are difficult to interpret due to experimental design. For example, one study compared priming conditions in which single concrete target nouns were preceded by prime pictures [5.Dikker S. Pylkkanen L. Predicting language: MEG evidence for lexical preactivation.Brain Lang. 2013; 127: 55-64Crossref PubMed Scopus (91) Google Scholar]. Predictive primes showed the referent of that noun, but less predictive ones a range of elements. Differences between priming conditions were interpreted in terms of prediction, although, in this and similar cases, factors ranging from prime stimulus complexity to attention level and search processes could act as confounds, thus complicating an interpretation in terms of prediction. Several studies report a slow negative-going potential shift starting already hundreds of milliseconds prior to predictable stimuli, which is absent or reduced for less predictable ones [6.Leon-Cabrera P. et al.Ahead of time: early sentence slow cortical modulations associated to semantic prediction.Neuroimage. 2019; 189: 192-201Crossref PubMed Scopus (14) Google Scholar, 7.Leon-Cabrera P. et al.Electrophysiological correlates of semantic anticipation during speech comprehension.Neuropsychologia. 2017; 99: 326-334Crossref PubMed Scopus (23) Google Scholar, 8.Grisoni L. et al.Somatotopic semantic priming and prediction in the motor system.Cereb. Cortex. 2016; 26: 2353-2366Crossref PubMed Scopus (43) Google Scholar, 9.Grisoni L. et al.Neural correlates of semantic prediction and resolution in sentence processing.J. Neurosci. 2017; 37: 4848-4858Crossref PubMed Scopus (40) Google Scholar, 10.Grisoni L. et al.Prediction mechanisms in motor and auditory areas and their role in sound perception and language understanding.Neuroimage. 2019; 199: 206-216Crossref PubMed Scopus (13) Google Scholar, 11.Kilner J.M. et al.Motor activation prior to observation of a predicted movement.Nat. Neurosci. 2004; 7: 1299-1301Crossref PubMed Scopus (304) Google Scholar]. The slow potential shift is elicited by expected visual and auditory, as well as written and spoken language stimuli in tasks forcing subject to attend to these, but likewise if subjects are instructed to ignore the stimuli and focus their attention elsewhere. Two recent studies investigated brain activity in anticipation of sentence final words in a typical N400 paradigm, where sentence fragment and target were separated by 1 second, and subjects were instructed to attend to and memorize the stimuli [6.Leon-Cabrera P. et al.Ahead of time: early sentence slow cortical modulations associated to semantic prediction.Neuroimage. 2019; 189: 192-201Crossref PubMed Scopus (14) Google Scholar,7.Leon-Cabrera P. et al.Electrophysiological correlates of semantic anticipation during speech comprehension.Neuropsychologia. 2017; 99: 326-334Crossref PubMed Scopus (23) Google Scholar]. It could be shown that spoken and written sentence fragments with well-predictable endings elicited a slow negative shift of brain potentials before onset of the predictable target, whereas sentence fragments that did not strongly predict a target produced a much reduced potential shift (Figure 1). The authors argued that the observed anticipatory activity before the critical stimulus was most likely related to prediction, but acknowledge the possibility that also differences in attention could provide an explanation [6.Leon-Cabrera P. et al.Ahead of time: early sentence slow cortical modulations associated to semantic prediction.Neuroimage. 2019; 189: 192-201Crossref PubMed Scopus (14) Google Scholar]. In addition, physical or psycholinguistic differences between the sentence fragments preceding the predictable versus unpredictable items could potentially have contributed to the observed effect. To decide whether attention plays a role in eliciting the predictive potential shift, it is useful to replace the attention-demanding task by a paradigm where subjects are instructed to direct their attention away from the predictive stimuli, for example the passive oddball paradigm. In such passive paradigms, a slow negative-going potential preceded predictable basic acoustic stimuli (tones) and action-related sounds (e.g., hand clapping), but not unpredictable ones [8.Grisoni L. et al.Somatotopic semantic priming and prediction in the motor system.Cereb. Cortex. 2016; 26: 2353-2366Crossref PubMed Scopus (43) Google Scholar,10.Grisoni L. et al.Prediction mechanisms in motor and auditory areas and their role in sound perception and language understanding.Neuroimage. 2019; 199: 206-216Crossref PubMed Scopus (13) Google Scholar]. Similar negative-going predictive potentials were also found in a passive N400 paradigm with spoken sentences, where subjects were instructed to focus their attention on a silent movie and ignore the acoustic input. The fact that the potential shift occurs not only in attention-demanding tasks but also under distraction and instruction to direct attention away from the expected stimuli discourages an interpretation in terms of focused attention. In contrast, the potential shift before anticipated stimuli reflects a prediction process whose elicitation does not require preset attention towards the eliciting stimuli, but is partially ‘automatic’ in the sense that it emerges even if subjects actively focus their attention elsewhere. However, all previously mentioned reports and arguments were based on a contrast between more or less predictable items. Therefore, one may argue that different levels of predictability implicate differences with regard to other cognitive operations, including some form of stimulus-driven attention or arousal. In other words, even if predictive potentials are automatic in the sense that they appear even if subjects do not actively attend to the predictable stimuli, it could still be that predictable and unpredictable conditions automatically raise the level of attention or arousal to different degrees. To avoid this crucial drawback and resultant ambiguity of results, recent experiments compared predictive brain activity with equally predictable action sounds and words which, however, differed in their indexical or symbolic semantic meaning [9.Grisoni L. et al.Neural correlates of semantic prediction and resolution in sentence processing.J. Neurosci. 2017; 37: 4848-4858Crossref PubMed Scopus (40) Google Scholar,10.Grisoni L. et al.Prediction mechanisms in motor and auditory areas and their role in sound perception and language understanding.Neuroimage. 2019; 199: 206-216Crossref PubMed Scopus (13) Google Scholar]. Should there be brain activity reflecting specific semantic features of the predicted stimuli before these appear, this would be the strongest support that the anticipatory brain activity is indeed a (semantic) prediction potential. These studies took advantage of previous reports that the meaning of action-related sounds and words leads to markedly different brain responses. They activate cortical circuits that include representations of the bodily actions semantically related to these signs. For example, a hand clapping sound or an action word such as ‘grasp’ would activate ‘semantic neurons’ in the hand motor cortex, whereas a tongue click sound or the word ‘talk’ elicits activity in areas controlling the articulators [12.Pulvermüller F. Neural reuse of action perception circuits for language, concepts and communication.Prog. Neurobiol. 2018; 160: 1-44Crossref PubMed Scopus (114) Google Scholar]. Therefore, one important issue that needs to be clarified is whether or not brain activation patterns similar to those following and distinguishing between meaningful signs are also detectable before expected action-related sounds, words or other meaningful stimuli. If semantic predictive processes are at the heart of symbolic processing, the same neurophysiological signatures of semantic differences are expected in understanding and in predicting meaningful items. Furthermore, it is controversial whether these post-stimulus activations are true reflections of semantic meaning or rather reflect processes following and thus epiphenomenal to language understanding. If the cortical sources underlying the prediction potential reflect the body-part relationship of action sounds and words before appearance of these stimuli, there would be strong support against a postunderstanding epiphenomenal interpretation (as the upcoming items cannot be understood before they appear) and, crucially, for a role of the anticipatory potential shift in semantic prediction. Experiments using face- and hand-related action sounds [8.Grisoni L. et al.Somatotopic semantic priming and prediction in the motor system.Cereb. Cortex. 2016; 26: 2353-2366Crossref PubMed Scopus (43) Google Scholar,10.Grisoni L. et al.Prediction mechanisms in motor and auditory areas and their role in sound perception and language understanding.Neuroimage. 2019; 199: 206-216Crossref PubMed Scopus (13) Google Scholar] and words [9.Grisoni L. et al.Neural correlates of semantic prediction and resolution in sentence processing.J. Neurosci. 2017; 37: 4848-4858Crossref PubMed Scopus (40) Google Scholar] indeed showed topographically different activations in sensorimotor cortices before predictable action sounds and words (but not before unpredictable ones), which further establishes the role of the slow potential shift as a prediction potential (PP) and supports its semantic nature (Figure 2; see the supplemental information online). These recent results point to important progress in mapping the brain correlates of prediction. In particular, the slow negative-going potential shift preceding predictable stimuli appears as a robust and replicable index of predictability. Crucially, the sources underlying this potential shift differ between stimulus modalities and even index aspects of the meaning of the stimuli. Therefore, it can be called a semantic prediction potential (SPP). PPs and SPPs may be of interest for future neurocognitive research on prediction. One possibility is to use them for assessing theories about prediction. For example, it has been suggested that the motor system provides the machinery for prediction [2.Pickering M.J. Gambi C. Predicting while comprehending language: a theory and review.Psychol. Bull. 2018; 144: 1002-1044Crossref PubMed Scopus (145) Google Scholar]. The reviewed results certainly offer some support for this idea, but also demonstrate that prediction mechanisms involve multiple cortical areas outside motor systems (Figure 2). Among the many specific questions to be addressed in future is the relationship between prediction- and integration-related brain activity. To what degree are the brain responses following the critical stimuli, including mismatch negativity and N400, related to, or even influenced by the SPP [10.Grisoni L. et al.Prediction mechanisms in motor and auditory areas and their role in sound perception and language understanding.Neuroimage. 2019; 199: 206-216Crossref PubMed Scopus (13) Google Scholar]? This and many other questions may guide future neurocognitive research addressing prediction by applying the novel measures. We thank Alessandro D’Ausilio, Isabella Boux, Luciano Fadiga, Teija Kujala, Kristof Stijkers, Martin Pickering, Rosario Tomasello, three anonymous referees and the editor, Lindsey Drayton, for their invaluable comments on an earlier version of this text and a related talk. This work was supported by the European Research Council, European Union (ERC-2019-ADG 883811 MatCo - 'Material Constraints Enabling Human Cognition'), and by the Deutsche Forschungsgemeinschaft, Germany (projects DFG Pu 97/22-1 'The sound of meaning' and Pu 97/25-1, 'Phonological Networks'), Download .docx (.06 MB) Help with docx files Supplementary material

Topics & Concepts