Draft:Prediction in Reading

Constraint/Predictability Effects

The ability to predict which word will come next in a sentence is thought to support fluent and efficient reading. How predictable a word is in a sentence depends on its position and on the words that precede it. For example, in the sentence “Before a movie starts, they tell you to locate all of the emergency _____,” the knowledge about movie theaters and the preceding word “emergency” provides context that makes “exits” a likely continuation. When a word can be predicted in this way, readers can process the sentence more easily and may be less likely to reread earlier parts of the sentence after encountering the new word. Psycholinguists, who study the relationship between language and the mind, investigate how predictable or unpredictable words influence sentence processing. Researchers have used behavioral, eye-tracking, and electroencephalography (EEG) measures to study questions about language processing.

Predictability Measures

One of the most common measures of predictability is a “fill-in-the-blank” approach known as the cloze procedure. This task was originally established in 1953 by Wilson Taylor for educational purposes, where words were removed from a passage and students were asked to fill in the blanks using information from the surrounding context and their prior knowledge. Psycholinguistic researchers have adapted this task for single sentence contexts to estimate how likely specific words are in a given context. In this adapted version, participants are presented with the beginning of a sentence (for example, The teacher asked the students to get out a pen and a piece of _____) and are asked to provide the next word. In this example, the available context leads most participants to respond with the same word (paper).

Responses for each sentence are collected from many participants to calculate a cloze probability (or cloze value), defined as the percentage of respondents who provided a particular word. Higher cloze values indicate higher predictability, whereas lower cloze values indicate lower predictability within that context. Although the task is widely used, there is some variability in specific task instructions: participants may be asked to provide the “most natural” continuation, “most plausible” continuation, or “best” continuation. All of these instructions are intended to index predictability, but subtle differences in wording may influence how people respond.

While the cloze procedure is helpful for estimating which upcoming words people would predict, this method can be resource intensive and time consuming. As a result, other methods that do not require human responses have been developed. These methods use computational measures that rely on statistics to determine a word’s predictability. Surprisal is one such measure. It reflects the probability that a given word will occur based on preceding context (its conditional probability). Surprisal is calculated using a language model, a statistical system trained on a large body of text (a corpus) made up of samples from written and spoken language. Corpora are language specific; examples for English language corpora include the American National Corpus, the Corpus of Contemporary American English, and the British National Corpus. This measure indexes the opposite of cloze values. Cloze values capture how commonly people supply a particular word, but surprisal values indicate how unlikely a word is. Higher surprisal values indicate lower predictability (a reader would be surprised to encounter the word), whereas lower surprisal values indicate higher predictability (the word is strongly expected). An advantage of surprisal is that values can be generated for every word in a sentence, rather than only for a single position, as in the standard cloze task.

Using both human-based and computational measures, researchers can manipulate the predictability of any word in a sentence by adjusting how constraining the prior context is. High-constraint sentences provide sufficient context to limit the possible continuations, and they are usually completed with the word that has the highest cloze value. For example, in the sentence “When I got to the doctor’s office, they told me I needed to fill out a new patient _____,” the word with the highest cloze value is “form”. Low-constraint sentences, in contrast, are relatively vague, and no single continuation has a high cloze value. The sentence “The woman showed her daughter how to properly _____” does not strongly constrain any particular word. A wide variety of continuations may be appropriate; for example, the woman could show her daughter how to apply makeup, complete a task, or dress for an event. Because the context is less specific, there is unlikely to be agreement on a single continuation. Researchers may manipulate the amount of context they provide to generate sentences with a range of agreement levels for any particular word.

Behavioral Research

Early research on the effects of sentence constraint (high vs. low) on reading used lexical decision tasks, in which participants see either a real word or a non-word and respond “yes” or “no” to indicate whether they saw a real word. Participants respond more quickly to words that are supported by prior context (for example, when they have just read the sentence up to the missing word) than to words presented without accompanying context. Additionally, participants respond faster when the real-word option in the lexical decision task is the best completion, or highest cloze word, for that sentence than when the real-word option is less expected, or lower cloze words. Similarly, participants respond faster when the real-word option is contextually supported by the sentence frame than when the real-word option is unrelated to the sentence.

Eye-Tracking Research

Eye tracking provides information about precisely where and for how long a reader looks at different parts of a text. In psycholinguistics, this makes it possible to examine processing at the level of individual words within a sentence. Eye-tracking studies of natural reading have shown that expected or predictable words tend to receive shorter initial reading times than unexpected or unpredictable words, as reflected in fixation durations (periods when the eyes are relatively still and able to take in visual information). In addition, predictable words are more likely to be skipped over and never directly focused on. Findings for these early reading measures, such as fixations and word skipping, which are associated with word identification, are robust. In contrast, effects of predictability on later reading measures, which are thought to reflect integrating the word into the unfolding sentence context, are less consistent. These later measures include eye movements that return to earlier portions of the text (regressions).

While there is broad agreement that a word’s predictability influences the amount of time required to process it, the pattern and size of this benefit were not clear until recently. A meta-analysis by Brothers and Kuperberg (2021) investigated research on constraint to address debates about whether predictability effects on reading times follow linear or logarithmic patterns. Combining results from 8 studies, including 218 participants, they found support for easier processing for high-constraint contexts, as indicated by shorter reading times. This effect was observed in all 4 measures: first fixation (15.9ms faster), gaze duration (21.4ms faster), total reading time (33ms faster), and fixation rate (6.4% reduction). Additionally, the relationships between cloze values and all measures were best fit by a linear pattern, which the authors interpreted as consistent with a proportional prediction in which multiple words are pre-activated in proportion to their likelihood.

Many researchers argue that predictive processing plays an important role in written language comprehension. Some accounts suggest that older adults may rely on predictive processing to a greater extent than younger adults, at least in part to compensate for slower processing in other domains. However, meta-analytic evidence does not entirely support this claim. In alphabetic languages, there is little difference between younger and older adults in the size of predictability effects in any eye-movement measure, with the largest group difference being about 40ms. In non-alphabetic languages, such as Chinese, there is no such difference. This meta-analysis also revealed no significant differences in predictability effects on word skipping between age groups, which does not support riskier reading for older adults.

Electroencephalography (EEG) Research

EEG studies of reading generally focus on event-related potentials (ERPs), which are brain response patterns associated that are time-locked to specific events, such as the presentation of a word. One of the most widely studied ERP components in language research is the N400, discovered by Kutas and Hillyard. The N400 is sensitive to how well a word fits its context. The N400 shows an inverse relationship with cloze values, such that encountering highly predictable target words (high cloze values) elicits smaller N400 responses than encountering less predictable words. The amplitude of the N400 is often interpreted as reflecting the effort required to integrate a word into the current context, with smaller amplitudes for expected words indicating that they require less effort to process.

In a study manipulating whether participants actually encountered an expected word, readers were able to detect mismatches between their expectations and the input based on an article before the target noun. Encountering contextually inappropriate (or unexpected) articles produced larger N400 responses than encountering contextually appropriate articles, suggesting that readers may activate specific upcoming words before they are visually presented during reading.

In addition to the N400, the later Frontal Positivity (FP) is an ERP component that has been linked to disconfirmed predictions. Disconfirmed predictions occur when the word that appears does not match the word that was predicted based on the preceding context. Specifically, the FP is elicited by encountering an unexpected, but plausible word. Encountering anomalous, non-plausible, words do not produce the same pattern of ERP activity. This elicitation pattern of FP differs from that of the N400 and has been taken as evidence that readers may predict specific lexical items and that there is a processing cost when those predictions are wrong.

Modulation of Predictability

The effects of predictability are also sensitive to readers’ task goals, which influence how deeply they engage with the text. When generating predictions is unlikely to benefit readers, such as during proofreading or in contexts where predictions are frequently violated, predictability effects on reading times become smaller. In contrast, compared with reading for comprehension, explicitly asking participants to predict the upcoming word tends to lead to larger N400 differences between expected and unexpected words.

Mechanisms of Prediction Effects

There has been long-standing debate about what predictability effects in reading actually reflect. Some accounts propose that context can only influence processing after a word has been identified. If so, the observed effects reflect easier integration of an already recognized word into its context. Alternatively, other models argue that contextual information can be used early, supporting anticipatory processing in which likely upcoming words are partially activated before they appear. Although recent studies provide evidence that supports this view, some researchers still argue that highly predictable sentences are extremely rare in natural reading due to the recursive nature of language.

Aspects of Words Predicted in Anticipatory Models

If readers do use anticipatory processes, an important question concerns the nature of the information predicted. Do readers just make predictions about the meaning of upcoming words? Or do they also make predictions about the appearance of upcoming words?

Evidence for predictions based on visual characteristics has been found in both eye-tracking and ERP studies. Much of this research has used orthographic neighbors (words that differ from each other by only a single letter, such as cat and cap) or letter transpositions (words created by swapping letters within the word, such as swapping the “a” and “l” from “salt” to form “slat”), which share visual features with an expected word. In eye-tracking studies, when an unexpected word is visually similar to the predicted word, it tends to receive shorter reading times, and readers are less likely to return to earlier portions of the text to assess contextual fit. In ERP studies, visually similar words elicit a smaller N400 than words that do not look like an expected word. This effect is not observed for low-constraint sentences, which offer insufficient contextual support for generating specific predictions. Under these low-constraint conditions, N400 amplitudes do not robustly differ between visually similar and dissimilar words, consistent with the idea that readers are not forming strong expectations about the appearance of upcoming words.

Although eye-tracking and EEG findings converge in supporting orthographic or visually based prediction, results are less consistent for semantic expectations. Semantic expectations have often been examined using the gaze-contingent boundary paradigm. In this paradigm, an upcoming word in parafoveal vision (just outside the current point of visual focus) is initially replaced by another word. When the reader’s eyes cross an invisible boundary and move onto that word, it is replaced with the intended target. In eye-tracking studies using this method, there is typically no benefit when the manipulated word is semantically related to the expected word (for example, replacing “daisy” with “tulip” versus “pizza”). In contrast, ERP studies show there is an observed benefit for semantically related words. When readers encounter unexpected words that belong to the same category as the expected word (for example replacing “daisy” with “tulip”), N400 responses are small compared to N400 responses to words from an unrelated category (for example, replacing “daisy” with “pizza”). When words are semantically related to an expected word, the N400 response is smaller than when they are unrelated to the expected word, indicating semantically related words are easier to process.

Future Directions

Predictability is widely regarded as an important factor that influences reading, and there remain many open questions about how predictable contexts, and violations of those predictions, affect both eye movements and brain responses. In the eye-movement literature, predictability effects are robust for early measures linked to word recognition but are not as consistent for later measures. In ERP literature, work has focused on the N400, but recent research increasingly examined later measures, such as the frontal positivity. These later components are often discussed in relation to prediction error costs.

References