29: Visual Processing and Action

• Successive Fixations Focus Our Attention in the Visual Field

  • Attention Selects Objects for Further Visual Examination

  • Activity in the Parietal Lobe Correlates with Attention Paid to Objects

• The Visual Scene Remains Stable Despite Continual Shifts in the Retinal Image

• Vision Lapses During Saccades

• The Parietal Cortex Provides Visual Information to the Motor Syste

• An Overall View

Vision requires eye movements. Small eye movements are essential for maintaining the contrast of objects that we are examining. Without these movements the perception of an object rapidly fades to a field of gray, a phenomenon correlated with the decreased response of neurons in area V1 (see Chapter 25). Large eye movements direct the fovea from one object to another. These movements or saccades bring the high resolution of the fovea to bear on different regions of the visual field, exploiting the high density of photoreceptors in the central fovea. Without saccades this high-resolution processing could be achieved only by moving the head or body.

The preceding chapters have described how visual images are constructed, beginning with the processing of intensity and contrast, then the integration of visual primitives, and finally the high-level processing that leads to the recognition of objects. But the visual system involves more than just object recognition. It must also support the brain's goal of assigning significance to objects in order to develop strategies for interacting with the environment. Thus the brain must be able to select some objects for greater examination while ignoring others.

In this chapter we consider how saccades support that goal. We first consider the essential benefits that saccades provide, shifting attention in the visual field and assisting with the preparation to grasp objects. We then consider the brain mechanisms that solve a major problem created by saccades—the fact that the retinal image is abruptly displaced with every saccade.

In shifting our attention from how the brain constructs a visual scene to how it uses visual information to plan actions, we now concentrate on the region of the brain referred to as the dorsal visual pathway (Figure 29–1). This pathway extends from V1 to the regions in parietal cortex that continue the intermediate level of visual processing, such as the middle temporal area, and then to other regions of parietal and frontal cortex. The regions particularly relevant to this chapter are in the parietal cortex, such as the lateral intraparietal area, but include also the frontal eye field region of the frontal cortex.

A saccade usually lasts less than 40 ms and redirects the center of sight in the visual field. Saccades occur several times per second, and each intervening period of fixation lasts several hundred milliseconds. The Russian psychologist Alfred Yarbus was the first to show that the pattern of saccades made by a human looking at a picture reflected the cognitive purpose of vision. He found that saccades were not directed equally to all parts of a scene. Areas of apparent interest were fixated most frequently, whereas background objects were ignored. For example, the faces of people were fixated repeatedly (Figure 29–2).

The image on the fovea shifts with each saccade, yet we perceive a stable visual world. How does that come about? One possibility is that the brain creates a representation of the entire visual scene from a series of visual fixations across the scene and that what we see is this summed representation of the visual world. If that were so, we should have detailed knowledge of the entire visual scene at any given instant.

A series of experiments on change blindness showed that this is not the case. These experiments involved changing a picture during the brief time when the viewer made a saccade from one part of the scene to another. If there were a relatively complete internal representation of the scene before the saccade, then any substantial change made during the saccade should have been recognized. But even a large change frequently went unrecognized. This change blindness occurred even when there were no actual eye movements, as when two pictures were shown in succession with only a brief blank between them to simulate the effect of an eye movement (Figure 29–3).

The results of the change-blindness experiments are inconsistent with the hypothesis that we are continually updating a complete representation of the visual field from second to second. Instead we seem to pay attention to only certain fragments of the scene. This selective visual attention relies on the saccades that bring the images of desired parts of the visual field onto the fovea.

Attention Selects Objects for Further Visual Examination

In the 19th century William James described attention as "the taking possession by the mind in clear and vivid form, of one out of what seem several simultaneously possible objects or trains of thought. It implies withdrawal from some things in order to deal effectively with others." James went on to describe two kinds of attention: "It is either passive, reflex, non-voluntary, effortless or active and voluntary. In passive immediate sensorial attention the stimulus is a sense-impression, either very intense, voluminous, or sudden … big things, bright things, moving things … blood."

More recently these two kinds of attention have been termed involuntary (exogenous) and voluntary (endogenous) attention, or bottom-up and top-down attention. Your attention to this page as you read it is an example of voluntary attention. If a bright light suddenly flashed, your attention would probably be pulled away involuntarily from the page.

Voluntary attention is closely linked to saccades because the fovea has a much denser array of cones than the peripheral retina (see Figure 26–1), and this permits a finer-grain analysis of objects than is possible with peripheral vision.

Attention, both voluntary and involuntary, has several measurable effects on human visual performance: It shortens reaction time and makes perception more sensitive. This increased sensitivity includes the abilities to detect objects at a lower contrast and ignore distracters close to an object. The abrupt appearance of a behaviorally irrelevant cue such as a light flash reduces the reaction time to a test stimulus presented 300 ms later in the same place, but increases reaction time when the test stimulus appears at a different place. The light flash involuntarily draws attention to itself, and attention to that location is maintained for a brief period, thus accelerating the visual response to the later test stimulus at the location. Similarly, if a subject plans a saccade to a particular part of the visual field, the contrast threshold at which any object there can be seen is lowered by 50%. The saccade, under voluntary control, draws attention to its goal.

Activity in the Parietal Lobe Correlates with Attention Paid to Objects

Clinical studies have long implicated the parietal lobe in the process of visual attention. Patients with lesions of the right parietal lobe have normal visual fields when their visual perception is studied with a single stimulus in an uncomplicated visual world. However, when presented with a more complicated world, with objects in the right (ipsilateral) and left (contralateral) visual hemifields, they tend to report more of what lies in the right visual hemifield.

This deficit, known as neglect syndrome (see Chapter 17), arises because attention is focused on the visual hemifield ipsilateral to the lesion. Even when patients are presented with only two stimuli, one in each field, they report seeing only the stimulus in the ipsilateral hemifield. They do not have the ability to focus attention in the hemifield contralateral to the lesion, and as a result they may not see everything in that hemifield, even though the sensory pathway from the eye to the striate and prestriate cortex is intact.

This neglect of the contralateral visual hemifield extends to the neglect of the contralateral half of individual objects. Patients with right parietal deficits often have difficulty reproducing drawings. When asked to draw a clock, for example, they may force all of the numbers into the right side of the face (see Figure 17–11), or when asked to draw a candlestick they may draw only its right side (Figure 29–4).

The process of attentional selection is evident at the level of single parietal neurons in the monkey. The responses of neurons in the lateral intraparietal area to a visual stimulus depend not only on the physical properties of the stimulus but also on how the monkey behaves toward it. When a monkey fixates a spot, a stimulus in the neuron's receptive field evokes a moderate response. When the animal must attend to the same stimulus, the stimulus evokes a greater response, often by a factor of two. Conditions that evoke both involuntary and voluntary attention—the abrupt onset of a visual stimulus in the receptive field or the planning of a saccade to the receptive field of the neuron—evoke still greater responses (Box 29–1).

Neurons in the lateral intraparietal area collectively represent the entire visual hemifield, but the neurons active at any one moment represent only the important or salient objects in the hemifield. That is, a few salient objects—such as the goal of an eye movement or a recent flash—evoke responses in a subset of neurons, and the activity of these neurons is greater than the background activity of the entire population of cells. Both the attention mechanisms and saccades are directed to the peak of the map.

The absolute value of the response evoked by a salient stimulus does not determine whether the stimulus is the most likely saccade target or most highly attended stimulus. When a monkey plans a saccade to a stimulus in the visual field, attention is on the goal of the saccade, and the activity evoked by the saccade plan lies at the peak of the salience map. However, if a bright light appears elsewhere in the visual field, attention is involuntarily drawn to the bright light, which evokes more neuronal activity than does the saccade plan. Thus the locus of attention can be ascertained only by examining the entire salience map and choosing its peak; it cannot be identified by monitoring activity at one point alone.

Saccades create a major challenge for visual processing. Successive saccades produce a series of discrete images, each centered where the eye is looking (Figure 29–8). Although this result might be expected to resemble a home movie with the camera moving about in a jumpy fashion, it does not. How visual scenes remain stable despite repeated shifts in focus has been a source of speculation since the 1600s.

Although the basis of this stability is not understood, changes in perception at the time of a saccade offer clues. At the time of a saccade, objects in the perceived scene do not have exactly the same spatial arrangement as in the visual field. The perceived scene appears spatially compressed such that stimuli presented just before the saccade appear closer to the presaccade point of fixation, whereas stimuli presented after the saccade appear to be closer to the saccade's target (Figure 29–9).

This spatial compression is usually no larger than half the size of the saccade and occurs only when there is a larger visual scene. It is not due to stimuli falling on different parts of the retina because of the saccade, for stimuli presented before the saccade also appear compressed. These considerations indicate that some extravisual information is involved in the processing of saccade commands.

What neuronal mechanism might underlie this apparent shift in images at the time of saccades? Neurons in the parietal cortex alter their activity preceding saccades in ways that seem remarkably related to the perceptual phenomenon. When a monkey is about to make a saccade, a neuron becomes less sensitive to the stimulus already present in its receptive field and begins to respond to the stimulus that will be within the receptive field after the saccade (Figure 29–10).

This shift in selectivity can begin even before the saccade commences. Not every stimulus in the visual field will activate the neuron, only those that will fall in the neuron's receptive field after the saccade. After completion of the saccade, the neuron is again activated only by those stimuli that are actually within its receptive field.

These observations reveal two neuronal mechanisms that contribute to the stabilization of the visual scene during saccades. First, neurons shift their receptive fields from one part of the visual field to another before the saccade occurs. This shift in receptive field is comparable to the change in perceived location of stimuli before the saccade and might contribute to the compression in the perceived visual scene at the time of the saccade. The output of the parietal neurons is interpreted as indicating that stimuli are present in one part of the visual field, although at the time of the saccade the neurons are responding to stimuli that will be in their receptive field after the saccade as well as to stimuli presently within the receptive field. If the brain assumes that stimuli at both locations are joined together, this could account for the compression of the perceived scene.

Second, because the new receptive-field position after a saccade depends on the amplitude and direction of the saccade, the parietal neurons must have advance information about the saccade. There are only two potential sources of this information. One is the feedback to the brain from the peripheral proprioceptors in the eye muscles that could inform the brain that the eye is moving. This is unlikely because both the receptive field shifts and the perceptual compression begin even before the eye moves, and the cortical representation of eye position lags the eye movement by nearly 100 ms. The second potential source is the motor system that controls movement of the eyes (see Chapter 39). This system could send a copy of its signals, known as a corollary discharge or efference copy, to the parietal cortex to inform it that the saccade is about to occur. This signal would provide the information on the amplitude and direction of the impending saccade needed to compute the new receptive-field location before the saccade actually occurs.

The specific source of a corollary discharge to the parietal cortex remains unknown, but neurons in the frontal eye field that show shifts in receptive-field sensitivity similar to those in the lateral intraparietal area do receive an identified corollary discharge. The source of this corollary discharge is the saccade-generating neurons in the superior colliculus. Because the parietal cortex and frontal eye field have strong reciprocal connections, the same corollary discharge input could affect both structures.

The corollary discharge from the superior colliculus reaches the frontal eye field through a relay in a higher-order thalamic nucleus, the medial dorsal nucleus (Figure 29–11A). Neurons in this nucleus signal the amplitude and direction of an impending saccade and could therefore provide the information needed to shift the receptive fields of neurons in the frontal eye field and lateral intraparietal area before the saccade occurs. But do they? Inactivation of neurons in the thalamic relay greatly reduces the ability of frontal eye field neurons to shift their receptive fields prior to a saccade (Figure 29–11B), providing evidence that the shift does depend on an efferent copy of the saccade motor program.

During a saccade the visual scene is not only displaced but also swept rapidly across the retina as the eye moves at high speed. Because we are unaware of this sweep, the American psychologist Edwin Holt at the beginning of the 20th century posited that there must be a central anesthesia during the eye movement.

But this cannot be true, for there are instances in which vision is quite clear during a saccade. An object can be seen during a saccade if it is moving as fast as the eye and in the same direction, as occurs for example during a saccade in the direction of a car passing the observer in a train. What may instead be the case is that much of the visual scene is ordinarily blurred by the speed of eye movement. But why does the blur not reach consciousness? Two underlying mechanisms in combination probably account for this perceptual omission.

The first mechanism is visual masking, which is the effect one image has on another when the two are presented in rapid succession. For example, a higher-contrast image reduces or eliminates the perception of a lower-contrast image. This happens during a saccade: The rapid movement of the eye over the scene produces a blurred, low-contrast image that is masked by the stable, high-contrast images before and after the saccade. This masking can be demonstrated by an experiment in which high-contrast images are not present before or after the saccade. Under these conditions a blurred image is seen. If a high-contrast image is present after the saccade, however, the blur vanishes (Figure 29–12A).

There are conditions, however, when extravisual input, such as a corollary discharge, also must be present. For example, a coarse grating can be seen even when it moves across the retina at the high speeds of saccades if the bars are oriented in the same direction as the saccade (Figure 29–12B). When the eyes are fixated, low- and middle-frequency gratings can be seen but very high-frequency gratings are not well seen. In contrast, during a saccade the detection of low-frequency gratings is markedly impaired but the highest-frequency gratings remain visible. Thus the reduction of sensitivity during the saccade, known as saccadic suppression, is specific for low-frequency gratings.

These low spatial frequencies are precisely those most likely to escape the effects of visual masking, so that a reduction in sensitivity to these stimuli must result from extravisual input such as a corollary discharge. They are also the best stimuli for the magnocellular component of the geniculostriate pathway, for the parvocellular pathway is more sensitive to high spatial frequencies. Both masking and corollary discharge must jointly act to produce the saccadic suppression that usually eliminates the intrusive motion of the scene during saccades.

Changes in neuronal activity that are associated with a reduction of visual sensitivity have been found along the visual pathway and are probably due to both visual masking and corollary discharge. Neuronal correlates of visual masking are evident in the responses of neurons in the primary visual cortex. Many V1 neurons fire when eye movement sweeps a lone stimulus across their receptive fields but do not fire when another stimulus falls within their receptive fields just before the saccade (Figure 29–13A). The masking effect of the stimulus falling within the receptive fields before the saccade is analogous to the stationary visual scene before and after a saccade masking the blur of the scene during the saccade. The same masking occurs when moving stimuli sweep across the receptive fields at saccade speeds while the monkey continues to fixate. This neuronal masking effect therefore stems entirely from visual processing; no saccade is required.

Saccadic suppression due to corollary discharge is most pronounced for stimuli with low spatial frequency and high contrast, the stimuli that are most effective for activating neurons in the magnocellular pathway of the visual system. During saccades the activity of neurons in the lateral geniculate nucleus is altered and frequently reduced, but the suppression is limited and not confined to the magnocellular layers. In the absence of visual masking, neurons in cortical area V1 show little suppression during saccades so the effect of corollary discharge on the geniculocortical pathway is limited. But in the regions of extrastriate cortex devoted to motion processing, the middle temporal and medial superior temporal areas, there is clear suppression of visual stimuli during saccades that could result from a corollary discharge.

When eye movement sweeps the stimulus across the retina, neurons do not respond, but when the stimulus is moved at the same speed in front of the stationary eye, these same neurons fire robustly (Figure 29–13B). The likely explanation of this difference is that a corollary discharge of the saccade command reduces the response of the neurons to the stimulus. One possible source of this corollary discharge is the superior-collicular signal that shifts receptive fields in frontal and parietal cortex. This is a likely possibility because visual neurons in the superior colliculus are also suppressed during saccades and this suppression stems from corollary discharge.

At both the neuronal and behavioral levels, then, there is evidence that visual masking and corollary discharge act together to reduce the disruption of vision during saccades.

Vision interacts with the supplementary and premotor systems to prepare the hands for action. When you pick up a pencil your fingers are separated from your thumb by the width of a pencil; when you pick up a drink your fingers are separated from your thumb by the width of the glass. The visual system helps to adjust the grip width before your hand arrives at the object. Similarly, when you insert a letter into a mail slot your hand is aligned to place the letter in the slot. If the slot is tilted, your hand tilts to match.

Patients with lesions of the parietal cortex cannot adjust their grip width or wrist angle using visual information alone, even though they can verbally describe the size of the object or the orientation of the slot. Conversely, patients with intact parietal lobes and deficits in the ventral stream cannot describe the size of an object or its orientation but can adjust their grip width and orient their hands as well as normal subjects can. The parietal lobe is a critical source of information about the visual properties of the targets of movement.

As described in Chapter 16, the representation of space in the parietal cortex is not organized into a single map like the retinotopic map in primary visual cortex. Instead it is divided into four areas that analyze the visual world in ways appropriate for individual motor systems and project to areas of premotor cortex that control individual movements (Figure 29–14).

The neural operations behind visually guided movements involve identifying targets, specifying their qualities, and ultimately generating a motor program to accomplish the movement. Neurons in the parietal cortex provide the visual information necessary for independent movements.

The anterior intraparietal cortex has neurons that signal the size, depth, and orientation of objects that can be grasped. Neurons in this area respond to stimuli that could be the targets for a grasping movement, and also respond when the animal makes the movement (Figure 29–15). Similarly, neurons in the medial intraparietal cortex represent the targets for reaching movements and project to the frontal area that generates the premotor signal for these movements.

The ventral intraparietal area has bimodal neurons that respond to both tactile and visual stimuli in the face (Figure 29–16). This area projects to the face area of premotor cortex. Finally, neurons in the lateral intraparietal area describe the targets for saccadic eye movements and project to the frontal eye field.

The visual cortex is interconnected by ventral and dorsal streams of neuronal processing. The ventral stream provides information about the nature of the objects represented; the dorsal stream provides information that the oculomotor and skeletal motor system can use for movement.

A critical function of the dorsal stream is to mediate attention, which is necessary in order to select objects in the visual field for further analysis. Neurons in the parietal lobe of the monkey respond more vigorously to attended objects than to unattended ones, and the activity of neurons in the monkey's parietal lobe mirrors the attention paid to certain objects in the environment.

Because visual information enters the brain through eyes that are constantly moving, the brain requires a mechanism to compensate for eye movements, constructing a stable visual world from successive fixations. Information supplied by the motor system through corollary discharges is used to adjust the receptive fields of visual neurons to compensate for intervening eye movements. The parietal lobe has a number of distinct representations of the visual field, each of which provides information to a specific motor subsystem.