Motor outputs are neural commands that act on the muscles, causing them to contract and generate movement. These outputs are derived from sensory inputs in circuits that represent sensorimotor transformations. Sensory inputs include extrinsic information about the state of world as well as intrinsic information about our body. Extrinsic information, for example the spatial location of a target, can be provided by auditory and visual inputs. Intrinsic information includes both kinematic and kinetic information about our body.
Kinematic information includes the position, velocity, and acceleration of the hand, joint angles, and lengths of muscles without reference to the forces that cause them. Kinetic information is concerned with the forces generated or experienced by our body. These different forms of intrinsic information are provided by different sensors. For example, information about muscle lengths and their rate of change is provided mainly by muscle spindles, whereas Golgi tendon organs in the muscles and mechanoreceptors in our skin provide information about the force we are exerting.
Simple reflexes, such as a tendon-jerk reflex, involve a simple sensorimotor transformation: Sensory inputs cause motor output directly without the intervention of higher brain centers. However, voluntary movement requires multistage sensorimotor transformations. The involvement of multiple processing centers actually simplifies processing: Higher levels plan more general goals, whereas lower levels concern themselves with how these goals can be implemented.
Such a hierarchy accounts for the fact that a specific motor action, such as writing, can be performed in different ways with more or less the same result. Handwriting is structurally similar regardless of the size of the letters or the limb or body segment used to produce it (Figure 33–1). This phenomenon, termed motor equivalence, suggests that purposeful movements are represented in the brain abstractly rather than as sets of specific joint motions or muscle contractions. Such abstract representations of movement, able to drive different effectors, provide a degree of flexibility of action not practical with preset motor programs.
The ability of different motor systems to achieve the same behavior is called motor equivalence. For example, writing can be performed using different parts of the body. The examples here were written by the same person using the right (dominant) hand (A), the right hand with the wrist immobilized (B), the left hand (C), the pen gripped between the teeth (D), and the pen attached to the foot (E). (Reproduced, with permission, from Raibert 1977.)
How do sensorimotor transformations generate movement to a desired location? For a person to reach toward an object, sensory information about the target's location must be converted into a sequence of muscle actions leading to joint rotations that will bring the hand to the target.
First, the target is localized in space relative to some part of the body such as the head or arm (egocentric space). Several sources of information are combined in this process. For example, the location of the target relative to the head is computed from the location of the target on each retina together with the direction of gaze of the eyes (Figure 33–2A). A person also needs to know the initial location of his hand or the tip of the tool that he wishes to place on the target (the end-effector or endpoint). The initial location of the endpoint can be estimated by combining visual inputs, proprioceptive signals, and tactile sensations, each of which can provide location information. Once the current configuration of the arm and location of the target are calculated, a movement can be planned. A plan typically has to specify both a particular path, the successive spatial positions of the endpoint, and also the trajectory, the time course over which these positions will be covered, and thus the accelerations and speeds of the movement (Figure 33–2B).
Sensorimotor transformations used to generate a particular movement.
The task of generating a goal-directed movement is often broken down into a set of sequential stages, the details of which are still being elucidated. The figure shows one possible set of stages to generate a reaching movement, and the arrows indicate the processes required to move between the stages.
A. Spatial orientation. To reach for an object, the object and hand are first located visually in a coordinate system relative to the head (egocentric coordinates).
B. Movement planning. The direction and distance the hand must move to reach the object (the endpoint trajectory) are determined based on visual and proprioceptive information about the current locations of the arm and object.
C. Inverse kinematic transformation. The joint trajectories that will achieve the hand path are determined. The transformation from a desired hand movement to the joint trajectory depends on the kinematic properties of the arm, such as the lengths of the arm's segments.
D. Inverse dynamic transformation. The joint torques or muscle activities that are necessary to achieve the desired joint trajectories are determined. The joint torques required to achieve a desired change in joint angles depend on the dynamic properties of the arm such as the mass of the segments.
In a hierarchical model of planning the goal can be expressed in kinematic terms, such as the desired positions and velocities of the hand, or in kinetic terms, such as the force exerted by the hand. Movement can be planned as an endpoint trajectory, a desired change in the configuration of the limb expressed in coordinates intrinsic to the limb. Such a coordinate system could determine the change in joint angles or be based on a desired change in proprioceptive feedback. For example, the endpoint trajectory could be defined kinematically as the distance and direction the hand has to move to reach the target, as well as the speed along the path to the target.
Transformations can be expressed as changes in kinematic variables, such as the position of the hand and the joint angles that place the hand at that position. The calculation of an endpoint from a set of joint angles is termed forward kinematic transformation, whereas calculation of a set of joint angles that can reach an endpoint is termed inverse kinematic transformation (Figure 33–2C). This transformation must take into account the geometric parameters of the arm, such as the lengths of the upper arm and forearm (recall that kinematics considers motion without reference to the forces that cause it). The motor system controls joint angle by activating muscles that produce torques (rotational forces) at the joint.
The action of motor commands on muscles that results in a set of angular positions and velocities is known as the forward dynamic transformation. The term "dynamic" refers to the forces required to cause motion. However, to generate a desired joint angle trajectory the system must convert kinematic parameters into motor commands. That is, the system must calculate the torques at each joint necessary to achieve the motion and relate the force required to cause this motion to the desired acceleration of the limb. This transformation is known as the inverse dynamic transformation (Figure 33–2D). In general, to cause any acceleration the forces applied must exceed any resistive forces arising from the viscosity or stiffness of the limb, from gravity, and from external loads. The force not required to overcome the total resistive force will cause an angular acceleration, with the acceleration being dependant on the limb's inertia; the lower the inertia, the higher the acceleration.
Thus through a series of sensorimotor transformations, sensory input is finally converted into muscle contractions that generate movement. Although we have described one possible series of transformations that can achieve a movement, the actual computations used by the central nervous system are still under active investigation.
The Central Nervous System Forms Internal Models of Sensorimotor Transformations
We know from cellular studies that the central nervous system contains internal representations ("neural maps") of the various sensory receptor arrays and the musculature. Experimental and modeling studies strongly suggest that the central nervous system also maintains internal representations that relate motor commands to the sensory signals expected as a result of movement.
Given the fixed lengths of our limb segments, there is a mathematical relationship between the joint angles of the arm and the location of the hand in space. A neural representation of this relationship allows the central nervous system to estimate hand position if it knows the joint angles and segment lengths. The neural circuits that compute such sensorimotor transformations are examples of internal models (Box 33–1). Such neural representations may not exactly match true relationships because of structural differences (the models only approximate the true relationship between joint angles and hand position) or errors in the model's parameters (incorrect estimates of segment lengths).
Box 33–1 Internal Models
The utility of numerical models in the physical sciences has a long history. Numerical models are abstract quantitative representations of complex physical systems. Some start with equations and parameters that represent initial conditions and run forward, either in time or space, to generate physical variables at some future state. For example, we can construct a model of the weather that predicts wind speed and temperature two weeks from now. In general, the algorithms and parameters of the model should lead to one correct answer.
Other models start with a state, a set of physical variables with specific values, and operate in the inverse direction to determine what parameters in the system account for that state. When we fit a straight line to a set of data points, we are constructing an inverse model that estimates slope and intercept based on the equations of the system being linear. An inverse model may thus inform us how to set the parameters of the system to obtain desired outcomes.
Over the last 50 years the idea that the nervous system has similar predictive models of the physical world to guide behavior has become a major issue in neuroscience. The idea originated in Kenneth Craik's notion of internal models for cognitive function. In his 1943 book, The Nature of Explanation, Craik was perhaps the first to suggest that organisms make use of internal representations of the external world:
"If the organism carries a 'small-scale model' of external reality and of its own possible actions within its head, it is able to try out various alternatives, conclude which is the best of them, react to future situations before they arise, utilize the knowledge of past events in dealing with the present and future, and in every way to react in a much fuller, safer, and more competent manner to the emergencies which face it."
In this view an internal model allows an organism to contemplate the consequences of current actions without actually committing itself to those actions.
Considering the human body from the viewpoint of sensorimotor control, we should ask two fundamental questions. First, how can we generate actions on the system so as to control its behavior? Second, how can we predict the consequences of our actions?
The central nervous system must exercise both control and prediction to achieve skilled motor performance. Prediction and control are two sides of the same coin, and the two processes map exactly onto forward and inverse models. Prediction turns motor commands into expected sensory consequences, whereas control turns desired sensory consequences into motor commands.
An internal model that represents the causal relationship between actions and their consequences is called a forward model because it estimates future sensory inputs based on motor outputs. A forward model anticipates how the motor system's state will change as the result of a motor command. Thus, a copy of a descending motor command acting on the sensorimotor system is passed into a forward model that acts as a neural simulator of the musculoskeletal system moving in the environment. This copy of the motor command is known as an efference copy (or corollary discharge) to signify that it is a copy of the efferent signal flowing from the central nervous system to the muscles. We will see later how such simulations can be learned and used in sensorimotor control.
An internal model that calculates motor outputs from sensory inputs is known as an inverse model. Such a model can determine what motor commands are needed to produce the particular movements necessary to achieve a desired sensory consequence.
Forward and inverse models can be better understood if we place the two in series. If the structure and parameter values of each model are correct, the output of the forward model (the predicted behavior) will be the same as the input to the inverse model (the desired behavior) (Figure 33–3).
Internal models represent relationships of the body and external world.
The inverse model determines the motor commands that will produce a behavioral goal, such as raising the arm while holding a ball. A descending motor command acts on the musculoskeletal system to produce the movement. A copy of the motor command is passed to a forward model that simulates the interaction of the motor system and the world and can therefore predict behaviors. If both forward and inverse models are accurate, the output of the forward model (the predicted behavior) will be the same as the input to the inverse model (the desired behavior).
Movement Inaccuracies Arise from Errors and Variability in the Transformations
Motor control is often imprecise. Indeed, society celebrates those who can throw a dart into a small area of a board or hit a small white ball into a hole with a club. However, even the movements of the most skilled players show some degree of variability. In the 1890s the psychologist Robert Woodworth showed that fast movements are less accurate than slow ones. People slow their movements when accuracy is demanded. Inaccuracy can arise either from variability in the sensory inputs and motor outputs or from errors in the internal representations of this information.
An important component of sensorimotor variability is the intrinsic variability of our sensors and motor neurons because of fluctuations in their membrane potential. Because of these fluctuations, known as neural noise, the level of input signals required to trigger postsynaptic action potentials also varies. On the input side, neural noise limits the accuracy of estimates of the location of a target or limb (how near an estimate is to the true value) as well as their precision (how accurate the estimate is when repeated). On the output side, neural noise limits the accuracy and precision with which we contract our muscles. Moreover, the amount of noise in motor commands tends to increase with larger motor commands, limiting our ability to move rapidly and accurately at the same time. This increase in variability is caused by random variation in both the excitability of motor neurons and the recruitment of the additional motor units needed to produce increases in force.
Incremental increases in force are produced by progressively smaller sets of motor neurons, each of which produces disproportionately greater increments of force (see Chapter 34). Therefore, as force increases, fluctuations in the number of motor neurons lead to greater fluctuations in force. The consequences of this can be observed experimentally by asking subjects to generate a constant force or a force pulse of fixed amplitude. Not only are subjects unable to maintain constant force, but the variability of force also increases with the level of the force. Over a large range this increase in variability is captured by a constant coefficient of variation (the standard deviation divided by the mean force). This dependence of variability on force corresponds to the increase in the variability of pointing movements with the average speed of movement. The decrease in accuracy of movement with increasing speed is known as the speed-accuracy trade-off (Figure 33–4).
Accuracy of movement varies in direct proportion to its speed.
Subjects held a stylus and were required to try and hit a target line lying perpendicular to the direction in which they moved. Each subject started from three different initial positions and was required to complete the movement within three different times (140, 170, or 200 ms). A successful trial was one in which the subject completed the movement within 10% of the required time. Only successful trials were used for analysis. Subjects were informed if their movements were more than 10% different from the required duration. The plot shows the variability in the motion of the subjects' arm movements as the standard deviation of the extent of movement versus average speed (for each of three movement starting points and three movement times, giving nine data points). The variability in movement increases in proportion to the speed and therefore to the force producing the movement. (Reproduced, with permission, from Schmidt et al. 1979.)
Errors can also arise from inaccuracy in the internal models that compute sensorimotor transformations. Neural representations of the musculature cannot easily capture the complex biomechanical properties of the musculoskeletal system, and this in turn significantly complicates the ability of the brain to compute accurate sensorimotor transformations. Indeed, the dependence of muscle force on the motor command is itself highly complex. A model prescribing motion in a system with just a single joint must not only estimate the muscle force (or torque) but also take into account inertia (the mass resisting acceleration), viscosity (resistive forces proportional to velocity), stiffness (elastic forces proportional to displacement) produced by the muscles and tendons opposing movement, and gravity.
The dynamic relationship between segments of limbs further complicates sensorimotor transformations. The motion of each segment produces torques, and potentially motions, at all other segments through mechanical interactions. For example, flexion of the upper arm through shoulder rotation can lead to either extension or flexion of the elbow depending on the initial elbow angle. In general, because of the interactions between linked segments, the torques needed to produce a specific change in angle at a particular joint depend not only on the muscles acting directly at this joint but also on the configurations and the motions of all other joints, and especially their acceleration. The brain develops an internal model of these complex mechanical interactions through learning early in childhood. We will see later that this learning is updated throughout life and depends critically on proprioception, which provides the brain with information about changes in muscle length and joint angles.
Different Coordinate Systems May Be Employed at Different Stages of Sensorimotor Transformations
Different coordinate systems are used in sensorimotor transformations and are encoded in several brain regions. Coordinate systems are either extrinsic or intrinsic to the body. Extrinsic coordinate systems relate objects in the outside world to other objects (allocentric coordinates) or to our body (egocentric coordinates) using exteroceptive information, usually visual or auditory (Figures 33–5A and B). Intrinsic coordinate systems, such as the set of muscle lengths or set of joint angles (Figure 33–5C), are based on information provided primarily by proprioceptive systems.
The location of the finger in space can be specified in different egocentric coordinate systems.
A. Cartesian coordinates centered on the eyes.
B. Spherical polar coordinates centered on the shoulder (distance r, azimuth φ, and elevation Θ).
C. An intrinsic coordinate system based on shoulder angles (α1 and α2) , which relate the orientation of the upper arm to the Cartesian axes, and elbow angle (α3) , which specifies the angle between the upper and lower arm.
Elucidating the coordinate systems used in sensorimotor transformations is a major endeavor in neuroscience. We will see in later chapters that this issue can be fruitfully studied by examining how the firing patterns of neurons in different parts of the brain encode task features or movement parameters. Such studies aim to determine the variables (such as position or velocity) or type of coordinate system (such as allocentric or egocentric) that the neurons encode.
Behavioral studies also have used a variety of methods to examine the coordinate systems used in directing movement. One way has been to examine the details of the errors made during movement in different tasks. When subjects are asked to reach rapidly and repeatedly to a target, the error in the movements can be measured in different ways. If we average the final location of the hand across many trials, we may find a constant error or bias in the movement. We can examine the distribution of the final locations of the hand about this average position and infer from the patterns of constant and variable error the coordinate system used in the movement (Box 33–2).
Box 33–2 The Brain's Choice of Spatial Coordinate System Depends on the Task
When subjects are shown a visual target and asked to reach for it repeatedly, the pattern of errors they make varies with the circumstance of the task. By examining these errors it is possible to assess which coordinate system is used to represent the target position under different conditions.
For example, when subjects are required to move their hands on a horizontal surface and can estimate the starting position of their hands before movement, the pattern of errors indicates planning in "hand-centered" coordinates. The distributions of the endpoints of the movements demonstrate that, under the conditions of the task, errors in distance and direction are independent of each other (Figure 33–6) and thus that errors in the extent of a movement cannot be predicted from errors in direction. The independence of the two types of errors suggests that for this type of task subjects estimate distance and direction relative to a specific starting location (that is a movement vector) in Cartesian coordinates.
Conversely, when subjects make large three- dimensional movements to remembered visual targets in the dark, a different pattern of error is observed. When the target and finger locations at the end of the reach are plotted against each other in terms of spherical coordinates centered on the shoulder, angular errors (elevation and azimuth) are small, whereas errors in the radial extent of the movement are significant (Figure 33–7). Moreover, the two types of errors are not correlated. However, if the target and finger locations are plotted in terms of spherical coordinates centered on the head, the errors are correlated.
The fact that the spherical coordinate system centered on the shoulder produces uncorrelated errors suggests that at some stage in the sensorimotor transformation the target is represented in shoulder-centered coordinates. Recent work suggests that this pattern of errors reflects planning for a final hand position rather than a particular hand trajectory.
Errors in distance and direction of movement are independent of each other.
Distribution of endpoints for reaching movements to 16 targets (eight directions and two distances) by one subject. Targets (red circles) were presented 24 times in random order, and each time the subject was asked to place a finger on the target. All movements begin from a central starting position (designated +). Endpoints for individual movements are represented by blue dots. The endpoints for the reaches to each target are fitted with an ellipse, demonstrating that errors in distance and direction are independent of each other. (Adapted, with permission, from Gordon, Ghilardi, and Ghez 1994.)
Distance errors are greater than direction errors for movements in the dark.
A subject was asked to place a finger on the remembered location of a target in the dark. Final finger position and target location are plotted in spherical coordinates centered on the shoulder (see Figure 33–5B). The straight line represents perfect performance and the dots individual reaching movements. Radial distance errors and angular errors (azimuth and elevation) are plotted separately. (Adapted, with permission, from Soechting and Flanders 1989.)
Stereotypical Patterns Are Employed in Many Movements
Given a task, motor plans are underconstrained. For example, the hand can move to a target along an infinite number of possible paths, and for each path there is an infinite number of trajectories. Having specified the path and velocity, each point along the path could be reached by any number of combinations of arm joint angles and, owing to the overlapping actions of muscles and the ability to co-contract, each arm configuration could be achieved by many different combinations of muscles.
Although we have described different types of sensorimotor transformations, in general the inverse transformations cannot be uniquely specified. For example, the inverse kinematic transformation that transforms hand positions into joint angles can have many outputs based on the same input. This is because many different combinations of joint angles will put the hand in the same place. The ability of the motor systems to achieve a task in many different ways is called redundancy. If one way of achieving a task is not practical, there is usually an alternative.
Some of the earliest studies of movement examined how the brain determines the duration of a movement. Fitts's law describes the relationship between the amplitude, accuracy, and duration of a movement. This law relates the duration of a movement to the accuracy required of the task, as determined by the target width and amplitude of the movement, and applies to a variety of tasks such as reaching, placing pegs in holes, and picking up objects (Figure 33–8).
Fitts's law describes the speed-accuracy trade-off.
Subjects were required to move a stylus between two targets of width W separated by distance A. The width of the targets was changed on each trial. Each line in the plot represents the results for a different target width (from narrow to wide) over four different movement amplitudes. Subjects were required to move as fast as possible while still hitting the targets. Over a large range of target widths and distance between targets, movement duration is linearly dependent on log2 (2A/W), the index of difficulty. (Adapted, with permission, from Jeannerod 1988.)
Despite variations in movement direction, speed, and location, several aspects of reaching movements are stereotypical or invariant. First, the hand tends to follow roughly a straight path (Figure 33–9A), although significant curvature is observed for certain movements, particularly vertical movements and movements near the boundaries of the reachable space. The tendency to make straight-line movements characterizes a large class of movements and is surprising given that the muscles act to rotate joints. Second, a plot of hand speed over time is typically smooth, unimodal, and roughly symmetric (bell-shaped) (Figure 33–9B). This is not the case when movement accuracy requirements are high or corrections to the movement are made.
Hand path and velocity have stereotypical features.
(Adapted, with permission, from Morasso 1981.)
A. The subject sits in front of a semicircular plate and grasps the handle of a two-jointed apparatus that moves in one plane and records hand position. The subject is instructed to move the hand between various targets (T1–T6). The record on the right shows the paths traced by one subject.
B. Kinematic data for three hand paths shown in part A (c, d, and e). All paths are roughly straight, and all hand speed profiles have the same shape and scale in proportion to the distance covered. In contrast, the profiles for the angular velocity of the elbow and shoulder for the three hand paths differ. The straight hand paths and common profiles for speed suggest that planning is done with reference to the hand because these parameters can be linearly scaled. Planning with reference to joints would require computing nonlinear combinations of joint angles.
In contrast, the motions of the joints in series (such as the shoulder, elbow, and wrist) are complicated and vary greatly with different initial and final positions. Because rotation at a single joint would produce an arc at the hand, both elbow and shoulder joints must be rotated concurrently to produce a straight path. In some directions the elbow moves more than the shoulder; in others, the reverse occurs. When the hand is moved from one side of the body to the other (see Figure 33–9A, movement from T2 to T5) one or both joints may have to reverse direction in midcourse. The fact that hand trajectories are more invariant than joint trajectories suggests that the motor system typically controls the hand by adjusting joint rotations and torques to achieve desired hand trajectories.
Invariances can also be seen in more complex movements. The nervous system puts together complex actions from elemental movements that have highly stereotyped spatial and temporal characteristics. For example, the seemingly continuous motion of drawing a figure eight actually consists of several discrete movements that are roughly constant in duration, regardless of their size. Moreover, there is a relationship between the curvature of each elemental movement and speed: Subjects tend to slow the hand as the curvature of the path increases. Empirical studies have shown that for many tasks a power law relation, the two-thirds power law, governs the relationship between hand speed and path curvature (Figure 33–10).
Complex movements obey the two-thirds power law.
(Reproduced, with permission, from Lacquaniti, Terzuolo, and Viviani 1983.)
A. Nondirected scribbling is a complex movement.
B. Instantaneous values of the angular velocity of the hand while scribbling are plotted against the curvature of the hand's path raised to the power of two-thirds. The relation between the two variables is piecewise linear. Each segment in the bundle, corresponding to nonoverlapping segments of the trajectory, has a different slope. The slopes cluster around the average (dashed line), a typical result for this type of experiment. Therefore the relationship between the speed of hand motion and the degree of curvature of the hand path is roughly constant: Velocity varies as a continuous function of the curvature raised to the power of two-thirds. This two-thirds power law governs virtually all movements and expresses an obligatory slowing of the hand during movement segments that are more curved and a speeding up during segments that are straight. Because angular velocity is the speed of the hand multiplied by the path curvature, in the plot an increase in angular velocity represents a decrease in hand speed.
The simple spatiotemporal elements of a complex movement are called movement primitives or movement schemas. Like the simple lines, ovals, or squares in computer graphics programs, movement schemas can be scaled in size or time. The neural representations of complex actions, such as prehension, writing, typing, or drawing, are thought to be stored sets of these simple spatiotemporal elements.
Recent computational studies of a variety of tasks suggest that a repertory of movement schemas is the result of a process in which all possible ways of moving are ranked and the best is selected. This idea implies that either through evolution or motor learning our movements improve progressively until some limit is reached.
To quantify how good or bad a movement the brain assigns a cost to each possible movement, and the movement with the lowest cost is executed. The cost is specified as some function of the movement and task, and the challenge to researchers has been to determine, from observed movement patterns and perturbation studies, the form of this function.
The cost may be kinematic; for example, lack of smoothness in a movement can be corrected by minimizing the rate of change of hand acceleration summed over a movement. Alternatively, the cost may be dynamic. For example, because the variability in the motor output is proportional to the magnitude of the motor command, repetition of the same sequence of intended motor commands many times will lead to a distribution of actual movements. Modifying the sequence of motor commands can control aspects of this distribution, such as the spread of positions of the hand at the end of the movement. In a simple aiming movement the cost is the final error, as measured by the variability about the target. A model that minimizes this cost would accurately predict the trajectories of both eye and arm movements.