Close
About
FAQ
Home
Collections
Login
USC Login
Register
0
Selected
Invert selection
Deselect all
Deselect all
Click here to refresh results
Click here to refresh results
USC
/
Digital Library
/
University of Southern California Dissertations and Theses
/
Modeling motor memory to enhance multiple task learning
(USC Thesis Other)
Modeling motor memory to enhance multiple task learning
PDF
Download
Share
Open document
Flip pages
Contact Us
Contact Us
Copy asset link
Request this asset
Transcript (if available)
Content
MODELING MOTOR MEMORY TO ENHANCE MULTIPLE TASK LEARNING
by
Jeong-Yoon Lee
________________________________________________________________________
A Dissertation Presented to the
FACULTY OF THE USC GRADUATESCHOOL
UNIVERSITY OF SOUTHERN CALIFORNIA
In Partial Fulfillment of the
Requirements for the Degree
DOCTOR OF PHILOSOPHY
(COMPUTER SCIENCE)
May 2011
Copyright 2011 Jeong-Yoon Lee
ii
Acknowledgements
First of all, I would like to thank Dr. Nicolas Schweighofer for being the best
advisor with his endless support, insight, and patience. I am also thankful to my
dissertation committee members Drs. Stefan Schaal, Terence Sanger, and Carolee
Winstein as well as my qualification exam committee member Dr. Michael Arbib for
offering valuable feedback on this work. I would also like to thank Dr. James Gordon for
his insightful comments through my Ph.D, Dr. Robert Scheidt for the collaboration at
Marquette University, and Dr. Hiroyuki Nakahara for hosting me to RIKEN Brain
Science Institute.
I feel thankful to all of my fellows at University of Southern California for their
moral support and encouragement: Cheol Han, Younggeun Choi, Feng Qi, Yukikazu
Hidaka, Sungshin Kim, Yupeng Xiao, Amarpreet Bains, Hyeshin Park, and Sujin Park in
Computational Neuro-Rehabilitation and Learning Lab; Shailesh Kantak, Hui-Ting Goh,
Jill Steward, Shuya Chen, Charalambos Charalambous, Ya-Yun Lee, Matt Konersman,
and Eric Wade in Motor Behavior and Neuro-Rehabilitation Lab; and Junkwan Lee,
Jinyong Lee, and Tong Sheng.
Last but not least, I would like to thank my family, especially my wife, Young Ok
Lee for supporting me through the toughest times with love.
iii
Table of Contents
Chapter 1. Introduction
1.1. Optimal multi-task motor learning
1.2. Organization of the dissertation
Chapter 2. Background: computational models of motor learning
2.1. Introduction
2.2. Computational neuroscience
2.3. Motor learning
2.3.1. Why motor learning
2.3.2. What is motor learning?
2.3.3. Motor adaptations
2.4. Computational models of motor learning
2.4.1. Computational framework for the motor system
2.4.2. Models of motor learning
Chapter 3. Dual adaptation supports a parallel architecture of motor memory
3.1. Introduction
3.2. Methods
3.2.1. Experimental procedure
3.2.2. Candidate models
3.2.3. Simulation parameters
3.2.4. Model parameter fitting
3.2.5. Model comparison
3.3. Results
3.3.1. Simulation of spontaneous recovery supports fast and slow
timescales
3.3.2. Simulation of anterograde interferences supports a contextual-
independent process
3.3.3. Simulation of dual-adaptations supports a context-dependent
slow process
3.3.4. Dual-adaptation experiment supports the parallel 1-fast n-slow
model
3.3.5. Comparison with time-varying parameter model in savings in
relearning experiment
3.4. Discussion
Chapter 4. Mechanisms of the contextual interference effects in individuals post
stroke
4.1. Introduction
4.2. Methods
1
1
2
4
4
5
7
7
7
8
9
9
15
22
22
24
24
26
33
35
37
38
38
39
40
43
45
46
51
51
53
iv
4.2.1. Participants
4.2.2. Learning tasks
4.2.3. Short-term memory tests
4.2.4. Data analysis
4.2.5. Computational model
4.3. Results
4.3.1. The CI effect in the computational model
4.3.2. The CI effect in healthy individuals
4.3.3. The CI effect in individuals post stroke
4.4. Discussion
Chapter 5. Optimal schedule in multi-task motor learning
5.1. Introduction
5.2. Methods
5.2.1. Models of multi-task motor learning
5.2.2. Optimal schedules in multi-task motor learning
5.2.3. Simulation
5.2.4. Experiment
5.3. Results
5.3.1. Simulation results
5.3.2. Experiment results
5.4. Discussion
Chapter 6. Conclusion: In search of optimal motor learning
6.1. Summary
6.2. Future work
6.2.1. Neural substrates of multi-task motor learning
6.2.2. Effects of randomness in practice schedules on motor learning
6.2.3. Adaptive optimal schedules in multi-task motor learning
Bibliography
Appendices
A. Supplementary materials for Chapter 2
B. Pontryagin’s maximum principle
53
55
59
60
61
63
63
67
70
74
80
80
82
82
84
88
89
92
92
94
95
110
110
113
113
114
116
118
127
127
139
v
List of Tables
Table 2.1. An apparent divide between the approaches to motor behavior taken
by motor control physiologists versus that taken by clinical neuropsychologists
Table 4.1. Six Characteristics of the healthy individuals
Table 4.2. Characteristics of the individuals post-stroke
14
54
55
vi
List of Figures
Figure 2.1. A schematic model for generating goal-directed movements
Figure 3.1. Ten possible motor adaptation models
Figure 3.2. The two possible architectures of the 1-fast n-slow model
Figure 3.3. Simulation of spontaneous recovery for all ten models considered
Figure 3.4. Simulation of anterograde interference for the eight remaining
candidate models
Figure 3.5. Simulations of two dual-adaptation experiments for the remaining
six models considered
Figure 3.6. Average performance data across subjects during learning
Figure 3.7. Simulation of the wash-out paradigm with the two-state model, the
varying parameter model, and the parallel 1-fast n-slow model
Figure 4.1. Motor learning tasks
Figure 4.2. Examples of force trajectories for four subjects for 1 task
Figure 4.3. Computer simulations: CI effect in the “ healthy‟ model
Figure 4.4. Computer simulations: reduced CI effect in a model with “ poor
short-term memory‟
Figure 4.5. Computer simulations: Forgetting as a function of the time constant
of decay of short-term memory after either blocked (A) or random (B)
schedule for two tasks
Figure 4.6. Data: CI effect in healthy participants
Figure 4.7. Data: CI effect in participants post-stroke
Figure 4.8. Data: Forgetting in individuals post-stroke in a 24 hour post-
training period as a function of Wechsler visual memory score (figural)
following training in either blocked (A) or random schedule (B)
11
26
28
31
32
33
42
44
57
59
64
66
67
69
71
73
vii
Figure 5.1. Supporting simulations showing how long-term retention is related
to task switching in the 1-fast N-slow model
Figure 5.2. Models of multi-task motor learning
Figure 5.3. Apparatus for the optimal schedule experiment
Figure 5.4. Three visuomotor transformations
Figure 5.5. Schedules of each group for the optimal schedule experiment
Figure 5.6. Switching probabilities (A) and ratio between numbers of trials for
task 1 and task 2 (B) of optimal schedules predicted by the multi-task motor
learning models
Figure 5.7. Optimal schedules in the dual-task learning paradigm predicted by
the multi-task motor learning models
Figure 5.8. Optimal schedules in the dual-task learning of gain and rotation
(A), and gain and shearing (B) predicted by the 1-fast N-slow model
Figure 5.9. The numbers of trials required for the optimal (green) and
alternating (blue) schedules to reach to delayed retention performance levels in
dual-adaptation of gain and shearing
Figure 5.10. Practice performance of the parameter estimation groups for the
(top) gain, (middle) rotation, and (bottom) shearing tasks
Figure 5.11. Generalization test performance of parameter estimation groups
for the (left) gain, (middle) rotation, and (right) shearing tasks
Figure 5.12. Errors of individual subjects in the alternating schedule group for
the (top) gain and rotation tasks and (bottom) gain and shearing tasks
Figure A.1. Ranges of parameters of the parallel 1-fast N-slow model
Figure A.2. Ranges of parameters of the parallel 1-fast N-slow model to
reproduce simultaneous recovery
Figure A.3. Ranges of parameters of the parallel 1-fast N-slow model to
reproduce anterograde interference
Figure A.4. Ranges of parameters of the parallel 1-fast N-slow model to
reproduce dual-adaptation
98
99
100
101
102
103
104
105
106
107
108
109
132
133
134
135
viii
Figure A.5. Ranges of parameters of the parallel 1-fast N-slow model to
reproduce savings in wash-out paradigm
Figure A.6. Ranges of parameters of the parallel 1-fast N-slow model to
reproduce all four experimental data
136
137
ix
Abstract
Although recent computational modeling research has advanced our
understanding of motor learning, previous studies focused on single-task motor learning
and did not account for multiple task motor learning which is the norm in sports, music,
professional skill development, and neuro-rehabilitation.
In this dissertation, we took the combined approach of theoretical analysis,
computational modeling, and behavioral experiments to understand the mechanisms of
multi-task motor learning, and based on this understanding, to optimize multi-task motor
learning.
We first suggested a parallel architecture of motor memory in multi-task motor
learning: By examining systematically how possible architectures account for
experimental results, we showed that the human brain engages a fast-learning-fast-
forgetting learning process in parallel with multiple slow-learning-slow-forgetting
learning processes.
We then investigated how practice schedules and the integrity of short-term
memory affect long-term learning: Based on our model, we found that for healthy
individuals with intact short-term memory, random practices schedule lead to better long-
term learning than blocked practice schedules. However for individuals post-stroke with
deficits in short-term memory, the effect of practice schedules in long-term learning was
mitigated.
We finally derived optimal schedules for multi-task motor learning by applying
optimal control theory to our computational model of multi-task motor learning. We
x
found that alternating schedules are optimal only if tasks have equal difficulties. If
differences in difficulties between tasks increase, our algorithms provide optimal
schedules that have the potential to enhance long-term learning in multi-task motor
learning.
１
Chapter 1
Introduction
1.1. Optimal multi-task motor learning
Life is a continuum of motor learning: Through our lives, we keep learning new
movement skills (ex. Walking for toddlers) and adapting those skills into different
environments (ex. Walking on the hard floor vs. on the white sand beach) or changes of
bodies (ex. Getting taller and gaining weights). Sometimes, we also try to regain skills
lost either by natural forgetting along the passage of time, or by movement disorders
followed by stroke or other diseases like Parkinson’s disease.
Nevertheless, we are far from being an expert in motor learning in terms of
maximizing long-term retention performance: During active learning, where people have
to choose a motor task to practice each time among multiple tasks to learn, their choices
are mostly suboptimal (Huang et al., 2008). Also, after stroke, many patients learn non-
use of their affected arms: Patients give up using their affected arms, even when they can
still perform tasks with the affected arms to certain degrees and the impaired performance
of those arms can be recovered by practice (Sunderland and Tuke, 2005).
Then, how can we learn motor tasks in the optimal way with limited time and
efforts? It is important to answer this question for children development, sport and music
education, fitness, and rehabilitation.
In this dissertation, we aim to find the optimal method for multi-task motor learning.
For this purpose, we first investigate the mechanisms of multi-task motor learning in
２
human using a combined computational, behavioral and neuroimaging approach. Then,
we investigate the methods to maximize long-term retention performance based on the
computational model of multi-task motor learning.
1.2. Organization of the dissertation
In Chapter 2, as a background of the other studies, we provided reviews about
contemporary computational models of motor learning. In Chapter 3, we first
investigated the architecture of human motor memory by systematically testing possible
architectures via a combination of simulations and a dual visuomotor adaptation
experimental paradigm. We found that only one parsimonious model can account for
both previous motor adaptation data and our dual-task adaptation data: a fast-learning-
fast-forgetting process that contains a single state is arranged in parallel with a slow-
learning-slow-forgetting process that contains multiple states switched via contextual
cues. For further investigation on neural substrates of the model, we proposed the fMRI
experiment with virtual TMS lesions of the fast process.
Then, in Chapter 4, we compared the effects of the simultaneous practices of tasks
(random schedule) vs. sequential practices of tasks (massed schedule) on multi-task
motor learning both during training and at a 24-hour-delayed retention test. Grip force
modulation experiments confirmed generalization and contextual interference effects
between tasks in both schedules. We explained different effects of two schedules using a
novel multi-timescale motor primitive model, and further showed how short-term
３
memory deficit in patients after stroke led poor performance at the delayed retention test
in the random schedule.
Previous studies also showed that among simultaneous presentations of tasks, the
random schedule led superior performance to a simple alternating schedule. In Chapter 4,
we presented a preliminary study, which investigated the differences between the
alternating and random schedules using a hybrid schedule of both schedules in the dual-
visuomotor adaptation experimental paradigm. We showed that in the random schedule,
the resistance to interference between tasks was developed, but in the following
alternating schedule, that resistance was washed out. We discussed about possible
explanations of data based on uncertainty and proposed a new experiment for further
investigation.
In Chapter 5, we showed to find the optimal practice schedule predictively (i.e.
before information about long-term retention is available) in multi-task motor learning
based on computational models of motor memory and optimal control theory:
Computational models will allow us to predict delayed long-term retention performance
based on current performance and given practice schedules. With the optimal control
theory, we can find the optimal practice schedule, which maximizes such predicted
retention performance. The optimal practice schedule will be validated by comparing it
with the random schedule in behavioral experiments.
Summary of the dissertation and future work follow in Chapter 6.
４
Chapter 2.
Background: Computational models of motor learning
2.1. Introduction
In this chapter, we want to provide backgrounds for the rest of chapters.
Throughout the dissertation, we investigated computational models of multi-task motor
learning in humans. In other words, we studied how motor systems of humans learn, with
a focus on the cases that there are multiple tasks to learn, using computational approaches
of modeling. Here we first introduced computational neuroscience: what it does; why it is
useful, and what are its examples. Then, we talked about motor learning: why we study
motor systems; what motor learning is, and the computational models of motor learning.
Recently, computational models of motor learning have successfully explained a
wide range of data in motor learning such as savings, spontaneous recovery, interferences,
generalizations, and consolidation (Thoroughman and Shadmehr, 2000; Smith et al., 2006;
Kording et al., 2007; Ethier et al., 2008; Zarahn et al., 2008; Lee and Schweighofer,
2009).
However, to neurophysiologists, behavioral neuroscientists, and even to some of
computational neuroscientists, it still seems unclear how these models are related to the
brain because these models often neither account for the effect of physiological factors
such as muscle mechanics, limb dynamics etc., nor define what the states represent
neurophysiologically.
５
Here we introduce contemporary computational models of motor learning to
neuroscience communities with various backgrounds: Where they come from, how they
are validated, how they can be beneficial for neuroscience researches.
2.2. Computational neuroscience
Recently, computational neuroscience is flourishing day by day. Not long after
Nature Neuroscience devoted a special issue to computational neuroscience to change its
bad reputation among experimentalists (2005), the number of computational neuroscience
papers listed in PubMed has nearly doubled and Nature Neuroscience devoted another
special issue as a celebration of the field that has come into its own (2011). In this section,
we introduce this rising field briefly to provide background for studies that follow later.
What does computational neuroscience? One of the best and most quoted answers
to this question might be David Marr’s. In his seminal work on vision (Marr, 1982), Marr
described three levels of analysis on the brain: computational level, which clarifies what
the system does and why it does these things; algorithmic level, which describes how the
system does what it does, what representations it uses and what processes it employs to
build the representations; implementational level, which discusses how the system
physically realized. In other words, computational neuroscience explains what the brain
does and how it might do in the normative viewpoints.
Why is computational neuroscience useful? Recently, in his article of reviewing the
rise of theoretical neuroscience over the past years in Neuron (Abbott, 2008), Abbott
summarized the contributions of theoretical neuroscience as follows: First, by expressing
６
conceptual models (or word models) as mathematical equations, it forces models to be
precise, complete and self-consistent. Second, it enables to formulate, explore, and reject
models at a pace that no experimental program can match.
Other than these, computational neuroscience allows that neuroscientific findings
can be implemented in forms that help humans, such as neural prosthesis (see (Green and
Kalaska, 2011) for a recent review) and robots (Schaal and Schweighofer, 2005). Also it
encourages crosstalk among sub-disciplines of neuroscience, ranging from the molecular
level to psychology, by constructing compact representations of what has been learned,
building bridges between different levels of description, and identifying unifying
concepts and principles (Dayan and Abbott, 2001).
Important contributions of computational neuroscience to neuroscience can be
dated back as early as 1907 when Louis Lapicque proposed the integrate-and-fire model,
which formed the basis for later models of the membrane (see (Brunel and van Rossum,
2007)). Since then, computational neuroscience has been advancing the understanding of
neural systems. For example, earlier work such as Hudgkin and Huxley’s action potential
model (Hodgkin and Huxley, 1957) and Wilfred Rall’s axonal cable theory (Rall, 1959)
as well as recent work such as the temporal difference model of Dopaminergic systems
(Schultz et al., 1997). For more comprehensive introduction of computational
neuroscience, we invite readers to read Theoretical Neuroscience (Dayan and Abbott,
2001).
７
2.3. Motor learning
2.3.1. Why motor learning?
In this dissertation, we investigated on motor learning. Why do we study learning
in the motor system among other neural systems? In (Shadmehr and Wise, 2005), Reza
Shadmehr and Steven Wise answered to the question as follows: First, all learning
depends on motor learning in a sense that initially the vertebrate central nerve systems
(CNS) evolved to learn how, when, and where to move, then later this basic neuronal and
synaptic mechanisms support other forms of learning. Second, the motor system has
advantages for studying learning over other systems because its outcomes, such as forces,
velocities, etc, are more physical and less abstract.
In addition to its fundamental importance listed above, motor learning is also
crucial in practice. Motor learning is essential for rehabilitation after brain disease or
injury causing movement disorders. Thus, by better understanding motor learning, we can
improve physical well-being and quality of life of humans. Not only patients with
movement-impairment, but also healthy individuals can benefit from motor learning
because it is basis of sports, physical exercise, musical instrument learning, dance
instruction etc.
2.3.2. What is motor learning?
We can define motor learning as narrow as skill acquisition (Schmidt and Lee,
1999) or as wide as learning including instinctive behaviors, habituation, reflexes, skill
acquisition, motor adaptation, decision making and etc. (Shadmehr and Wise, 2005). Here
８
we focused on motor learning as skill acquisition, which is learning to become more
capable to perform a new skill, and motor adaptation, which is learning to adapt to the
changes in circumstances based on sensory prediction errors and to regain the former
performance. Also we focused on the well-studied subject of error-driven learning in
goal-directed tasks (see (Krakauer and Shadmehr, 2007; Shadmehr and Krakauer, 2008;
Shadmehr et al., 2010) for reviews). In remaining parts of this section, we introduced
models of motor learning with the limited focus listed above.
2.3.3. Motor adaptation
Researchers have investigated motor adaptation extensively in experiments since
Hermann von Helmholtz conducted the first-recorded visuomotor adaptation experiment
in 1867. Typical experimental paradigms in studies of motor adaptation introduce one or
multiple kinds of perturbations (also called tasks) that distort the motor output of the
motor command. For example, in visuomotor rotation paradigms, while subjects move a
cursor on the screen to target positions, the cursor position is rotated either clockwise or
counter-clockwise relative to the hand position.
A wide range of data has revealed the characteristic of motor adaptation such as:
savings, wherein the second adaptation to a task is faster than the first(Kojima et al.,
2004);anterograde interference, wherein learning a second task interferes with the recall
of the first task(Miall et al., 2004); spontaneous recovery, wherein if an adaptation period
is followed by a brief reverse adaptation period, a subsequent period in which errors are
clamped to zero causes a rebound toward the initial adaptation(Smith et al., 2006; Ethier
et al., 2008); generalization, wherein learning transfers to unpracticed regions of task
９
space (Krakauer et al., 2000), and consolidation, wherein, the learning consolidates both
over time and with increased initial training (Brashers-Krug et al., 1996; Krakauer et al.,
2005). Held and colleagues showed that motor adaptation requires goal-directed
movements (Held and Freedman, 1963). Mazzoni and Krakauer showed that motor
adaptation is an implicit learning rather than an explicit learning without cognitive
compensation (Mazzoni and Krakauer, 2006).
2.4. Computational models of motor learning
2.4.1. Computational framework for the motor system
Before we further talk about models of motor learning, let us first introduce the
computational framework for the motor system, especially in case of goal-directed
movements.
In their review (Shadmehr and Krakauer, 2008), Shadmehr and Krakauer
summarized three problems that the motor system faces to generate accurate movements:
system identification, which involves to predict the sensory consequences of motor
commands; state estimation, which involves to estimate the state of our body and the
world using the prediction from system identification and actual sensory feedbacks, and
optimal control, which involves to adjust gains of sensorimotor feedback loops to achieve
maximum performance.
Our motor system is noisy: the relationship between motor commands and outputs
are variable, and the sensory feedback delivered to the system is also noisy and delayed.
Nevertheless, after it learns movements, the motor system can generate the movements
１ ０
reliably with the fast speed and small variability. Therefore, the motor system must be
using not only the sensory feedback, but also a “forward model”, which predicts the
consequence of a motor command without feedback. Since the output of the system
depends on the motor command as well as the current state of the system, the forward
model must know the current state to predict the consequence of a motor command. This
requires a “state estimation”, which estimates the state of the system and updates its
estimation based on sensory feedback. The motor system should combine this estimate of
the state with the sensory feedback to generate a motor command to execute a movement.
Since there are two sources of information which do not always agree with each other, the
system must decide which information it weighs more. The “feedback controller” with
“cost function” does this job optimally in a sense that it minimizes the total cost or
equivalently maximizes the total reward (Fig. 2.1A).
Here, by “state”, we mean a variable which contains all the information necessary
for identifying the outcomes of the system. For example, knowing the current position,
velocity and spin of a thrown ball, that is the ball’s state, allows predictions of its future
path without the need to know the configurations of all the atoms in the ball. Often the
state in the models of motor system is a physical property such as position, velocity,
or/and force. However, sometimes, the state can be arbitrary and represent conceptual
amount such as level of adaptation or memory (Smith et al., 2006; Kording et al., 2007;
Lee and Schweighofer, 2009).
１ １
Figure 2.1.A schematic model for generating goal-directed movements. Adopted from
(Shadmehr and Krakauer, 2008)
To understand what the state is, you may find the explanation of Wolpert and
Ghahramani helpful (Wolpert and Ghahramani, 2000):
If we consider the 600 or so muscles in the human body as being, for extreme
simplicity, either contracted or relaxed. This leads to 2
600
possible motor
１ ２
activations, more than the number of atoms in the universe. ... Fortunately for
control, a compact representation with far lower dimensionality than the full
sensor and motor array can generally be extracted, known as the ‘state’ of the
system. When taken together with fixed parameters of the system and the
equations governing the physics, the state contains all the relevant time-varying
information needed to predict or control the future of the system.
Let us look at “feedback controller” more detail to see how this framework
generates optimal motor commands that can generate fast and accurate movements. Once
the motor system can estimate the state of the system in the system estimation, and
predict the sensory consequence of a motor command in the forward model correctly,
next thing to do is to generate the optimal motor command which can achieve a goal
while minimizing costs and maximizing rewards.
Given the cost function to minimize and system dynamics from the forward model,
finding the optimal series of motor commands becomes the problem of well-established
optimal control theory (Kalman, 1960). Especially if the system allows assumptions of
the quadratic cost function, linear system dynamics and additive Gaussian noise, as
follows:
System dynamics or forward model:
1 t t t t
A B ε
+
= + + x x u
Observation or sensory feedback:
t t t
y Hx ω = +
Cost function:
T T
t t t t t
x Q x u Ru +
the solution becomes the classical linear-quadratic-Gaussian (LQG) solution that is:
1
ˆ
ˆ ˆ ˆ ( )
t t t
t t t t t t
G
A B K H
+
= - = + + - u x
x x u y x
where, ˆ x is the estimate of the state x , G is the time-varying gain of the system for
１ ３
the state estimate ˆ x , and K is the Kalman gain which optimally updates the state
estimate based on the estimate error observed and estimate error covariance, which
represents the uncertainty of its estimation. ε andω are Gaussian noises with zero mean
and standard deviations of
ε
σ and
ω
σ respectively. For details of the classical LQG
solutions are available in (Kalman, 1960).
However, the motor system does not hold the assumptions for the LQG system. It
has been shown that the motor command carries signal-dependent noise, of which
variance is proportional to the square of the motor command magnitude. i.e.
ε
σ ∝ u
(Harris and Wolpert, 1998). Todorov and his colleagues derived the solution for this case
of signal-dependent noise for the motor command and showed that the motor system with
this optimal feedback controller reproduces task-constrained variability, goal-directed
corrections, motor synergies, controlled parameters, simplifying rules and discrete
coordination modes emerge naturally without further constraints (Todorov and Jordan,
2002; Todorov, 2005).
In this framework, the cerebellum plays the role of the forward model predicting
the sensory consequence of a motor command; the parietal cortex plays the role of the
state estimation estimating and updating the state of the motor system using the
prediction from the cerebellum and delayed sensory feedback; the premotor and primary
motor cortex plays the role of the feedback controller to generate the optimal motor
command to minimize the total cost, and the basal ganglia plays the role of the cost
function computing the expected cost of the motor commands and reward of the sensory
states (Fig. 2.1B).
１ ４
It is important to understand that, the framework described here is for quantitative
models, which deal with relatively low-level aspects of motor control and reproduce
experimental data for simple movements as accurate as possible. There is another type of
approaches, which is for qualitative and descriptive models, which deal with high-level
aspects of the motor system and investigate higher order motor deficits (see table 2.1,
(Krakauer and Shadmehr, 2007)). Also, not surprisingly, there are frameworks to describe
the motor system quantitatively other than one described here. Kawato and his colleagues
proposed a hierarchical neural network model for control and learning voluntary
movements (Kawato et al., 1987). The main difference between two frameworks is
whether the motor system generates and uses a series of desired motor output to find the
optimal motor command on-line. For the example of reaching movement, unlike the
framework in Figure 2, the hierarchical neural network model generates the desired
trajectories of movement in advance and used it as a part of cost function to constraint
actual movements. Earlier models of reaching movements used such approach (Flash and
Hogan, 1985; Uno et al., 1989).
Table 2.1.An apparent divide between the approaches to motor behavior taken by motor
control physiologists versus that taken by clinical neuropsychologists. Adopted
from(Krakauer and Shadmehr, 2007)
１ ５
2.4.2. Models of motor learning
In the framework described in section 2.4.1, the motor system can make the fast
and accurate motor output if the forward model can predict the sensory consequence of
motor command correctly. However, when the relationship between a motor command
and related motor output is changed as in motor adaptation experiments, the prediction of
the forward model will not be valid anymore. To keep its performance at a desired level,
the forward model of the motor system must learn how to adapt to those changes.
Therefore, in this framework, the models of motor learning can be seen as models of how
the forward model of motor system adapts to changes either in the motor system itself or
in the environments.
As we discussed in section 2.3.2, there are a large body of experimental findings
on motor adaptation. Here we introduced important models of motor adaptation, which
account for those findings.
Savings and spontaneous recovery in motor learning
Savings has been shown in the experimental paradigm, called A-b-A, which
exposes subjects to a perturbation A (called adaptation) that distorts movements to one
direction, followed by another perturbation b (called de-adaptation) that distorts
movements to the opposite direction and brings subjects’ movements close to the
movements before the first perturbation, and then exposes subjects to the perturbation A
again (called re-adaptation). In this A-b-A paradigm, the adaptation to the second A
perturbation is faster than the adaptation to the first A perturbation. This savings in motor
１ ６
adaptation suggests that, after exposed to the perturbation b, although subjects’
movements look the same as before the perturbation A (called baseline), learning of A is
saved and recalled for re-learning.
Spontaneous recovery has been shown in the experimental paradigm, called the
error-clamping, which is the extended version of A-b-A. The error-clamping paradigm
exposes subjects to two opposite perturbations in the order of A-b-A as the A-b-A
paradigm, but places the error-clamping trials (or channel trials) between b and A, where
subjects’ movement errors are clamped to zero by experimenters. During the error-
clamping trials, subjects’ motor outputs are initially the same as the baseline, but
gradually become similar to motor outputs learned under the perturbation A, and then,
even more slowly return to the baseline again. Lastly, when subjects are exposed to A
again, subjects show faster relearning as in the savings paradigm. This spontaneous
recovery in motor adaptation suggests that de-adaptation during the perturbation b is not
erasing the previous adaptation to A, but creates a new adaptation to b.
To account for savings and spontaneous recovery, Smith and his colleagues
proposed a simple dual-timescale model, in which the forward model make the prediction
of the sensory consequence of motor output, ˆ y using two states with different
timescales: one fast
f
x and one slow
s
x (Smith et al., 2006):
ˆ
f s
y x x = + , and
( )
( )
f f f f
s s s s
x A x B y y
x A x B y y
= ⋅ + ⋅ - = ⋅ + ⋅ - )
)
,
where,
f
A and
f
B are the forgetting and learning rates of the fast process, and
s
A and
１ ７
s
B are the forgetting and learning rates of the slow process. We use the term of process
in a sense that there are two learning processes with different timescales, each learning
process has one state to estimate, and the final estimate of the consequence of sensory
motor output is determined by the sum of two estimated states.
This model accounts for savings: in the A-b-A paradigm, initially the states of
both fast and slow processes start at zero. After A, both processes learn A. After b, when
the motor output returns back to the baseline (i.e. 0 y = ), the states
f
x and
s
x do not
represent the baseline because of difference in timescales between two processes:
f
x
represents b while
s
x still represents A. When subjects are exposed to A again,
f
x
quickly forgets b and re-adapts to A and
s
x continues to adapt to A. Therefore, the sum of
two can represent A faster in relearning compared to in initial learning.
This model also accounts for spontaneous recovery: in the error-clamping
paradigm, at the end of A-b, it’s the same as in the savings paradigm:
f
x represents b
while
s
x still represents A. During the error-clamping trials after A-b, both the fast and
slow processes don’t observe the prediction error ( ˆ y y - ) and cannot update their states,
and
f
x quickly forgets b and
s
x slowly forgets A. Therefore, the sum of two moves from
the baseline (i.e. 0 y = ) towards A until
f
x becomes zero and slowly returns back to the
baseline showing the slow forgetting of A in
s
x .
Later Kording and his colleagues extended this dual-timescale model to the
multi-timescale model (Kording et al., 2007). Although their model behaves in a similar
way as the dual-timescale model by Smith and his colleagues, they started not
１ ８
postdictively from experimental results but predictively from the nomadic question of
how the motor system could optimally deal with changes of the body (or disturbances)
which are multi-timescale in nature. They rationalized that to adapt optimally, the motor
system must have representations (or a generative model) of disturbances and predict
what kind of disturbances the motor system is exposed to most likely based on the
prediction of the generative model. They used the generative model of multi-timescale
disturbances as follows:
( ) (1 1/ ) ( ) x t x t
τ τ τ
τ ε + Δ = - ⋅ +
y(t)= ( ) x t
τ
ω +
∑
where x
τ
represents the disturbance with the time constant of τ and y represents the
total disturbances experienced in the motor system.
τ
ε is Gaussian noise with mean of
zero and standard deviation of
τ
σ , which represents the uncertainty of the estimate of
each disturbance. The smaller time constant of the disturbance is, the larger the
uncertainty is. Lastly, ω is the observation noise. They used 30 different time constants
from 2 to
5
3.3 10 × . When the motor system is exposed to a disturbance, it first estimates
x
τ
in a Bayesian way using Kalman filter (Kalman, 1960), and based on its belief on x
τ
,
it adapts to x
τ
.
By introducing the Bayesian update accounting for uncertainty, (Kording et al.,
2007) allows the model to reproduce faster learning after removal of sensory inputs. It
has been shown that after depriving sensory information (ex. Extended period of darkness
in saccadic eye movement experiments), the learning rate increases (Kojima et al., 2004).
１ ９
Unlike (Smith et al., 2006), which always updates both the fast and slow processes with
the fixed learning rates
f
B and
s
B , (Kording et al., 2007) adapts proportional to the
uncertainty of x
τ
: i.e. It adapts faster when it is less certain about disturbances. Also with
the wider range of timescales, it successfully reproduced long-timescale adaptation data
(Robinson et al., 2006).
Consolidation in motor learning
After the practice of motor tasks, learning (or motor memory) gained during the
practice consolidates from a fragile state, which is easy to lose either over time or by
learning interfering tasks, to a more robust state, which lasts longer and robust to
interferences (Krakauer and Shadmehr, 2006). Consolidation in motor learning is
measured by checking savings at the delayed retention test in the A-B-A paradigm, while
the start time and duration of the interfering task B varies and the second A is given
relatively long time after the practice. For example, Brashers-Krug and his colleagues
showed that motor memory of the first task A in the force-field adaptation consolidated
gradually over 4 hours: i.e. learning the second task B after this period of time did not
prevent savings at the 24-hour after delayed retention test of A (Brashers-Krug et al.,
1996).
Can the multi-timescale model account for consolidation in motor learning?
Joiner and Smith showed in the force-field adaptation experiments that motor output at
the delayed retention test correlated with the state of the slow process
s
x at the end of
the practice in the multi-time scale model (Joiner and Smith, 2008). However, according
２ ０
to the forgetting rate of the slow process found in experiments (Smith et al., 2006; Ethier
et al., 2008), the time constants of the slow process ( 1 1/
s s
A τ = - ) are relatively short:
from 8 min for saccadic eye movements to 10 min for force-field adaptation: i.e.
According to these time constants, after 24 hours without practice, even the slow process
will become close to the baseline. Zarahn and his colleagues pointed out this issue and
proposed the variable parameters in different practice contexts (in other words, meta-
parameters; see (Doya, 2002)) such as the initial adaptation, de-adaptation, re-adaptation
(Zarahn et al., 2008). Furthermore, Criscimagna-Hemminger and Shadmehr measured
forgetting rates
f
A and
s
A both during practice and after practice, and found the
forgetting rates of the fast and slow processes during rest after practice are different from
forgetting rates during practice: In terms of time constants of the slow process, the time
constant during rest was 18.5 hours and the one during practice was 600 msec. Also in the
error-clamping paradigm, by varying the duration of rest after the second disturbance b
before error-clamping trials, they found that during rest learning in the fast process
transferred (in other words, consolidated) to the slow process (Criscimagna-Hemminger
and Shadmehr, 2008). In summary, the multi-time scale model with meta-parameters can
account for consolidation at least in part.
Limitations of the multi-timescale motor learning models
In spite of their great success in accounting for a wide range of motor learning
phenomena, the multi-timescale models described above (Smith et al., 2006; Kording et
al., 2007; Criscimagna-Hemminger and Shadmehr, 2008) cannot account for dual- or
２ １
multi-task learning because sufficient adaptation to a new task overrides adaptation of a
previous task in such models. When given contextual cues and sufficient trials, humans
can simultaneously adapt to two opposite force fields (Osu et al., 2004; Nozaki et al.,
2006; Howard et al., 2008), two saccadic gains (Shadmehr and Wise, 2005),or several
visuomotor rotations (Imamizu et al., 2007; Choi et al., 2008). Multi-task motor learning
is natural and important because in real life, most of time, humans must learn and
perform multiple tasks simultaneously. Thus to apply the model of motor learning in
practice, we need to understand the mechanism of multi-task learning better.
２ ２
Chapter 3.
Dual adaptation supports a parallel architecture of motor
memory
3.1. Introduction
Recent studies support the hypothesis that motor adaptation to external
perturbations such as force-field, saccadic gain shift, and visuomotor transformations
occurs at multiple time scales (Kojima et al., 2004; Hatada et al., 2006; Smith et al.,
2006). To account for this multi-timescale adaptation, Smith et al. (2006) proposed a two-
state model, in which a fast process contributes to fast initial learning, but forgets quickly,
and a slow process contributes to long-term retention, but learns slowly. This model
successfully accounts for a number of adaptation phenomena, including: savings, wherein
the second adaptation to a task is faster than the first (Kojima et al., 2004); anterograde
interference, wherein learning a second task interferes with the recall of the first task
(Miall et al., 2004); and spontaneous recovery, wherein if an adaptation period is
followed by a brief reverse-adaptation period, a subsequent period in which errors are
clamped to zero causes a rebound toward the initial adaptation (Smith et al., 2006).
How these proposed fast and slow processes area organized, however, is ambiguous.
Because the two-state model is linear, it can account for the above data with either a
serial organization, in which the fast process updates its state from motor errors and sends
its output to the slow process, or a parallel organization, in which both the fast and slow
２ ３
processes simultaneously update their states from errors (Smith et al., 2006).
Furthermore, such two-state models cannot explain dual or multiple task adaptation,
because sufficient adaptation to a new task overrides adaptation of a previous task in such
models. When given contextual cues and sufficient trials, humans can simultaneously
adapt to two opposite force fields (Osu et al., 2004; Nozaki et al., 2006; Howard et al.,
2008), two saccadic gains (Shelhamer et al., 2005), or several visuo-motor rotations
(Imamizu et al., 2007; Choi et al., 2008). The MOdular Selection and Identification for
Control (MOSAIC) model (Wolpert and Kawato, 1998) naturally accounts for dual- or
multiple-adaptation via non-linear switching among multiple parallel internal models.
However, because MOSAIC uses only a single time scale for learning and no forgetting
(that is, it does not contain distinct fast and slow processes), it cannot explain large
increases of errors at the beginning of each block in a dual adaptation experiment with
alternating blocks (Imamizu et al., 2007), or phenomena such as spontaneous recovery.
Here, we systematically addressed two following questions: Are the proposed fast
and slow processes arranged serially or in parallel? Are there one or more states for each
proposed fast and slow process? Systematic simulations of motor adaptation of candidate
models in different adaptation experimental paradigms show that only two models, one
parallel and one serial, both with a fast process with one state and a slow process with
multiple states that are switched non-linearly by a contextual cue, can account for all
simulated data. To further differentiate between these two models, we then designed a
visuomotor-rotation experiment and compared dual-adaptation in healthy human subjects
to dual-adaptation predicted by the serial and the parallel models.
２ ４
3.2. Methods
3.2.1. Experimental procedure
12 right-handed healthy subjects (7 men, 5 women, 23-33 years of age) signed an
informed consent to participate in the study, which was approved by the local IRB.
Subjects sat in front of a LCD monitor with holding a joystick. At each trial, subjects
moved a cursor to a target using the joystick. At the beginning of a trial, a cursor
appeared at the center position. 2 seconds later, a target appeared at one of four positions
of the screen (top, right, left, and bottom) 15 cm from the center, and the cursor
disappeared. Subjects had 2 seconds to move the cursor to the target without visual
feedback of the cursor trajectories. To provide feedback on performance, the cursor then
appeared again for 1 second at a position 15 cm from the center along the direction of the
final cursor position. Inter-Trial Intervals (ITIs) were varied randomly from 2 seconds to
14 seconds. At each trial, we measured the directional error between the target direction
and the final cursor direction from the initial cursor position. When subjects did not move
within 2 seconds in a trial, the trial was regarded as a missed trial, and a next trial started.
In the training session, we altered the mapping between the joystick and cursor
directions using four different visuomotor rotations (Krakauer et al., 1999; Wigmore et al.,
2002; Miall et al., 2004; Krakauer et al., 2005; Hinder et al., 2007; Seidler and Noll,
2008): 25
o
(task A), -25
o
(task B), -50
o
(task C), and 50
o
(task D). For subjects to
distinguish between the different tasks, we used target positions as a contextual cue:
Targets for each of four visuomotor rotation tasks appeared at one of four different
positions (top, right, left, and bottom). The cue positions were counterbalanced across
２ ５
subjects. In the first 100 trials of the training session, subjects practiced tasks A and B in
a massed schedule, which consisted of three consecutive blocks of 50 trials of task A, 25
trials of task B, and 25 trials of task A. In the second 100 trials of the training session,
subjects practiced task C and D in a pseudo-random schedule: In every two-trial block,
one of two tasks was chosen randomly and presented followed by the other task.
In our experiment, we used the A-B-A paradigm as a massed schedule in the first
half of the training session for two reasons. First, such paradigm has been widely used in
previous motor adaptation studies (Brashers-Krug et al., 1996; Miall et al., 2004;
Krakauer et al., 2005). Second, it is the simplest schedule that allowed us to estimate
model parameters reliably with small confidence intervals (see Figure 2.6): With simpler
schedules such as 100 trials of A or 50 trials of A followed by 50 trials of B, the data set is
not sufficient to estimate four parameters of the models reliably.
Before the training session, subjects performed 200 trials of a baseline session, in
which there was no rotation and targets appeared on four positions in a pseudo-random
order.
3.2.2. Candidate models
We searched for the most parsimonious model that can simultaneously account
for all the following motor adaptation data: savings, spontaneous recovery, anterograde
interference, and dual adaptation in both blocked and random schedules. We modeled
motor adaptation via the summation of the multiple internal states, each modeled with a
linear differential equations (see below) with a learning term and a forgetting term (Smith
et al., 2006). We studied all possible models with either a serial or a parallel organization
２ ６
of the fast and slow processes, where each process containing either a single state or
multiple states. Furthermore, although previous experiments and modeling studies are
consistent with the idea that motor adaptation occurs at multiple timescales rather than at
a single timescale (Kojima et al., 2004; Hatada et al., 2006; Smith et al., 2006; Kording et
al., 2007; Criscimagna-Hemminger and Shadmehr, 2008; Ethier et al., 2008), we also
studied models with a single process, with either a single state or multiple parallel states.
for the completeness of comparisons.
Figure 3.1: Ten possible motor adaptation models that address the three follwing
questions: (1) Are there slow and fast timescales? (2) Are there one or more states for
each timescale? (3) Are the fast and slow processes arranged serially or in parallel?
Such systematic search led to 10 different possible models (Figure 3.1): (1) a 1-
state model, (2) a serial 1-fast 1-slow model, (3) a parallel 1-fast 1-slow model, (4) a
parallel n-state model, (5) a serial n-fast n-slow model, (6) a parallel n-fast n-slow model,
２ ７
(7) a serial 1-fast n-slow model, (8) a parallel 1-fast n-slow model, (9) a serial n-fast 1-
slow model, and (10) a parallel n-fast 1-slow model.
The 1-state model and the parallel and serial 1-fast 1-slow models are identical to
those proposed and studied in Smith et al. (2006). For all other models, we added
multiple inner states in either the fast or the slow process, or both. The differential
equations for all states within a process have the same parameters, but the states receive
different contextual cue inputs. As in MOSAIC, the contextual cue input has two roles: it
selects the appropriate state(s) to be summed in the total output, and it allows the
updating of these selected state(s) from motor errors. Forgetting is not gated by the
contextual input (see model equations below). For the sake of simplicity, we make the
following assumptions: (1) no interference between multiple states, (2) perfect switching
between multiple states, and (3) identical learning and forgetting rate parameters for all
states within a process. Thus, except for the 1-state model and the parallel n-state model,
which contain only 2 parameters (1 forgetting rate A and 1 learning rate B ), all models
contain 4 parameters: 1 forgetting rate and 1 learning rate for each fast and slow process
(
f
A ,
f
B ,
s
A ,
s
B ) (model parameters are given below).
For all models, at each trial n , the motor error input e is determined by the
difference between an external perturbation f and the motor output y :
( ) ( ) ( ) e n f n y n = - (1)
For the 1-state model, the state update equation is simply given by:
( ) ( ) y n x n = , and (2)
( 1) ( ) ( ) x n A x n B e n + = ⋅ + ⋅ , (3)
２ ８
where x is a learning process with a single state, A is a forgetting rate, and B is a
learning rate.
In the 1-fast 1-slow models, the fast and slow processes have a single inner state
each. The state update rules for the parallel representation of the 1-fast 1-slow models are
thus given by (Smith et al., 2006) :
( ) ( ) ( )
f s
y n x n x n = + , and (4)
( 1) ( ) ( )
( 1) ( ) ( )
f f f f
s s s s
x n A x n B e n
x n A x n B e n
+ = ⋅ + ⋅
+ = ⋅ + ⋅
, (5)
Figure 3.2: The two possible architectures of the 1-fast n-slow model: (A) Serial and (B)
parallel. e is a motor error, c is a contextual cue, and x is a motor output. Multiple boxes
in the slow process represent internal states switched by the contextual cue input.
２ ９
where
f
x and
s
x are a fast and a slow learning process with a single state respectively.
In the parallel n-state model, there is only one process, which has multiple inner
states.
( ) ( ) ( )
T
y n n n = x c , and (6)
( 1) ( ) ( ) ( ) n A n B e n n + = ⋅ + ⋅ ⋅ x x c , (7)
where x is a learning process with
task
N internal states, and c is the contextual cue.
These two variables are vectors of length
task
N , equal to the number of tasks in the
experiment. Because, we assumed no interference and perfect switching among states in
the slow process, we use a unit vector for c . For example, for the first task,
(1,0,...,0)
T
= c , for the second task, (0,1,...,0)
T
= c , and so on.
In the n-fast n-slow models, both the fast and slow processes have multiple inner
states and (and thus both receive a contextual cue input). The state update rules for the
parallel representation of the n-fast n-slow models are thus given by:
( ) ( ) ( ) ( ) ( )
T T
f s
y n n n n n = + x c x c , and (8)
( 1) ( ) ( ) ( )
( 1) ( ) ( ) ( )
f f f f
s s s s
n A n B e n n
n A n B e n n
+ = ⋅ + ⋅ ⋅
+ = ⋅ + ⋅ ⋅
x x c
x x c
, (9)
where
f
x and
s
x are fast and slow processes with
task
N internal states.
The parallel 1-fast n-slow model (Figure 2.2A) has a fast and a slow process
organized in parallel, with a single state in the fast process, and multiple states in the slow
process. The state update rules for the parallel representation for this model are given by:
( ) ( ) ( ) ( )
T
f s
y n x n n n = + x c , and (10)
３ ０
( 1) ( ) ( )
( 1) ( ) ( ) ( )
f f f f
s s s s
x n A x n B e n
n A n B e n n
+ = ⋅ + ⋅
+ = ⋅ + ⋅ ⋅ x x c
. (11)
Similarly, in the parallel n-fast 1-slow models, only the fast process has multiple
inner states. The state update rules for the parallel representation of the n-fast 1-slow
models are:
( ) ( ) ( ) ( )
T
f s
y n n n x n = + x c , and (12)
( 1) ( ) ( ) ( )
( 1) ( ) ( )
f f f f
s s s s
n A n B e n n
x n A x n B e n
+ = ⋅ + ⋅ ⋅
+ = ⋅ + ⋅
x x c
. (13)
All serial models are identical to their parallel counterparts except that the slow
process does not receives the motor error input e directly, but receives the output of the
fast process
f
x . For example, the state update rule of the slow process for the serial
representation of the 1-fast n-slow model is (compare to equation 11):
( 1) ( ) ( 1) ( )
s s s s f
n A n B x n n + = ⋅ + ⋅ + ⋅ x x c (14)
Finally, it should be noted that we attempted to model the common neuronal
mechanism of motor adaptation as in Smith et al. (2006) or Kording et al. (2007), but not
the mechanism of specific type of motor adaptation. Therefore, our model does not
account for the effect of physiological factors, such as muscle mechanics, limb dynamics,
etc, in a certain motor adaptation paradigm.
３ １
Figure 3.3: Simulation of spontaneous recovery for all ten models considered: 1-state
model, parallel n-state model, serial and parallel 1-fast 1-slow models, serial and parallel
n-fast 1-slow models, serial and parallel n-fast n-slow models, serial and parallel 1-fast n-
slow models: (A) Schedule of the error-clamping paradigm used to induce spontaneous
recovery, which consists of 180 trials for adaptation to one stimulus (1), 20 trials for
adaptation to the opposite stimulus (-1, de-adaptation), and 50 error-clamping trials,
during which errors are clamped to zero. (B) Model predictions of adaptation
performance for all models. The parallel and serial models are superimposed in all panels
except for the 1-state and parallel n-state models. Check marks and crosses and are used
to show which models account or do not account for the data, respectively. All models
except the 1-state and parallel n-state model can reproduce the characteristic of
spontaneous recovery: the output for the first error-clamp trial starts near the baseline
(zero), increases trial by trial, and decays slowly.
３ ２
Figure 3.4: Simulation of anterograde interference for the eight remaining candidate
models: serial and parallel n-fast n-slow models, serial and parallel 1-fast 1-slow models,
serial and parallel n-fast 1-slow models, serial and parallel 1-fast n-slow models. The
parallel and serial models are superimposed in all panels. (A) Schedule of the A-B-A
paradigm used to induce anterograde interference, which consists of 100 trials for
adaptation to one stimulus (1), 100 trials for adaptation to the opposite stimulus (-1, de-
adaptation), and 100 trials for re-adaptation to 1. (B) Model predictions of adaptation
performance in the A-B-A paradigm. (C) Comparisons of initial errors in each session. As
in Figure 3.3, check marks and crosses are used to show which models account or do not
account for the data, respectively. All models except the serial and parallel n-fast n-slow
models can reproduce the characteristic of anterograde interference: Both the initial
errors of de-adaptation and re-adaptation are greater than the initial error of the first
adaptation.
３ ３
Figure 3.5: Simulations of two dual-adaptation experiments for the remaining six models
considered: serial and parallel 1-fast 1-slow models, serial and parallel n-fast 1-slow
models, serial and parallel 1-fast n-slow models. The parallel and serial models are
superimposed in all panels. (A) Intermittent alternation of two tasks and (B) random
alternation between two tasks. For each model, the same parameters are used in (A) and
(B). Only the serial and parallel 1-fast n-slow models can reproduce dual-adaptations in
both intermitted and random conditions. Note that the parallel and serial 1-fast n-slow
models behave identically in (A) but differently in (B): The parallel 1-fast n-slow model
shows faster adaptation rates than the serial 1-fast n-slow model in random dual-
adaptation.
3.2.3. Simulation parameters
Here, we chose parameters for all models to reproduce previous experiment results
qualitatively. Note however that the simulation results are not limited by these particular
parameter values. These qualitative results are valid across wide ranges of parameters
３ ４
(see Supplementary Materials for details). The parameters of the serial models were
determined such that these models behave identically to the corresponding parallel
models in massed schedules. (see Supplementary Materials).
In the simulations of spontaneous recovery (Figure 3.3) and anterograde
interference (Figure 3.4), we used the parameters give by Smith et al. (2006): 0.92
f
A = ,
0.996
s
A = , 0.03
f
B = , and 0.004
s
B = for the parallel 1-fast 1-slow, n-fast 1-slow, n-
fast n-slow, 1-fast n-slow models. For the serial 1-fast 1-slow, n-fast 1-slow, n-fast n-slow,
1-fast n-slow models, we used parameters of 0.92
f
A = , 0.996
s
A = , 0.0337
f
B = , and
0.0091
s
B = . For the 1-process model and parallel n model, we used 0.996 A = ,
0.004 B = .
In the simulations of intermittent and random dual-adaptation paradigms (Figure
3.5), we chose parameters for all models to qualitatively reproduce saccadic adaptation
results of (Shelhamer et al., 2005). We used parameters of 0.6
f
A = , 0.998
s
A = ,
0.1
f
B = , and 0.025
s
B = for the parallel 1-fast 1-slow, n-fast 1-slow, n-fast n-slow, 1-
fast n-slow models, 0.6
f
A = , 0.998
s
A = , 0.115
f
B = , and 0.087
s
B = for the serial 1-
fast 1-slow, n-fast 1-slow, n-fast n-slow, and 1-fast n-slow models.
In the simulations of wash-out paradigm (Figure 3.7), we chose parameters for all
models to qualitatively reproduce results of Zarahn et al. (2008): (1) For the two-state
model, 0.519
f
A = , 0.983
s
A = , 0.193
f
B = , and 0.159
s
B = . (2) For the varying-
parameter model, as in Zarahn et al. (2008), in the initial learning phase, 0.492
f
A = ,
0.986
s
A = , 0.077
f
B = , and 0.116
s
B = , in the wash-out phase, 0.480
f
A = ,
３ ５
0.975
s
A = , 0.230
f
B = , and 0.330
s
B = , and in the relearning phase, 0.548
f
A = ,
0.975
s
A = , 0.088
f
B = , and 0.330
s
B = . (3) For the parallel 1-fast n-slow model,
0.953
f
A = , 1
s
A = , 0.141
f
B = , and 0.032
s
B = .
In the simulations of experimental paradigms in which contextual cues were given
explicitly, such as in paradigms of anterograde interference (Miall et al., 2004) or dual-
and multiple-adaptation (Osu et al., 2004; Shelhamer et al., 2005; Choi et al., 2008) we
assumed that contextual switching occurred at the time of task-switching. The models
thus use a switched contextual input c from the first trial of the new task. For example,
if task A had been presented from 1
st
trial to 100
th
trial and was changed to task B on 101
st
trial, then, c(1) = … c(100) = c
A
and c(101) = c
B.
In the simulations of experimental paradigms in which contextual cues were not
given explicitly, such as in paradigms of spontaneous recovery (Smith et al., 2006) or
wash-out (Zarahn et al., 2008), we assumed that the errors after the first trial of the new
task served as the switching trials, and that contextual switching occurred after those
trials. The models thus use a switched contextual input c from the second trial of the
new task. From above example, c(1) = … c(100) = c(101) = c
A
and c(102) = c
B
.
3.2.4. Model parameter fitting
We estimated the parameters of the parallel and serial 1-fast n-slow model using
data in the massed schedule (i.e. the first 100 trials of the training session), so that both
models predicted the data in the massed schedule equally well. To find confidence
intervals of model parameter estimates, we used the bootstrap-t method because it is more
３ ６
accurate than standard parametric confidence intervals especially for samples with
unknown distributions and small sample numbers (DiCiccio and Efron, 1996): First, we
calculated an observed data mean ˆ x by averaging the data of the 12 subjects at each
trial. We also generated 10,000 bootstrap estimates of data mean
*
ˆ x (our notations are
those used in (DiCiccio and Efron, 1996):
ˆ
θ is an estimate for a parameter of interest
θ ,
*
θ is the bootstrapped data of θ , and
*
ˆ
θ is the estimate of bootstrapped data
*
θ ):
For this purpose, we re-sampled the 12 subjects’ data 10,000 times with replacement, and
took averages of the resampled data sets. We then fitted the models to both the observed
data mean and to each of data mean estimates in the massed schedule. For the actual data
and each of the 10,000 bootstrap sets, we used the MATLAB fmincon function to find the
model parameters that maximized the log likelihood:
2
2
2
1
( ( ) ( ))
log ( | , , , ) log(2 )
2 2
N
f s f s
i
N x i y i
P A A B B πσ
σ
=
- = - - ∑
x , (15)
where { (1), (2),..., ( )} x x x N = x , ( ) x i is an average performance on i
th
trial across
subjects either in original data or in each bootstrapped data set, ( ) y i is a model prediction
on i
th
trial, N is the number of trial, and
2
σ is the variance of the model output which
represents the effects of output and state noises. As the estimate of
2
σ , we used the
average sample variance of the data ,
2
1
1
( )
N
i
i
N
σ
=
∑
, where
2
( ) i σ is the sample variance
of the data on i
th
trial across subjects either in original data or in each bootstrapped data
set.
３ ７
We then computed the 95% confidence intervals of parameters { , , , }
f f s s
A B A B θ =
of each model: For both parallel and serial models, the differences between parameter
estimates
ˆ
θ found for the observed data mean ˆ x and parameter estimates
*
ˆ
θ found
for 10,000 data mean estimates
*
ˆ x were used to estimate the distribution of the
bootstrap t-statistics T
θ
using
*
T
θ
given below:
*
*
*
ˆ ˆ
ˆ
T
θ
θ
θ θ
σ
- = , (16)
where
*
ˆ
θ
σ is the standard deviation of each bootstrap parameter estimates
*
ˆ
θ .
Here, we used the bootstrap standard errors s
θ
of parameters to approximate
*
ˆ
θ
σ . The
values of parameters related to the 2.5 and 97.5 percentile values of
*
T ,
*(0.025)
ˆ
s T
θ θ
θ +
and
*(0.975)
ˆ
s T
θ θ
θ + were used as the 95% of confidence intervals, where
*( )
T
α
θ
is the
percentile of the estimated t-distribution.
3.2.5. Model comparison
Our goal was to test which of the parallel or serial model better predicted the data:
Using the estimated model parameters in the massed schedule, we predicted data in the
random schedule (i.e. the second set of 100 trials of the training session). We then
compared the Mean Square Errors (MSEs) between the data and model predictions.
These MSEs can be seen as cross-validation errors because we used a part of data to
estimate model parameters, and used the other part of data to compare the predictabilities
of models.
３ ８
To compare the MSEs of the parallel and serial models, we used the bootstrap-t test
(DiCiccio and Efron, 1996). First, we estimated the distribution of bootstrap t-statistics
, d MSE
T using the differences between MSEs of the parallel and serial model predictions
ˆ
p s
MSE
d MSE MSE = - and
* *
*
ˆ
p s
MSE
d MSE MSE = - :
*
*
,
,
ˆ ˆ
MSE MSE
d MSE
d MSE
d d
T
s
- = , (17)
where
, d MSE
s is the bootstrap standard error of
*
MSE
d .
We then tested the null hypothesis that 0
MSE
d ≥ , i.e. in matched pairs, MSEs of the
serial model are equal to or less than the parallel model, by calculating a bootstrap one-
tailed p-value with a significance level 0.05 α = , p given:
*
, ,
, ,
ˆ
(#[ / ])
( )
d MSE MSE d MSE
d MSE c MSE
T d s
p P T t
B
≥ - = ≥ = , (18)
where
, c MSE
t are a test statistic corresponding to the hypothesis of no difference
between two means of MSEs, #[.] is the number of cases where the inner statement is
true and B is the number of bootstrap resampling 10,000.
3.3. Results
3.3.1. Simulation of spontaneous recovery supports fast and slow timescales
Spontaneous recovery is observed when a period of adaptation is followed by a
brief period of de-adaptation and a subsequent period in which errors are clamped to zero:
In the clamped period, the performance is initially near the baseline (zero), but quickly
３ ９
recovers in the next several trials, before decaying slowly back to zero again. All models
can reproduce such data, except the 1-state model and the parallel-n state model (Figure
3.3B). These models predict a monotonous decrease of the adaptation performance during
the error-clamp trials rather than spontaneous recovery. These results thus confirm and
extend previous studies (Smith et al., 2006; Criscimagna-Hemminger and Shadmehr,
2008; Ethier et al., 2008) showing that both fast and slow processes are necessary to
account for spontaneous recovery.
3.3.2. Simulation of anterograde interferences supports a context-independent
process
Next, we performed simulations of experiments that induce anterograde
interference with the eight remaining model candidates (Figure 3.4A). Anterograde
interference is observed when a period of adaptation is followed by a period of de-
adaptation and a subsequent period of re-adaptation. At the onset of the re-adaptation
period, recall of the initial adaptation is interfered by the previous de-adaptation: the
initial errors in de-adaptation and re-adaptation are greater than the initial error of the first
adaptation (Miall et al., 2004). All models except the parallel and the serial n-fast n-slow
models can reproduce this data (Figure 3.4B and C). Thus, at least one process with a
single state is necessary to account for anterograde interference. The parallel and the
serial n-fast n-slow models, which contain two processes with context-dependent
switching between states, predict no interference between first and second adaptations.
As a result, in the parallel n-fast n-slow model, the initial errors of de-adaptation and re-
４ ０
adaptation are equal to the initial error of the first adaptation; in the serial n-fast n-slow
model, these errors are smaller (Figure 3.4B).
3.3.3. Simulation of dual-adaptations supports a context-dependent slow process
To further distinguish between the six remaining candidate models that can
reproduce both spontaneous recovery and anterograde interference (the two 1-fast 1-slow
models, the two 1-fast n-slow models, and the two n-fast 1-slow models), we simulated
dual-adaptation with two types of schedules (see Methods for details): Intermittent block
schedules, in which one of two opposite adaptation tasks were presented in alternating
blocks (Figure 3.5A), and pseudo-random schedules, in which one of two opposite
adaptation tasks was presented pseudo-randomly each trial (Figure 3.5B). Previous
adaptation studies have shown gradual improvement in performance both across blocks
of trials in the intermittent schedule (Shelhamer et al., 2005), and across trials in the
random schedule (Osu et al., 2004; Choi et al., 2008). Out of the six remaining candidate
models, only the parallel and the serial 1-fast n-slow models can reproduce such data. In
the intermittent schedule, the two 1-fast 1-slow models (i.e., the two two-state models in
(Smith et al., 2006) and the two n-fast 1-slow models predict that, at the beginning of task
alternating blocks, the performance for one task did not gradually improve across blocks,
but instead was reset to zero following adaptation to the other task. Similarly, in the
random schedule, the two 1-fast 1-slow models and the two n-fast 1-slow models predict
no improvement across trials.
It is important to note that although both parallel and the serial 1-fast n-slow
models can reproduce dual-adaptation data qualitatively, the parallel models predict a
４ １
higher rate of adaptation in the random schedule than the serial model (Figure 3.5B). In
the following, we made use of these different rates of adaptation in an experiment
designed to differentiate between these two remaining candidate models.
４ ２
Figure 3.6: Average performance data across subjects during learning (black dots) and
predictions of serial (red stars) and parallel (blue crosses) 1-fast n-slow models. Red and
blue shaded areas show the ranges of ± standard errors of the serial and parallel model
predictions, respectively. Models parameter estimation was performed using the data in
the massed schedules. The models were then used to predict the data in the random
schedules. Both models fitted subject data well during the A-B-A massed schedule.
However, during the random schedule, the parallel model predicted the data better than
the serial model, as shown by smaller MSE between the data and the parallel model
predictions compared to the MSE between the data and the serial model predictions
( 0.0001 p = ). The estimated parameters (with 95% confidence intervals) are as follows:
For the parallel model, A
f
= 0.8251 (0.6338-0.9767), A
s
= 0.9901 (0.9876-0.9986), B
f
=
0.3096 (0.1585-0.5118), and B
s
= 0.2147 (0.0582-0.2729). For the serial model, A
f
=
0.8749 (0.7082-0.9643), A
s
= 0.9917 (0.9894-0.9984), B
f
= 0.4831 (0.2923-0.6655), and
B
s
= 0.0456 (0.0077-0.1178).
４ ３
3.3.4. Dual-adaptation experiment supports the parallel 1-fast n-slow model
The simulations described above show that among the ten models simulated, only
two models, the parallel and serial 1-fast n-slow models (Figure 3.2A and B) can account
for spontaneous recovery, anterograde interference, and dual-adaptation in intermittent
and random schedules. To further differentiate between these two remaining models, we
developed a new hybrid experimental schedule, in which a massed schedule is followed
by a random schedule. Because the parallel and serial 1-fast n-slow models account
equally well for learning data in massed schedules (Smith et al., 2006), but adapt at
different rates in random schedules (Figure 2.5B), we estimated the parameters of the two
models in the initial massed schedule, and then compared the model predictions to actual
data in the following random schedule.
We estimated the parameters of the parallel and serial models by fitting the models
to the average data of 12 subjects in the massed schedule, and obtained 95% confidence
intervals of the parameters using the bootstrap-t method (DiCiccio and Efron, 1996) (see
Methods for details). Figure 2.6 shows the average data of 12 subjects and model
predictions of both the parallel and the serial models in the hybrid schedule. As expected,
during the massed schedule, the models behave almost identically and give a good fit to
the data. In the random schedule, however, the model predictions of performance differ:
the serial model predicts slower learning for the two tasks; in contrast, the parallel model
predicts faster learning. As we can see in the Figure 2.6, such faster learning by the
parallel model appears to better match actual data from our subjects.
４ ４
Figure 3.7: Simulation of the wash-out paradigm with the two-state model, the varying-
parameter model, and the parallel 1-fast n-slow model. (a) Schedule of the wash-out
paradigm, which consists of 10 null trials, 80 learning trials, 40 wash-out trials, and 30
relearning trials. In learning and relearning trials, there was 45 degree of disturbance, and
in null and wash-out trials, no disturbance. (b) Model predictions of errors in the
relearning after wash-out paradigm. We used the same parameters in Zarahn et al. (2008)
for the varying-parameter model and two-state model, and chose parameters for the
parallel 1-fast 1-slow model to reproduce the results of the varying-parameter model (c)
Comparison of model predictions during the initial learning and relearning. First 30 trials
in learning and relearning trials are superimposed. The error traces of the two-state model
in learning and relearning trials are identical, and cannot reproduce savings after wash-
out trials. In contrast, the parallel 1-fast n-slow model can predict savings after wash-out
trials with fewer parameters than the varying-parameter model.
４ ５
To verify that the parallel 1-fast n-slow model better predicted the data in the
random schedule than the serial 1-fast n-slow model, we compared the Mean Squared
Errors (MSEs) in the random schedule between the data and the predictions of the
parallel and the serial models, respectively. The parallel model shows significantly
smaller MSEs (MSE, with 95% confidence intervals = 89.05 [38.1~226.6]) than the serial
model (MSEs 1007.43 [196.9~2012.9]; bootstrap-t test, 0.0001 p = ; see Methods for
details). Given this result, we henceforth only consider the parallel 1-slow n-fast model,
not the serial 1-slow n-fast model.
3.3.5. Comparison with time-varying parameter model in savings in relearning
experiment
The two-state models cannot explain savings during relearning in the wash-out
paradigm, in which a large number of wash-out trials (i.e. trials with zero-perturbation)
are inserted between the initial learning phase and the relearning phase (Zarahn et al.,
2008). A recent time-varying parameter two-state model with different decaying and
learning rates during the different perturbation conditions accounts for the changes in re-
learning speed (Zarahn et al., 2008).
To test whether the parallel 1-fast n-slow model can reproduce such data, we
performed the simulation of the wash-out paradigm, and compared the predictions of
three models: (1) the two-state (parallel) model, (2) the varying-parameter (parallel)
model with two states, and (3) the parallel 1-fast 1-slow model. The two-state model
cannot reproduce savings (Zarahn et al., 2008), but both the time-varying parameter
models and our parallel 1-fast n-slow model can account for savings, with only minute
４ ６
differences between predictions of both models (Figure 3.7). Thus, the parallel 1-fast n-
slow model explains savings during relearning after wash-out, without the need for extra
parameters and meta-learning process, but at the expense of multiple parallel states
however.
3.4. Discussion
We first showed in simulation that both a parallel and the serial models with one
fast and one slow process with multiple states can reproduce previous single- adaptation
data of spontaneous recovery, anterograde interference, and dual–adaptation data in both
intermittent and random schedules. Then, using an experimental dual-adaptation
paradigm, we showed that only a model architecture in which one fast and the slow
processes with multiple states are arranged in parallel provides a parsimonious
explanation for our data. This model furthermore accounts for detailed characteristics of
savings in relearning data. Our combined simulation and experimental analysis thus
support the view that human motor memory has three characteristics during motor
adaptation: (1) It contains a single fast-learning fast-forgetting process. (2) It contains a
slow process with multiple slow-learning slow-forgetting states, all with the same
learning rates and the same forgetting rates; these states are switched with contextual cues.
(3) The two processes are arranged in parallel and compete for errors during motor
adaptation.
Our model, unlike any previous models, can reproduce all of the following
adaptation data: savings, anterograde interference, spontaneous recovery, and dual-motor
４ ７
adaptation in both intermittent and random schedules. Because the fast process in our
model contains only a single state, the model can account for interferences between
different tasks in the experiments paradigms of savings (Kojima et al., 2004), anterograde
interference (Miall et al., 2004) and spontaneous recovery (Smith et al., 2006). In all
these cases, interferences were observed strongly at the beginning of task alternations
(Tong et al., 2002; Miall et al., 2004; Imamizu et al., 2007), when the fast process is the
most active. In contrast, because of the lack of contextual-independent process, the two n-
fast n-slow models and the parallel n-model cannot reproduce such data. Because the
slow process in our model contains multiple states switched via a contextual cue input,
our model explains dual or multiple motor adaptations (Shelhamer et al., 2005; Nozaki et
al., 2006; Imamizu et al., 2007; Choi et al., 2008; Howard et al., 2008): During learning
of different tasks, a separate state stores the learning for each task. Thus, in our model, as
has been recently reported in humans (Criscimagna-Hemminger and Shadmehr, 2008),
learning a new task does not alter the memory of a previously learned task, but produces
a new memory. In contrast, because of the lack of context-dependent multiple states, two
state models cannot account for dual-adaptation, because introducing a new task unlearns
the other task.
Our multi-state models can also differentiate between serial and parallel
organization of the fast and slow processes, because of the non-linearity in the slow
process arising from multiplying the motor error input by the contextual input (see Figure
3.2 and Equation (5) in Method). When the contextual input changes frequently, as it does
in random schedules, this non-linearity in the slow process makes the parallel model learn
differently from the serial model. Based on such different learning predictions of parallel
４ ８
and serial models in the random schedule, we found that our experimental data was better
supported by a parallel architecture.
To explain savings in re-learning data after variable number of wash-out trials,
varying-parameter models (Zarahn et al., 2008) requires continuous adaptation of the
parameters (i.e., meta-learning, e.g., (Schweighofer and Doya, 2003). Instead, our parallel
1-fast n-slow model uses multiple states in a slow process and can reproduce savings in
wash-out paradigm only with four free parameters. During wash-out trials, the net model
output returns close to the initial, non-adapted, condition because the fast process returns
to the initial state and the slow process is switched to the no-perturbation state based on
the given context. In the re-learning condition, the slow process corresponding to this
perturbation switched back to the previously adapted state, allowing savings. Because
both our model and the varying parameter model of Zarahn et al. (2008) reproduce
equally well these savings in relearning data, more detailed analyses with yet to be
devised experimental protocols are needed to differentiate between models. Note
however, that multiple learning and forgetting rates are needed to explain adaptation in
situations that we did not consider here – adaptation at largely different time scales such
as changing dynamics due to aging (Kording et al., 2007), and adaptation after a
consolidation (rest) phases (Criscimagna-Hemminger and Shadmehr, 2008); see also
(Fusi et al., 2007).
A number of brain areas and neuronal architectures are possibly engaged in slow
and fast processes during motor adaptation. These areas include the cerebellar nucleus
and the cerebellar cortex (Medina et al., 2002), as well as two cell types in the primary
motor cortex (M1) (Li et al., 2001) (see discussion in (Smith et al., 2006). Our 1-fast n-
４ ９
slow model further predicts that two separate cell populations learn from the same errors,
but at two different timescales. A possible candidate area for the locus of the fast process
is the posterior parietal cortex (PPC). The PPC is reported as maintaining the internal
representation of the body's state in visuomotor adaptation (Wolpert et al., 1998). Area 5
is known to receive motor errors (Kawashima et al., 1995; Diedrichsen et al., 2005) and
PPC learning-related activation decreases during the later stage of visuo-motor adaptation
(Graydon et al., 2005) (but see also (Della-Maggiore et al., 2004). A possible candidate
area for the locus of the slow processes with multiples states is the cerebellum, which
contributes to state estimation in visuomotor adaptation (Miall et al., 2007), increases
learning-related activation during the later stage of visuo-motor adaptation (Imamizu et
al., 2000; Graydon et al., 2005) (but see (Tseng et al., 2007), and receives motor errors
(Gilbert and Thach, 1977; Kawashima et al., 1995; Schweighofer et al., 2004;
Diedrichsen et al., 2005). Furthermore, functional imaging studies have revealed that the
cerebellum is involved in the modular organization of multiple states (Imamizu et al.,
2003; Imamizu et al., 2004; Imamizu and Kawato, 2008).
Because of its simplicity, our proposed 1-fast n-slow parallel model inevitably
suffers from a number of limitations: First, we used an artificial switch to select the
appropriate slow processes based on context. More realistic, automatic, and adaptive
contextual switch performances have been proposed (Wolpert and Kawato, 1998; Haruno
et al., 2001). Second, our model does not account for generalization across tasks. In our
dual-adaptation experiment, subjects learned task D (50 degrees) better than task C (-50
degrees) (Figure 3.6). This may be due to a greater transfer of learning from task A (25
degrees) to D, than from task B (-25 degrees) to C, as task A was given 50 more trials
５ ０
than task B. Our model could be extended to account for such generalization using
contextual cue inputs with a tuning-curve across tasks in the slow process (Thoroughman
and Taylor, 2005). Third, our model is only a model during motor adaptation and does not
account for consolidation following learning (Criscimagna-Hemminger and Shadmehr,
2008) or adaptation at longer time scales (Kording et al., 2007). Consolidation in
particular is well accounted for by a serial model. Thus, the picture emerges that during
adaptation, motor memory is organized in parallel, with one fast and multiple slow
processes competing for errors; during consolidation however, when the system is not
active, transfer of learning occurs serially. Finally, our model was inferred from
behavioral data only. It thus awaits confirmation from neural recording or from brain
imaging and virtual lesions using transcranial magnetic stimulation.
５ １
Chapter 4.
Mechanisms of the contextual interferences in individuals post-
stroke
4.1. Introduction
During neuro-rehabilitation after brain injury, also in activities such as sports,
technical training, and music, one must often learn, or re-learn, multiple motor tasks
within a given period. Intermixing the learning of different tasks via random schedules
reduces performance during training, but enhances long-term retention compared to
blocked schedules, e.g., (Pyle, 1919; Shea and Morgan, 1979; Tsuitsui et al., 1998;
Schmidt and Lee, 1999).
Although this contextual interference (CI) effect appears to be robust in motor
learning in healthy subjects, two previous studies with individuals recovering from stroke
affecting the motor system reported inconclusive results. (Hanlon, 1996) reported a CI
effect; however, the effect of schedules could not be unequivocally determined in this
study, because the learning sessions were of variable lengths. On the contrary, (Cauraugh
and Kim, 2003) reported no CI effect; however, the tasks used in this study were
strengthening tasks (such as wrist/finger extension), not goal-directed tasks that required
motor learning.
A pre-requisite for the CI effect is the ability to learn and retain motor skills. We
previously showed that unilateral stroke-related damage in the sensori-motor areas
５ ２
primarily affects the control and execution of motor skills, but not the learning and
retention of those skills (Winstein et al., 1999). However, the integrity of short-term
memory in individuals post-stroke may play a role in the CI effect in individuals with
stroke. According to the “forgetting-reconstruction” hypothesis of the CI effect, forgetting
between successive presentations of the same task during random training requires the
learner to “reconstruct the action plan at each presentation:, resulting in stronger memory
representations (Lee and Magill, 1983; Lee et al., 1985).
Recent computational models further suggest that rapid forgetting between
successive presentations of the same task during training may play a role in the CI effect.
It has notably been proposed that motor adaptation occurs via simultaneous update of a
fast process that contributes to fast initial learning but forgets quickly, and a slow process
store that contributes to long-term retention but learns slowly (Smith et al., 2006). We
recently extended this model to account for multiple task adaptation (Lee and
Schweighofer, 2009). In our model, a single fast process store is arranged in parallel with
multiple independent slow processes switched via contextual cues. During adaptation,
fast and slow processes are updated simultaneously from the same motor errors in a
competitive manner.
We have three related goals in the present study. First, using our previous
computational model (Lee and Schweighofer, 2009), we provide a mechanistic
explanation of the CI effect, and notably show how the fast process affects long-term
retention after blocked but not random schedules, and therefore modulates the CI effect.
Second, using an experiment in which subjects learn how to produce specific force
patterns, we contrast the effects blocked and random scheduling in healthy individuals
５ ３
and in individuals with stroke affecting primarily the motor system. Third, we test the
prediction derived from our model that the CI effect is modulated by the integrity of
short-term memory: accordingly, the CI effect should hold for healthy subjects and for
individuals with stroke with good visuo-spatial short-term memory, but not for
individuals with poor short-term memory.
4.2. Methods
4.2.1. Participants
The participants came to our laboratory for two sessions on two consecutive days.
The first session consisted of physical and cognitive tests, a training session, and an
immediate retention test (see below). A delayed retention tests was given 24 hours
following the first (immediate) retention test.
Healthy subjects. Twenty-four participants (12 females) with no reported
neurological deficits were randomly assigned either to a blocked training schedule or to a
pseudo-random training schedule (N = 12 in each group; because of problems with the
recording device on day with 1 subject, we analyzed data for 11 subjects in the random
group and 12 in the blocked group for the retention results). To be included in the study,
the participants needed to fulfill the following inclusion criterion: be over 18 years of age
and be right handed. Summary of the demographic data is reported in Table 4.1.
５ ４
Characteristics Blocked (n=12) Random (n=12) p-value
Men 6 6 p > 0.1
Age (yr) 25.5 ± 1.86 26.7± 2.65 p > 0.1
Power force (N) 863 ± 99 794 ± 111 p > 0.1
Wechsler Figural(10) 8.58 ± 0.29 8.33 ±0.33 p > 0.1
Table 4.1: Characteristics of the healthy individuals. * p< 0.05.
Individuals post-stroke. Twenty-four participants (7 females) with stroke at least
three months from onset were randomly assigned either to a blocked training schedule or
to a pseudo-random training schedule (N = 12 in each group). To be included in the study,
the participants needed to fulfill the following inclusion criteria: 1) be over 18 years of
age; 2) be at least 3 months post stroke; 3) be stable medical condition; 4) have a upper
extremity Fugl-Meyer score no less than 33 (Fugl-Meyer et al., 1975); 5) be able to
produce at least 10 Newtons of force in power and 2 Newtons in precision grasp; 6) have
a score at least 10 out of 17 on the Functional test of the Hemiparetic Upper Extremity
(FTHUE) (Wilson et al., 1984); 7) have a score of no less than 25 on a mini mental state
exam (MMSE). The participants were excluded from the study if they 1) demonstrated
excessive pain in any joint of the more affected extremity that could limit ability to
participate in the grasping tasks; 2) had any previous history of surgery of fracture in the
affected extremity that may impair the ability to perform the task. Summary of the
demographic data is reported in Table 4.2.
５ ５
Characteristics Blocked (n=12) Random(n=12) p-value
Men 8 9 0.51
Age (yr) 61.25±13.92 54.58±13.39 0.24
Left Hemiparesis 6 4 0.25
Concordance 7 5 0.24
Time post onset
(mo)
39.25±16.45 25.83±22.32 0.11
Power force
(kgN)
462 ± 67 326 ± 29 0.082
UEFugl-Meyer
ROM (24)
22.58±1.83
22.25±1.76
0.65
Pain (24) 24.00±0.00 23.25±2.30 0.27
Sensory (12) 12.00±0.00 11.00±2.23 0.17
Motor (66) 55.00±8.98 54.00±7.01 0.65
Wrist (10) 9.10±2.81 8.00±3.28 0.55
Hand (14) 12.25±1.54 11.58±2.31 0.41
FTHUE 14.50± 3.53 14.91± 2.94 0.76
MMSE (30) 29.25±1.21 29.41±0.90 0.71
Wechsler
Digital (24)
16.83±4.71
16.08±3.17
0.65
Figural(10) 6.16±1.69 6.67±1.97 0.51
Table 4.2: Characteristics of the individuals post-stroke.
4.2.2. Learning Tasks
A critical impairment after stroke is an inability to generate precise force output at
moderate levels of force with sufficiently rapid force rise. We thus designed motor tasks
that required learning to exert specific force patterns with unique magnitude and timing
according to visual inputs, in either a blocked or a random schedule condition. At each
trial, the participants were instructed to reach and grasp a plastic cylinder, and exert a
force profile with a power grasp that matched one of three target profiles displayed on a
computer screen in both magnitude and timing (Figure 4.1). Participants were instructed
５ ６
to exert a force profile that matches one of three target profiles displayed on a computer
screen in both magnitude and timing. The force sensor acquired data at 100 Hz.
All participants performed 150 training trials. In the blocked schedule condition,
each task was practiced separately in a single block of 50 consecutive repetitions. Task
order was randomized across participants in the blocked schedule condition. In the
random schedule condition, there were 50 blocks of three trials, with each task occurring
once per block at a random position within the block. At the onset of each trial, one of the
three target force profiles was shown. Two seconds later, a “GO” signal was displayed,
and the participants were instructed to produce a force trajectory that matched the target
force profile. Four seconds later, the actual force trajectory was shown superimposed on
the desired trajectory (Figure 4.1). An error value was also displayed indicating the total
root mean squared error (RMSE) between the desired and actual force trajectory. If the
participant did not grasp the cylinder within the 2 seconds of the “GO” signal, the
message “Next time move faster” was displayed. Participant post-stroke were instructed
to use their affected hand. Healthy subjects were instructed to use their dominant hand.
５ ７
Figure 4.1. Motor learning tasks. At each trial, one of three target force profiles was
shown during the “ready” period according to a predetermined schedule (random or
blocked). Two seconds later, a “GO” signal appeared and the subject was instructed to
reach and grasp the apparatus and modulate the power grip force to approximate the
target profile that lasted 2 seconds. The computer screen then became blank for 4 seconds,
and feedback was shown for 4 seconds. Feedback included the actual force profile
superimposed with the desired profile, the root mean squared error (RMSE) between the
two profiles, as a well as a “best score”, which reflected the smallest error so far, and was
included for motivational purposes.
To assess performance after training, retention tests were given immediately and
24-hour following training. Each test consisted of five trials per task, with each trial
similar to training trials, but without feedback. In the immediate test of the blocked
schedule group, the test for each task was given immediately following the 50 training
trial. In the immediate test of the blocked schedule group, the test for each task was given
５ ８
immediately following all 150 training trials. In delayed retention tests, as well as in the
two tests for the random schedule, the order of the three tasks was randomized across
participants.
For each participant, the maximal force exerted in three trials was recorded with the
apparatus before training. The maximum magnitude of each target force profile was set at
40% of maximum force such as all subjects could learn the tasks.
Figure 4.2 shows examples of force profiles and the actual force trajectory for the
first task presented for a subject in each group (healthy blocked, healthy random, stroke
blocked, and stroke random) for the first trial of training, and for the first trial of the
immediate and delayed retention tests.
５ ９
Figure 4.2: Examples of force trajectories for four subjects for 1 task. From left to right:
first trial of training, first trial of immediate retention test, and first trial of delayed test.
From top to bottom: subject post-stroke in blocked schedule, subject post-stroke in
random schedule, healthy subject in blocked schedule, and healthy subject in random
schedule. Thick line: desired force profile. Thin line: actual force (in N) exerted by the
subject during 2 seconds.
4.2.3. Short-term memory tests
In our task, subjects learned to match a visually displayed force profiles to a
desired target from delayed (4 seconds) feedback; thus visual short-term memory is
needed to link the visual information provided as cue and feedback to the actual
movement generated. We assessed visual short-term memory for each participant with the
figural memory subtest of the Wechsler Memory Test revised (WMS-R; (Wechsler,
６ ０
1987)). In the figural test, a target geometrical pattern is presented to the subject, and then
both the target pattern and alternative patterns are shown after 6 seconds. The participant
must identify the correct pattern. The maximum score for the figural test is 10. The
figural test has been used in previous research studies with various clinical populations to
assess visual-short-term memory, e.g., (Nixon et al., 1987; Hawkins et al., 1997). The
digital memory subtest of the Wechsler Memory Test was also given.
4.2.4. Data analysis
Because our main purpose was to compare relative forgetting following training in
two schedule conditions and between healthy and post-stroke individuals, we computed
performance for each task at each trial as 1 minus a normalized RMSE averaged. To
normalize the RMSE for each task, we divided the RMSE by the difference between the
maximum RMSE (often, but not always on the first trial) and minimum RMSE generated
during training.
Our main dependent measure was “forgetting”. Forgetting was computed as the
difference between the immediate and the 24-hour retention test in the average
performance in the first trial of each task (we did not use the 5 trials of each task in each
test because we noted that performance largely improved during the test even without
feedback in the healthy subject group; e.g., in immediate group, average of 3 tasks,
repeated ANOV A for the 5 test trials, p < 0.001 in blocked condition, p = 0.04 in the
random condition). To analyze change of performance during training, we used repeated
ANOVA and computed the mean performance in blocks of five trials for each task and
then over the three tasks (unless otherwise noted)
６ １
We classified the individuals with stroke as “high” and “low” function visuo-spatial
working memory, based on a cut-off of 7 on the figural Wechsler score (the cut-off value
was chosen because it yielded near equal group sizes in our data set) and compared the
effect of schedules for high and low level groups.
Because of the inconclusive reports of the CI effect in individuals post-stroke in
previous studies, we used 2-tail tests. All data (and residuals for regression models) used
were tested for normality with the Shapiro-Wilks' test, and for equality of variance with
the Levene’s test. When comparing 2 groups, when the data were normally distributed
and when the sample size was greater or equal 10 in each group, we used t-tests for
independent measures, and paired t-tests for repeated measures. Otherwise, we used the 2
sample Kolmogorov-Smirnov Z test for independent measures, and the Wilcoxon signed
ranked Test for repeated measures. In repeated ANOV A, non-spherical data was corrected
using Greenhouse-Geisser correction. Unless indicated, the size of each group was N =
12. Difference between groups in subjects baseline characteristics were tested with t-tests
when the data was numerical (e.g., age) and with chi-square tests when the data was
categorical (e.g., gender). Our significance level in all tests is set at p < 0.05.
4.2.5. Computational model
Our model(Lee and Schweighofer, 2009), unlike previous model of motor
adaptation, has the potential to simulate the change in performance in random or blocked
schedules because it was developed specifically to account for multiple task adaptation.
The model contains a common a fast process and multiple slow processes all organized in
parallel. The state update rules are given by:
６ ２
=
+
(1)
+1 =
.
+
.
+1 =
.
+
. (2)
where n is the trial number, the motor error determined by the difference between an
external perturbation f(n) and the motor output y(n) at time step n, the state of the fast
process, the state vector of the slow process, and c the contextual cue vector, with both
vectors of length equal to the number of tasks. Because, we assumed no interference and
perfect switching among states in the slow process in the model, c is a unit vector with a
single non zero element. We refer the reader to (Lee and Schweighofer, 2009) for
additional details of the model. Default parameters in the model in the current simulations
were Af = 0.8; Bf = 0.2; As = 0.995; Bs= 0.04.
To compare computer simulations to data in our current experiment, adaptation
level at the end of training in the model was taken as immediate retention performance
for the first task. Long-term retention was computed as the long-term memory level at the
end of training (see (Joiner and Smith, 2008) for rationale).
We made the simplifying assumption that, in the model, the integrity of short-term
memory is linked to the rate of decay of the fast process, with larger rates of decay linked
to poorer short-term memory. We thus modeled the integrity of short-term memory by
modulating the time constant τf in the fast process, with τf = 1/(1- f A )*T, where Af is
given in equation (2) and T corresponds to a simulated trial length, which was 12 seconds
as in the experiment: the default “normal” Af value gives τf = 60 sec. The default poor
short-term memory value was Af = 0.4, which gives τf = 20 sec.
６ ３
4.3. Results
4.3.1. The CI effect in the computational model
Our computational model reproduced the CI effect: slower improvements in
performance during training and less forgetting following training in random schedules
than in blocked schedules (Figure 4.3). During blocked presentations, the fast process
exhibited high activity levels because there was little forgetting during the short intervals
between presentations of the same task. Because fast and slow processes compete for
errors, there was relatively little update of the fast process (Figure 4.3A). As a result,
there was fast adaptation during training, but large forgetting in delayed retention (Figure
4.3B). In contrast, during random presentations, the fast process exhibited low and
jittered activity levels, both because of interferences between tasks and because of
passage of time between presentations of the same task (Figure 4.3C). This led to
relatively large update of the slow process. As a result, there was relatively slow
adaptation during training, but minimal forgetting in delayed retention (Figure 4.3D,
compare to Figure 4.3B).
６ ４
Figure 4.3. Computer simulations: CI effect in the ”healthy " model. Performance (blue
line), short-term memory (black line), and long-term memory (red line) during blocked
(A) and random schedule training (C). Immediate and long-term retention following
blocked (B) and random schedule (D) training. Imm: Immediate test. Del: 24 hour
delayed test. Notice the large forgetting following blocked training (arrow ”1”) and
slower performance improvement during random than during block training (arrow ”2”).
The jitter in short-term memory (reflected in performance) during random training in C
was due to both interferences between the tasks and decay.
Our model makes the prediction that the integrity of short-term memory
differentially affects delayed retention in blocked and random schedules. In Figure 4.4, a
simulated subject with poor short-term memory (parameter Af = 0.4) showed little long-
term forgetting following either blocked or random schedule training (Figure 4.4 B,D).
During blocked schedule training, compared to the simulated subject with normal short-
６ ５
term memory of Figure 4.4, there was reduced update of the fast process because of large
forgetting from one trial to the next. However, because of the competition between fast
and slow process, there was an enhanced update of the slow process (Figure 4.4A).
Forgetting following blocked training was then minimal (Figure 4.4B). During random
schedule training, the update of the slow process was nearly identical to that of the
“normal” simulated subject (Figure 4.4D).
６ ６
Figure 4.4. Computer simulations: reduced CI effect in a model with „poor short-term
memory ‟. Performance (blue line), short-term memory (black line), and long-term
memory (red line) during blocked (A) and random schedule training (C). Immediate and
long-term retention following blocked (B) and random schedule (D) training. Imm:
Immediate test. Del: 24 hour delayed test. In blocked schedules, there was little forgetting
in the delayed retention test (B; arrow ”1”; compare with Figure 3B) because of relatively
higher build-up of long-term memory and lower build-up of short-term memory during
training (A; compare with Figure 4.3A). In random schedules, there was little difference
during training (C) and testing (D) compared to the normal model (compare with Figure
4.3C, D).
In the model, forgetting following blocked schedule training positively correlates
with the time constant of short-term memory: for small time constants (i.e., poor short-
term memory), there was little forgetting in the delayed test (Figure 4.5 A left), because
６ ７
much forgetting happened between presentations during training (as in Figure 4.4A). For
larger time constants (i.e., good short-term memory), there was large forgetting in the
delayed test (Figure 4.5A right), because little forgetting happened between presentations
during training (as in Figure 4.3A). In contrast, forgetting following random schedule
training did not correlate with this time constant (Figure 4.5B): because of interferences
between tasks, there was little build-up of fast process during training, independently of
the time constant of short-term memory.
Figure 4.5. Computer simulations: Forgetting as a function of the time constant of decay
of short-term memory after either blocked (A) or random (B) schedule for two tasks.
Overall, and notably for larger time constants, there is less forgetting following random
schedule than blocked schedule.
4.3.2 The CI effect in healthy individuals
There was no difference in baseline characteristics between the two healthy groups
for gender, age, power-grip maximal force, pinch-grip maximal force, and Wechsler
figural score in healthy subjects (see table 1).
６ ８
Forgetting was positive following blocked training (Blocked forgetting: 0.183 ±
0.063; p = 0.011 one-sample t-test), but not following random training (forgetting: -
0.0004 ± 0.041; random: p = 0.99, one-sample t-test). Furthermore, forgetting was greater
following blocked schedule than following random training (p = 0.023, t-test) (see Figure
4.6C and D).
６ ９
Figure 4.6. Data: CI effect in healthy participants. Performance (mean and SE) during
blocked (A) and random (C) training. Immediate and long-term retention following
blocked (B) and random (D) training. The * in B indicates significant difference between
immediate and delayed retention test in the blocked schedule; the * in C indicates
significant continuous improvements in performance from trials 5 to 50 in the random
schedule (see text).
The performance of subjects in both groups improved during training in both
conditions (repeated measure ANOV A, 10 blocks of 5 trials, p = < 0.0001 for both
blocked and for random groups) (Figure 4.6A and C). Furthermore, there was no
difference in performance in the first block of 5 trials between groups (p = 0.129, t-test),
７ ０
no difference in the last block of 5 trials (p = 0.233, t-test), and no difference in the
overall change in performance between the first and the last block of 5 trials (p = 0.763, t-
test). However, performance improved faster in the blocked group compared to the
random group after the first 5 trials. To show this, we regressed the performance from
trials 6 to 50 as a function of trial number for each subject: the slopes were larger in the
random group (0.0051 ± 0.0006 trial
-1
) than in the blocked group (mean 0. 0015 ± 0.0011
trial
-1
; p = 0.008, t-test).
In sum, healthy subjects in our experiment exhibited the two hallmarks of the CI
effect: greater forgetting following random training compared to blocked training and
slower improvement in performance during random training than during blocked training.
4.3.3. The CI effect in individuals post-stroke
There was no difference in baseline characteristics between the two groups for
gender, age, side of paresis, concordance of stroke (i.e. whether the stroke affected the
dominant hand), time post-onset, maximal force, any of the Fugl-Meyer upper extremity
subscale scores (range of motion, pain, sensory, arm motor, hand, wrist), FTHUE, MMSE,
and Wechsler figural score (all values are given in Table 2). The only difference between
groups was power-grip maximal force (blocked: 20. 10±9.79 Newton; random:
11.77±3.50 Newton, p = 0.011; t-test).
Forgetting was marginally positive following blocked training, but not following
random training (blocked 0.076 ± 0.040, p = 0.085, random: p = 0.89, one sample t-test)
(see Figure 7C and D). We verified that there was no correlation between maximal power
grip force and forgetting in either group (blocked: p = 0.97; random: r = - 0.75; Pearson).
７ １
Figure 4.7. Data: CI effect in participants post-stroke. Performance (mean and SE)
during blocked (A) and random (C) training. Immediate and long-term retention
following blocked (B) and random (D) training. The * in B indicates significant
difference between immediate and delayed retention test in the blocked schedule; the * in
C indicates significant continuous improvements in performance from trials 5 to 50 in the
random schedule (see text).
Our simulations predict that the short-term memory would influence the degree of
forgetting after blocked schedules, but not after random schedules. We test this prediction
by with a repeated measure ANOVA with Time = [Immediate test, Delayed test] as the
repeated factor, and the figural Wechsler score as the covariate. Forgetting was positive
７ ２
(blocked 0.076 ± 0.029, p = 0.024) and the effect of figural Wechsler score covariate
extremely significant (p = 0.0067). Figure 4.8 illustrates the strong dependency of the
figural Wechsler score upon forgetting in the blocked schedule. A similar analysis in the
random schedule show no effect of the figural Wechsler score upon forgetting in the
random schedule (p = 0.56), again as predicted by our computer simulations.
The influence of visuo-spatial working memory on forgetting is well illustrated by
classifying subjects between high and low spatial working memories based on a cut-off of
7 on the figural Wechsler memory score. The difference between forgetting in blocked
and random schedule was significant in the high spatial working memory sub-groups (N
= 5 high spatial working memory in blocked, N = 8 high spatial working memory in
random, 2 sample Kolmogorov-Smirnov Z Test, 1-tail, p = 0.039). However, there was no
difference in forgetting in the low spatial working memory sub-groups, however (2
sample Kolmogorov-Smirnov Z Test, 1-tail, p = 0.997).
７ ３
Figure 4.8. Data: Forgetting in individuals post-stroke in a 24 hour post-training period
as a function of Wechsler visual memory score (figural) following training in either
blocked (A) or random schedule (B) in the individuals post-strokes who participated in
the study. Compare to the computer simulations of Figure 4.5.
The performance of subjects in both groups improved during training in both
conditions (repeated measure ANOV A, 10 blocks of 5 trials, p < 0.0001 for both groups;
Greenhouse-Geisser corrections) (Figure 4.7A and C). There was no difference in
performance in the first trial between groups (p = 0.1257, t-test). Furthermore, there was
no difference in performance in the first block of 5 trials between groups (p = 0.129, t-
test), no difference in the last block of 5 trials (p = 0.67, t-test), and no difference in the
overall change in performance between the first and the last block of 5 trials (p = 0.187, t-
test). There was no difference in slopes between group in the regression model of
performance vs. trials 6 to 50 (p = 0 0.6305, t-test – see above for details).
Finally, we verified that, in the 24 individuals post stroke enrolled in the study,
there was no correlation between the figural Wechsler memory score and any motor and
７ ４
sensory variables (maximum power force p = 0.482, UE Fugl-Meyer ROM p = 0.480, UE
Fugl-Meyer Pain p = 0.825, UE Fugl-Meyer Sensory p = 0.316), UE Fugl-Meyer
Motor p= 0.839), UE Fugl-Meyer Wrist p = 0.330, UEFugl-Meyer Hand (p =0.615),
FTHUE p = 0.839). There was no correlation with the digital Wechsler memory score
(p > 0.87). The only significant correlation between the figural score was with the MMSE
(r = 0.70, p <0.0005). These results are important for two reasons. First, since the MMSE
contains short-term memory component, this validate the use of the figural Wechsley as a
short-term memory test in our study. Second, the results show that it is figural short-term
memory, not any motor or sensory impairment after stroke that leads to the observed
changes in forgetting.
In sum, individual post-stroke in our experiment exhibited reduced contextual
interference effect compared to healthy individual, but this effect was mediated by figural
Wechsler score: following training in a blocked schedule, individuals with figural
Wechsler score comparable to those of healthy subjects exhibited large forgetting but
those individuals with low figural Wechsler score subjects exhibited no or little forgetting.
4.4. Discussion
Here, we replicated the CI effect in healthy subjects, and presented novel evidence
that the CI effect can hold in participants with stroke affecting the motor system, but is
modulated by visuo-spatial short-term memory. Our results indicate that individuals with
stroke with normal visual short-term memory, like healthy subjects, exhibit less forgetting
of visuo-motor skills acquired in random training schedules compared to blocked
７ ５
schedules. In contrast, individuals with stroke with low visual short-term memory exhibit
little forgetting after either random or blocked schedules. No other motor or sensory
variables correlated with forgetting after training. The effect of short-term memory on the
CI effect was predicted by our computational model of motor memory in which different
tasks interfere in a common short-term memory store, but are learned separately in
different long-term memory stores (Lee and Schweighofer, 2009).
Although the CI effect has led to a considerable amount of research(Magill and
Hall, 1990), its underlying mechanism has remained unclear. Three non-exclusive
hypotheses have been proposed to explain the CI effect in motor learning. First,
according to the “elaboration-distinctiveness” hypothesis, random training schedules
allow inter-task comparison during the planning stage that lead to distinctive memory
representations; e.g, (Shea and Morgan, 1979; Immink and Wright, 2001; Lin et al., 2008;
Cross et al., 2009). Second, according to the “deficient processing” hypothesis, blocked
repetitions lead to reduced rehearsal or attention in later presentations, e.g., (Hintzman et
al., 1973; Callan and Schweighofer, 2010). Third, according to the “forgetting-
reconstruction” hypothesis of the CI effect, forgetting between successive presentations
of the same task during random training results in stronger memory representations (Lee
and Magill 1983; Lee et al. 1985).
Here, we proposed a novel mechanistic account of the CI effect in motor learning.
As in the “forgetting-reconstruction” hypothesis, our account of the CI effect relies on
forgetting between presentations of the same task during training. However, the specific
mechanisms underlying the enhancement of long-term memory differ in the forgetting-
reconstruction hypothesis and in our model. According to the forgetting-reconstruction
７ ６
hypothesis, forgetting in short-term memory between spaced presentations necessitates
retrieval from long-term memory, which increases long-term retention. In our model,
forgetting in short-term memory between spaced presentations during training leads to
slower improvements in performance in random schedule than in blocked schedules. The
resulting greater errors during training benefit the update of the slow process, leading to
higher long-term retention. Note that our account of the CI effect does not exclude
additional explanations such as the “elaboration-distinctiveness” and the “deficient
processing” theories; further studies are needed to dissociate the possible contributions of
such mechanisms and to determine the putative neural substrates. For example, we
recently showed that the neural substrates of memory consolidation depend on practice
structure (Kantak et al., 2010).
Our results have implications for rehabilitation of patients with stroke. Although
there is good evidence that task training is effective for improving upper extremity
function after stroke (Butefisch et al., 1995; Kwakkel et al., 1999; Wolf et al., 2002; Wolf
et al., 2006), none of these studies addresses the “microscheduling” of individual tasks.
Thus, although the use of task schedule during training has been advocated (Krakauer,
2006), physical and occupational therapists rely on guidelines that simply suggest
inclusion of extensive and variable training; e.g., (Bach-y-Rita and Balliet, 1987; Lee et
al., 1991). Here, our results suggest that patients with stroke with normal visual short-
term memory should receive a training schedule that mixes tasks randomly. Because such
training is hard to implement in normal therapeutic situations, and places high demand on
the therapists, robots that can present functional tasks and can easily switch between
these tasks during training, could be developed - see our preliminary work in this
７ ７
direction (Choi et al., 2008). On the contrary, and in a rather counter-intuitive manner, our
results suggest that individuals with stroke with poor visual short-term memory who
practice in blocked schedules will exhibit as good long-term retention of motor skills as if
they had practiced in random schedules.
Our study has a number of limitations that need to be addressed in future work.
First, although our demonstration of the CI effect in individuals with stroke corroborate a
previous study (Hanlon, 1996), our study included only a relatively small number of
subjects (12) in each group, and needs replication with a larger sample size and different
populations, such as the elderly - see (Anguera et al., 2010). Second, a prediction of the
model is that poor short-term memory will reduce the rate of performance improvement
during acquisition in blocked practice. We found however, no correlation between the
figural memory and the rate of learning in the stroke group. This can be due to the large
variability between and within subjects at each trial during training in the stroke group
compared to the healthy group (compare the shaded areas of Figures 4.6A and 4.7A).
Increasing the number of subjects in learning tasks that lead to less trial by trial
variability may be desirable. Third, in our experiment, we gave a single test of visual
short-term memory, and in our computer simulations, we simply varied the rate of decay
of the short-term memory to mimic deficits in short-term memory. Since short-term
memory is thought to have a limited capacity, e.g., (Cowan, 2001), it is possible that, in
addition to, or in lieu of, shortened time-span, limited visual short-term memory in our
participants is due to a deficit in the number of items that can be stored in memory. Our
current model of motor memory cannot account for such limited storage mechanism.
Fourth, because we were only able to obtain the MRI scans for a small subset of our
７ ８
participants, we cannot conclude on the neural basis of the motor and visual working
memory deficits.
A final limitation is due to the simplicity of our model. Like others, e.g. (Smith et
al., 2006; Kording et al., 2007), only attempted to model the common neuronal
mechanism of error-driven motor adaptation or learning, but not the mechanism for
specific types of motor adaptation or learning. As such, the model accounts for dual-
adaptation data in eye movement, visuo-motor rotation, and force field adaptation – see
(Lee and Schweighofer, 2009). Therefore, our model does not account for the effect of
physiological factors, such as muscle mechanics, limb dynamics, etc, of our specific
experimental tasks. Also note that while this is a model of motor adaptation (where
“adaptation” is the change in motor performance that allows the motor system to regain
its former capabilities in altered circumstances), we used it here to account for visuo-
motor learning of motor learning in our experiment novel tasks. Such extension from
adaptation to motor learning requires us to make three assumptions about motor learning
in our experiment. The first is that learning in our experiment is error-driven. Since we
provided an error measure in the form of the actual force trajectory superimposed on the
desired trajectory (Figure 4.1), this appears reasonable. In addition, there is also a large
body of evidence that support the fact that skill learning are at least in part driven via
error reduction (reviewed for instance in (Hikosaka et al., 2002). The second assumption
is that skill learning, like motor adaptation, results from a combination of fast and slow
memory processes; there is a large body of evidence that supports this view (Anguera et
al., 2009). Finally, our model also assumes that there is no interference/generalization
between tasks in the long-term memory store. Thus, our model cannot reproduce any
７ ９
transfer of learning data. This is clearly an over simplification in light of the similarities
between tasks in our experiment and this prevented us to use the data to estimate the
model parameters. However, our simulations results show, post-hoc, that our model well
accounts for the CI effect when performance across tasks are averaged. Thus, while more
complex models (i.e. with larger number of parameters) that include cross-terms between
states in the slow process may be warranted to account for change of performance in
individual tasks, the current model is appropriate to model average task behavior.
In sum, our combined theoretical and experimental results suggest a relationship
between the integrity of short-term memory and motor learning. Such a relationship has
been previously reported in a number of recent studies, e.g., (Boyd and Winstein, 2004;
Brown and Robertson, 2007; Anguera et al., 2009; Bo and Seidler, 2009; Anguera et al.,
2010; Keisler and Shadmehr, 2010), and our results add to this body of work and extend
it by showing that short-term motor memory is involved in the CI effect. Causal evidence
for a role of short-term memory in the CI effect in motor learning may be obtained via
virtual lesion of the areas involved in short-term memory via repetitive transcranial
magnetic stimulation, after performing a fMRI localization task - see related (Tanaka et
al., 2010).
８ ０
Chapter 5.
Optimal schedule in multi-task motor learning
5.1. Introduction
Previous studies showed that the schedules of practice in multiple motor task
learning largely affect long-term retention. For instance random schedules show better
long-term retention than blocked schedules (Shea and Morgan, 1979). We also showed
that in simulation using genetic algorithms, in case of learning two tasks with the same
difficulty and same number of trials per task, the blocked schedule is the worst and the
alternating schedule is the best for long-term retention (Figure 5.1, (Kim et al., 2010)).
Although effective, it is still unclear whether random schedules optimally maximize long-
term retention in general; furthermore these fixed schedules do not take into account
possible differences in task difficulty. Here we ask which practice schedule maximizes
long-term retention performance in multi-task motor adaptation.
Although task selection at each trial could be based on predicted performance on
the next trials (see (Huang et al., 2008) for instance), current performance is a poor
predictor of long-term retention(Joiner and Smith, 2008). Thus, task selections should be
based on long-term retention performance. We previously showed that adaptive schedules
based on performance on a previous delayed retention tests substantially improved
learning compared to scheduling based on current performance (Choi et al., 2008).
However, such scheduling had to be performed postdictively (i.e. after information about
long-term retention was available) and was based on heuristics.
８ １
Here, we show how to find the optimal practice schedule predictively (i.e. before
information about long-term retention is available) in multi-task motor adaptation tasks
based on combined approached of computational models of motor adaptation and optimal
control theory. Our previous computational model of multi-task motor adaptation (Lee
and Schweighofer, 2009), which contain a common short term memory process and
multiple independent long-term memory processes, allows us to predict delayed long-
term retention performance based on current performance for a given practice schedule.
Since finding the optimal schedules via an exhaustive search is computationally
prohibitive, we use control theory to find optimal schedules. Although our adaptation
model is non-linear, the effect of scheduling tasks with our model can be seen as
switching between linear systems; as a result, the maximum principle for switching
dynamics can be applied (Riedinger et al., 2003). Using this combined approached, we
searched for the optimal schedules for learning multiple tasks with equal and different
difficulties.
Our simulations suggest that for learning multiple tasks with equal difficulties, the
alternating schedule is near optimal. When the difference between task difficulties
increases, the alternating schedule is not optimal anymore, but still exhibits good delayed
retention performance due to the combined effect of the passage of time and interferences
between tasks in the short-term memory store. We are currently testing these predictions
in dual motor adaptation experiment in which we compare alternate and optimal
schedules.
８ ２
5.2. Methods
5.2.1. Models of multi-task motor learning
Various multi-task motor learning models make different predictions on long-term
retention performance after learning multiple tasks given a practice schedule and tasks to
learn. Here, we compare three multi-task motor learning models: N-state model, N-fast
N-slow model, and 1-fast N-slow model.
First, the N-state model consists of a single-timescale (compared to multi-timescale
in the other two models) learning process maintaining N states, which are switched by
contextual inputs. Each time when the model practices a task, the learning process selects
one of states and updates it according to the error observed. The equations for the N-state
model are as follows:
, { 1,..., }
, for task
i i
i
x Ax Be i N
y x i
= + ∈
=
Where, x
i
is the state related to task i. A is a forgetting rate, B is a learning rate. e is
an error between the actual and desired outputs. In the N-state model, long-term retention
performance for a certain task i solely depends on the current performance y which is the
direct signature of x
i
.
The N-fast N-slow model consists of two different timescale learning processes: the
fast and slow processes. Similar to the N-state model, each of fast and slow processes has
N states, which are switched by contextual inputs. Each time when the model practices a
task, each learning process selects one of states and updates it according to the error
８ ３
observed. The equations for the N-state model are as follows:
, { 1,..., }
,
, for task
i i
f f f f
i i
s s s s
i i
f s
x A x B e i N
x A x B e
y x x i
= + ∈
= +
= +
where,
and
are the states of the fast and slow processes related to task i
respectively. Af and Bf are the forgetting and learning rates of the fast process. As and Bs
are the forgetting and learning rates of the slow process. e is an error between the actual
and desired output. Contrary to the N-states model, in the N-fast N-slow model, long-
term retention performance for a certain task i depends mostly on
xs_i, the state of the
slow process related to task I, and the current performance y is not a good indicator of
long-term retention performance anymore
The 1-fast N-slow model, like the N-fast N-slow model, consists of the fast and
slow processes. However, unlike the N-fast N-slow model, in the 1-fast N-slow model,
the fast process has a common state to update for any task, and only the slow process has
N states, which are switched by contextual inputs. Each time the model practices a task,
the slow process selects one of states and updates it according to the error observed, and
the fast process updates the common state regardless the task. The equations for the 1-fast
N-slow model are as follows:
, { 1,..., }
,
, for task
f f f f
i i
s s s s
i
f s
x A x B e i N
x A x B e
y x x i
= + ∈
= +
= +
８ ４
where,
and
are the states of the fast and slow processes related to task i
respectively. Af and Bf are the forgetting and learning rates of the fast process. As and Bs
are the forgetting and learning rates of the slow process. e is an error between the actual
and desired output. Similarly to the N-fast N-slow model, in the 1-fast N-slow model,
long-term retention performance for a certain task i depends mostly on
, the state of
the slow process related to task i.
5.2.2. Optimal schedules in multi-task motor learning
The computational models allow us to predict long-term retention performance
given a schedule and tasks to learn, and thus enable to compare different schedules. To
find the optimal schedule, however, we cannot compare all possible schedules, because,
as the number of practice trials increases, computational cost to an exhaustive search
increases exponentially and becomes intractable: For example, if a subject practices 2
tasks in 100 trials, the number of possible states and schedules will be
100 30
2 10 > and it
is impossible to compare all these schedules to find the optimal schedule.
Instead, we used the optimal control theory, which deals with the problem of
finding a control law for a given system such that a certain optimality criterion is
achieved. Specifically we used Pontryagin’s maximum principle (Appendix B) to find the
optimal schedule in the dual-task learning paradigm for the three multi-task motor
learning models described in 5.2.1. Here, for example, we showed how to apply the
maximum principle for the 1-fast N-slow model.
８ ５
The system dynamics according to the 1-fast N-slow model can be written as
follows:
, ,
T
k f k s k k
y x = + x c (19)
k k k
e f y = - (20)
, 1 ,
, 1 ,
f k f f k f k
s k s s k s k k
x A x B e
A B e
+
+
= ⋅ + ⋅
= ⋅ + ⋅ ⋅ x x c
(21)
where, y is a motor output, f is an external perturbation, and e is an error between
the motor output and external perturbation.
f
x
is a fast learning process with a single
state, and
s
x is a slow learning process with
task
N internal states.
f
A
and
s
A are
forgetting rates.
f
B
and
s
B are learning rates. c
is a contextual cue input. For dual-
task learning paradigm,
task
N is 2 and
1 2
( , )
T
s s s
x x = x ,
1 2
( , )
T
c c = c , where for the first
task,
1
1 c = ,
2
0 c = and for the second task,
1
0 c = ,
2
1 c = .
Equation 19 to 21 can be expressed in a more compact form as follows:
1 1 2
( , ) (1 )
k k k k k k k
f u u A u A
+
= = + - x x x x (22)
where,
1
2
1
f
s
s
x
x
x
=
x ,
1
1
1
0
0
0 0 0
0 0 0 1
f f f f
s s s s
s
A
A B B B f
B A B B f
a
=
- -
- -
,
2
2
2
0
0 0 0
0
0 0 0 1
f f f f
s
s s s s
A
A B B B f
A
B A B B f
=
- -
- -
,
８ ６
In equations above, we regard the discrete contextual cue input as a control for the
system dynamics. However, to use the maximum principle, the control should be
continuous. Therefore, for a moment, let’s suppose the continuous control [0,1]
k
u ∈
(We will show that the optimal value for
k
u should be always either 0 or 1 later)
To derive optimal schedules, we need to define the optimality criteria that we want
to achieve: We want to maximize long-term retention performance and learn both tasks
well. Therefore, we use the criteria that minimize the sum of root mean squared errors:
( )
1 :
* 2 2
1: 1 1 2 2
arg min ( ( )) ( ( ))
n
n s s
f x n f x n = - + - u
u
For convenience, we use new variables for slow process states in x as follows:
,
Then,
1
A and
2
A are also changed as follows:
,
The related cost functions are:
Immediate cost:
( , , ) 0 l x u k =
Final cost:
1
( )
2
T
n n n n n
h Q = x x x
'
1 1 1 s s
x x f = - '
2 2 2 s s
x x f = - 1
1 1 1
1 1 1 1 1
2 2 2
0 0
0 ( 1)
0 0 ( 1)
0 0 0 1
f f f
s s s s
s s
A
a b b
b a b a f
a a f
=
- -
- - -
-
2
2 2 2
1 1 1
2 2 2 2 2
0 0
0 0 ( 1)
0 ( 1)
0 0 0 1
f f f
s s
s s s s
A
a b b
a a f
b a b a f
=
- -
-
- - -
８ ７
where,
0 0 0 0
0 1 0 0
0 0 1 0
0 0 0 0
n
Q
=
Total cost J is:
1
0: 0:
0
( , ) ( ) ( , ) ( )
n
n n n k k n
k
J h l h
- =
= + =
∑
x u x x u x
In Pontryagin’s maximum principle, we use two additional terms, Lagrangian
multiplier λ and Hemiltonian H as follows:
( )
1 1
( , , ) ( , ) ( , )
k T
k k k k k k k k
H l f λ λ
+ +
= + x u x u x u
( )
1
( , ) ( , ) , 0
k T
k k k k k k
T
n f n
H l f k n
Q
λ λ
λ
+
∂
= = + ≤ <
∂
=
x x
x u x u
x
x
Then, the conditions for optimal schedules are determined as follows:
( ) ( ) ( )
{ }
1 1 2
2 1 2 2 1 2 2 1 2 1
* ( ) ( ) ( )
1 1 1 2 1
( , ) (1 )
( ) ( ) ( )
arg min ( , , ) arg min ( , ) (1 ) ( , )
k k k k k k k
T
T T
k k k k k k k
k k k
k k k k k k k
u
f u u A u A
A A A u Q A A A u x Q x A A A u
H uH u H
λ λ
+
+
+ + +
= = + - = + - + - - + + - = = + - u
x x x x
u x u λ x λ x λ
Last equation above is the form of:
{ }
* ( ) ( )
1 2
arg max (1 )
k k
k
u
u uH u H = + -
The value inside the bracket is the weighted average of
( )
1
k
H and
( )
2
k
H , and
*
u
is the weight to maximize the average. The weighted average become maximal when the
８ ８
greater between two is weighted all and the smaller is not. Therefore,
*
u should be
either 0 or 1 and the weighted average becomes the greater between two. So, even though
we relaxed the condition for u from the discrete control space of {0,1 } to the
continuous control space of [0,1] , the resulting optimal controls will always belong to
{0,1 }.
The optimal control sequence u can be determined as follows: (1) Given the
initial sequence of controls, we iterate system dynamics to get pairs of states and controls.
(2) With resulting pairs of states and controls, we calculate the Lagrange multipliers. (3)
Then, we find the optimal control sequence to minimize the Hamiltonian given the
sequence of Lagrange multipliers. (4) (1) ~ (3) are repeated until * u is converged.
The details about Pontryagin’s maximum principle are in Appendix B.
5.2.3. Simulation
We compared the optimal schedules predicted by multi-task motor learning models
in the dual-task learning paradigm. Each of three models described in the previous
section predicted the optimal schedules for three different relative difficulties between
two tasks: difficulty ratios of 1:1, 1:1.5, and 1:2. We used the speed of learning as
measure of the task difficulty: If the first task requires twice as much time to learn as the
second task, the difficulty ratio between two tasks is 2:1. For each difficulty ratio, using
the maximum principle, each model predicted the optimal schedule for 100 trials, which
８ ９
maximizes the long-term retention.
In simulations, we used the following parameters: in the N-state model, for the easy
task, 0.981 A = , 0.086 B = , for the task with moderate difficulty, 0.987 A = ,
0.0574 B = , and for the difficult task, 0.991 A = , 0.043 B = . In the N-fast N-slow and
1-fast N-slow models, for the easy task, 0.8
f
A = , 0.99
s
A = , 0.3
f
B = , 0.05
s
B = , for
the task with moderate difficulty, 0.87
f
A = , 0.993
s
A = , 0.21
f
B = , 0.033
s
B = , and
for the difficult task, 0.90
f
A = , 0.995
s
A = , 0.155
f
B = , 0.0250
s
B = .
5.2.4. Experiment
Healthy subjects signed an informed consent to participate in the study, which was
approved by the local Institutional Review Board. Subjects sat in front of a liquid crystal
display monitor and held a stylus pen on a tablet. Subjects put their right arms on the arm
support and placed the stylus pen at the home position which is the center of the tablet. A
wooden screen was placed between the display and tablet to keep subjects from seeing
their arm movements during the experiment. At each trial, subjects moved a cursor to a
target by moving the stylus on the tablet. At the beginning of each trial, a cursor appeared
at the center of the screen. When the cursor disappeared and a target appeared at a
position 8 cm from the center of the screen, subjects had 1 second to move the cursor to
the target. Only the initial part of cursor trajectories (up to 4 cm from the center) was
shown to minimize online corrections during movements. The cursor appeared again for
500msec at the final cursor location. Then, subjects had 2 second to move their arms back
to the home position. During homing movements, a circle appeared on the screen to
９ ０
indicate how far the stylus is away from the home position: The further away the stylus
was, the larger the circle was. Inter-trial intervals were 2.5 sec including homing
movements.
Subjects were randomly assigned one of eight groups: 3 parameter estimation
groups, 5schedule test groups with different schedules and task combinations. We
designed 3 parameter estimation groups to estimate parameters of three multi-task
learning models and to measure difficulties of three tasks (or transformation): the gain,
rotation, and shearing tasks.
Subjects in the parameter estimation groups had 160 baseline trials to be
familiarized with the apparatus. During the baseline trials, targets appeared one of eight
target positions, which were 45 degree apart from each other and 8cm away from the
center position. After the baseline trials, subjects practiced one of three tasks for 300
trials. In the middle of practice trials, two washout trial blocks were introduced: 25
washout trials after 50 practice trials and 50 washout trials after 150 practice trials
respectively. During the practice trials, targets appeared pseudo-randomly within one of
four ranges: from -45° to 45°, from 45° to 135°, from 135° to 215°, and from 215° to
305°. Four ranges were counter-balanced across subjects. Finally, 70 generalization test
trials followed, in which subjects were given 8 generalization targets to reach without
visual feedback 5 times. Generalization targets were 45 degree apart from each other and
presented in a pseudo random order. 6 reminder trials were given after each 4
generalization test trials, in which subjects practiced targets with visual feedback in a
range that they practiced during the practice trials.
９ １
We designed 5 schedule test groups to compare alternating schedules with
predictions of two models on the optimal schedules in two different difficulty ratios
between two tasks to learn. It was 5 schedules instead of 6 because for learning two tasks
with the difficulty ratio of 1:1.5, the optimal schedule predicted by the 1-fast N-slow
model was the same as the alternating schedule. Subjects in the schedule test groups had
experiments on two consecutive days. On day 1, subjects first had 160 baseline trials,
followed by 600 practice trials practiced two tasks, either the gain and rotation tasks or
the gain and shearing task on in one of three schedules: the alternating schedule, optimal
schedule predicted by the N-fast N-slow model, and optimal schedule predicted by the 1-
fast N-slow model. Details are as follows:
• Group 1 - parameter estimation group with gain transformation: On day 1, subjects
practice the baseline mapping, and learn the gain transformation followed by the
retention and generalization tests.
• Group 2 - parameter estimation group with rotation transformation: the same as
Group 1 except that subjects learn the rotation transformation.
• Group 3 - parameter estimation group with shearing transformation: the same as
Group 1 except that subjects learn the shearing transformation.
• Group 4 - Alternating schedule group with the gain and rotation transformations: On
day 1, subjects practice the baseline mapping, and learn the gain and rotation
transformation with the alternating schedule followed by the retention test. On day 2,
subjects have the retention test, and relearn the gain and rotation transformation with
the same schedule as on day 1 followed by the retention test.
９ ２
• Group 5 - Alternating schedule group with the gain and shearing transformations: the
same as Group 4 except that subjects learn the gain and shearing transformations.
• Group 6 – N-fast-N-slow optimal schedule group with the gain and rotation
transformations: the same as Group 4 except that subjects learn the gain and rotation
transformations with the N-fast-N-slow optimal schedule.
• Group 7 - N-fast-N-slow optimal schedule group with the gain and shearing
transformations: the same as Group 6 except that subjects learn the gain and shearing
transformation
• Group 8- 1-fast-N-slow optimal schedule group with the gain and shearing
transformations: the same as Group 8 except that subjects learn the gain and shearing
transformations
Note that group 4, the alternating schedule group with gain and rotation
transformations is also the 1-fast-N-slow optimal schedule group with the gain and
rotation transformation. Practice schedule for each group is shown in Figure 5.4.
5.3. Results
5.3.1. Simulation results
To compare optimal schedules predicted by models with the alternating schedule,
we used the switching probability and ratio between numbers of trials for two tasks (Fig.
5.6). Overall, when two tasks had the equal difficulty, all three schedules predicted the
alternating schedules as the optimal schedules. However, as the difference in relative
９ ３
difficulties between tasks increased, the optimal schedules got more different from the
alternating schedule and assigned more trials to more difficulty task.
Figure 5.7 shows optimal schedules predicted by models for 3 difficulty ratios:
1:1, 1:1.5, and 1:2. For two tasks with the same difficulty (difficulty ratio of 1:1), all
models proposed the alternating schedule as the optimal schedule. However, as the
difference in relative difficulties between tasks increased from 1:1 to 1:1.5, the optimal
schedules predicted by models were changed: The optimal schedule of the N-states model
assigns a block of trials to the easier task at the end; The optimal schedule of the N-fast
N-slow model is almost the same as the alternating schedule except two trials that are
assigned to more difficult tasks; The optimal schedule of the 1-fast N-slow model
assigned 10 more trials to more difficult tasks. When the difference in relative difficulties
between tasks increased further to 1:2: The optimal schedule of the N-state model
becomes a more blocked schedule assigning trials to the easier task at the end. The
optimal schedule of the N-fast N-slow model is still similar to the alternating schedule
with only 6 more trials to more difficult tasks; the optimal schedule of the 1-fast N-slow
model assigned 16 more trials to more difficult tasks.
We also compared delayed retention performance of the optimal schedules for
our dual visuomotor adaptation experiments with of the alternating schedule in
simulations. For dual adaptation of gain and rotation, although the optimal schedule was
different from the alternating schedule, delayed retention performance of the optimal
schedule was very similar to the one of the alternating schedule. Different between
delayed retention RMSEs of two schedules were less than 1% (Fig. 5.8A). However, for
dual adaptation of gain and shearing delayed retention performance of the optimal
９ ４
schedule was better than the alternating schedule as the number of trials increased (Fig.
5.8B). When we compared the number of trials to achieve the same delayed retention
performance, we could see the difference between the optimal and alternating schedules
more clearly (Fig. 5.9): The optimal schedule reached the delayed retention performance,
which reached after 640 trials with the alternating schedule, only after 260 trials.
5.3.2. Experiment results
Subjects take different time to learn gain, rotation, and shearing tasks. Subjects in
the parameter estimation groups (group 1, 2 and 3) were able to learn one of the gain,
rotation, and shearing tasks. Subjects who practiced either the gain or rotation task could
reach to the performance level at the baseline trials, but subjects who practiced the
shearing task couldn’t. We fit models to average subject performance in the parameter
estimation groups (Fig. 5.10). For the 1-fast N-slow model, parameters found by model
fit were as follows: in the N-state model, for the gain, 0.989 A = , 0.061 B = , for the
rotation, 0.988 A = , 0.066 B = , and for the shearing, 0.987 A = , 0.013 B = . In the
N-fast N-slow and 1-fast N-slow model, for the gain, 0.972
f
A = , 0.993
s
A = ,
0.153
f
B = , 0.037
s
B = , for the rotation, 0.965
f
A = , 0.993
s
A = , 0.199
f
B = ,
0.038
s
B = , and for the shearing, 0.918
f
A = , 0.993
s
A = , 0.080
f
B = , 0.007
s
B = .
The gain was generalized well across all unpracticed target positions, and
rotation was generalized to unpracticed target positions only near the practiced target
positions. However, shearing was not generalized to unpracticed target positions at all
(Fig. 5.11).
９ ５
With the alternating schedule for the dual-task learning of gain and rotation,
subjects learned both tasks well. However, with the alternating schedule for the dual-task
learning of gain and shearing, subjects learned the shearing poorly (Fig. 5.12)
5.4. Discussion
Here, we investigated the optimal practice schedules in dual-task motor learning
using the combined approach of computational models and novel visuomotor adaptation
experiments. Our previous motor learning model predicted that the alternating schedule is
the optimal schedule when difficulties of two tasks are similar, but not anymore as the
difference in relative task difficulties increases. Our experimental results showed that the
alternating schedule led to poor long-term retention performance when the difference in
the task difficulty between two tasks increased.
The well-established contextual interference (CI) effect suggests that, in multi-task
motor learning, the random schedule, in which each task is intermixed, enhances long-
term retention compared to the block schedule, in which each task is presented in one
trial block at a time. However, it has not been unclear whether the random schedule is the
optimal schedule which maximizes long-term retention, or there is a better schedule. Also,
it has not been investigated in which conditions the random schedule is the optimal and in
which conditions it is not.
In this study, we showed how to find the optimal schedule of a model using the
Pontryagin’s maximum principle. We compared the optimal schedules of three multi-task
learning models, the N-states model, N-fast N-slow model, and our previous 1-fast N-
９ ６
slow model. We computed optimal schedules of three models in three different dual-task
learning paradigms: learning two tasks with the same difficulty (i.e. both tasks take the
same time to learn), learning two tasks with the relative difficulty ratio of 1:1.5 (i.e. one
task takes 1.5 times longer to learn than the other), and learning two tasks with the
relative difficulty ratio of 1:2 (i.e. one task tasks twice longer to learn than the other).
Simulations of the 1-fast N-slow model predicted that in dual-task learning, the
pure alternating schedule is optimal to maximize long-term retention when two tasks
have the same difficulty. However as the difference in the task difficulty ratio increases to
1:1.5, the alternating schedule is not optimal anymore, but the schedule with more trials
for the difficult task, while easier task trials are away from each other as far as possible
across trials, becomes optimal.
On the other hand, simulations of the N-fast N-slow model predicted differently
that in dual-task learning, the alternating schedule is optimal to maximize long-term
retention when two tasks have the same difficulty as well as the difficulty ratio of 1:1.5.
However if two tasks have different difficulties, the alternating schedule is not optimal
anymore, but the schedule with more trials for the difficult task, while easier task trials
are away from each other as far as possible across trials, becomes optimal.
The different between predictions about optimal schedules in case with different
task difficulties arise from the different structure of the fast process between two models:
To maximize long-term retention in both N-fast N-slow and 1-fast N-slow models, the
slow processes should increase the most, in other words, the fast process should decrease
the most. In the 1-fast N-slow model, the fast process is decreased by forgetting as well
as interference between tasks. However, in the N-fast N-slow model, since there is no
９ ７
interference in the fast process, the fast processes are decreased only by forgetting.
Therefore, few task alternations in the 1-fast N-slow model are as effective to suppress
the fast process and more effective to increase the slow processes as frequent task
alternations in the N-fast N-slow model. Further experiments will determine which model
accounts for performance during multiple task learning.
This study has two major contributions. First, it theoretically analyzes the
optimality of the alternating schedule and reveals criteria that the alternating schedule is
optimal in multi-task motor learning. Second, it can be used to optimize multi-task motor
learning, such as sports training, musical instrument learning, motor-function
rehabilitations, and etc.
９ ８
Figure 5.1. Supporting simulations showing how long-term retention is related to task
switching in the 1-fast N-slow model for varying dual adaptation schedules in which both
the number of trials and task difficulty are equal for both tasks: 2 tasks and 50 trials per
task. The expected retention is computed from the average slow process for the two tasks.
The multiple schedules were generated with a genetic algorithm. For each schedule, the
average probability of task switching across the schedule, or switching probability (SP) is
computed. There is the strong correlation between SP and expected retention (R
2
> 0.84).
The schedule with lowest SP is the blacked schedule and the schedule with SP = 1
corresponds to the alternating schedule in which the two tasks alternate at each trial.
９ ９
Figure 5.2: Models of multi-task motor learning: (A) N-state model, in which long-term
retention is determined by output performance observed. (B) N-fast N-slow model, in
which long-term retention is determined by the slow process, and there is no interference
among fast processes. (C) 1-fast N-slow model, in which long-term retention is
determined by the slow process, and there is interference among fast processes.
１ ０ ０
Figure 5.3: Apparatus for the optimal schedule experiment at USC (upper) and Marquette
University (lower). A subject sat in front of the liquid crystal display monitor and held a
stylus pen on a tablet. Subject’s right arm was supported by the arm support. A wooden
screen was placed to prevent the subject from seeing his/her arm movements during the
experiment.
１ ０ １
Figure 5.4: Three visuomotor transformations: gain (left), rotation (middle) and shearing
(right). Dotted and solid lines show cursor movements before and after transformations
respectively.
１ ０ ２
Figure 5.5: Schedules of each group for the optimal schedule experiment.
１ ０ ３
Figure 5.6: Switching probabilities (A) and ratio between numbers of trials for task 1 and
task 2 (B) of optimal schedules predicted by the multi-task motor learning models: The
N-state model (blue), 1-fast N-slow model (green), and N-fast N-slow model (red). The
x-axis is the difficulty ratio between two tasks, which is measured by the ratio between
time constants for task 1 and task 2: For two tasks with the equal difficulty, the difficulty
ratio is 1. For two tasks that one task takes twice more time to learn than the other, the
difficulty ratio is 2. For the difficulty ratio of 1, optimal schedules are the alternating
schedules. However, as the difficulty ratio between two tasks increases, optimal
schedules get more different from the alternating schedule and assign more trials to more
difficult task.
１ ０ ４
Figure 5.7: Optimal schedules in the dual-task learning paradigm predicted by the multi-
task motor learning models. First, second, and third rows show the optimal schedules
predicted by the N-state model, N-fast N-slow model, and 1-fast N-slow model
respectively. First, second, and third columns show the optimal schedules for cases in
which difficulty ratios between two tasks are 1:1, 1:1.5, and 1:2 respectively.
１ ０ ５
Figure 5.8: Optimal schedules in the dual-task learning of gain and rotation (A), and gain
and shearing (B) predicted by the 1-fast N-slow model. In the top two panels, blue dots
on the top show practice schedules; red, blue, and green lines in the middle and bottom
show model predictions of performance, short-term learning by the fast process, and
long-term learning by the slow processes across trials respectively. In the bottom two
panels, blue, green, and red lines show model predictions of delayed retention RMSE of
the blocked, alternating, and optimal schedules respectively for different numbers of trials.
１ ０ ６
Figure 5.9: The numbers of trials required for the optimal (green) and alternating (blue)
schedules to reach to delayed retention performance levels in dual-adaptation of gain and
shearing: The optimal schedule reaches to the delayed retention performance levels with
fewer trials than the alternating schedule.
１ ０ ７
Figure 5.10: Practice performance of the parameter estimation groups for the (top) gain,
(middle) rotation, and (bottom) shearing tasks. Subjects in the parameter estimation
groups performed 300 practice trials for one of three task transformations with two
washout trial blocks: 25 washout trials after 50 practice trials and 50 washout trials after
another 100 practice trials. Block dots show average normalized errors. Green, red, and
pink lines show the model fit of the 1-fast N-slow model for the gain, rotation, and
shearing group’s data respectively.
１ ０ ８
Figure 5.11: Generalization test performance of parameter estimation groups for the (left)
gain, (middle) rotation, and (right) shearing tasks. Subjects in the parameter estimation
groups performed 40 generalization trials across target positions without visual feedback,
intermixed by 30 refresher trials in the trained target ranges with visual feedback. Green,
red, and pink lines show average performance (= 1 - normalized error) with standard
errors across target positions in generalization trials for the gain, rotation, and shearing
group’s data respectively. Black lines show average performance in refresher trials.
１ ０ ９
Figure 5.12: Errors of individual subjects in the alternating schedule group for the (top)
gain and rotation tasks and (bottom) gain and shearing tasks. On day 1, subjects
performed 600 practice trials in the alternating schedule. Green, red, and pink dots show
errors for the gain, rotation, and shearing group’s data respectively.
１ １ ０
Chapter 6.
Conclusion: In search of optimal motor learning
6.1. Summary
Through a line of studies in this dissertation, we have tried to achieve two goals.
First goal was to understand mechanisms of multi-task motor learning and second goal
was to optimize multi-task motor learning. For these purposes, we investigated multi-task
motor learning using combinations of computational models of motor learning,
behavioral experiments, and optimal control theory.
First, in Chapter 2, we reviewed computational models of motor learning and
discussed about why computational models are important and how computational models
can contribute to other research areas. Computational models are important because: First,
it can confirm that our understanding of observations is solid and valid. Second, it can
make predictions to test and lead to advances in our understanding. Third, it can be
implemented in forms that help humans.
In Chapter 3, we investigated the architecture of motor memory during multi-task
motor learning. To find a right model of multi-task motor learning, we compared all
possible architecture of motor memory by simulating large bodies of previous motor
adaptation data and fitting to our new dual-adaptation experimental data. Our combined
simulation and experimental analysis support the view that human motor memory has
three characteristics: (1) it contains a single fast-learning and fast-forgetting process. (2)
It contains a slow process with multiple slow-learning slow-forgetting states, all with the
１ １ １
same learning rates and the same forgetting rates; these states are switched with
contextual cues. (3) The two processes are arranged in parallel and compete for errors
during motor adaptation. Neural substrate for each of the fast and slow processes is going
to be investigated with functional magnetic resonance imaging (fMRI) and virtual lesions
using transcranial magnetic stimulation (TMS).
In Chapter4, we revealedthe mechanisms of the contextual interference (CI) effect
using the computational model of motor learning and multi-task motor learning
experiments in healthy individuals and patients post-stroke. Our model predicted that the
CI effect can be observed only when decay in the fast process is sufficiently slow. If, the
decay is rapid, as would be expected in individuals with deficits in short-term memory,
forgetting between trials in the blocked condition leads to small update of the fast process,
and little forgetting between the immediate and delayed retention test, hence a reduced CI.
In our experiments, healthy individuals (with high short-term memory scores) showed the
CI effect, but patients with stroke showed reduced CI. Further analysis confirmed the
model prediction by showing that stroke patients with high short-term memory scores
showed strong CI effect as healthy individuals, but stroke patients with low short-term
memory scores showed almost no CI effect. Therefore, our results suggest that CI effect
is due, at least in part, to greater forgetting in short-term memory between trials of the
same task during random schedules than during blocked schedules.
In Chapter5, we derived the optimal practice schedules for multi-task motor
learning using multi-task motor learning models and optimal control theory. By
comparing the alternating schedule with optimal schedules predicted by models in
different dual-task learning paradigms in simulations, we showed when the alternating
１ １ ２
schedule is optimal and when it is not. The alternating schedule is optimal when two
tasks are equally difficult and even when one task takes 1.5 times longer to learn than the
other task. However, as the difference between two task difficulties increases, the
alternating schedule is not optimal anymore, but the schedule with more trials for the
difficult task, while easier task trials are away from each other as far as possible across
trials, becomes optimal. We confirmed our simulation results in visuomotor learning
experiments.
We have achieved our goals to certain extents: Our 1-fast N-slow model accounts
for multi-task motor learning and explains a wide range of phenomena. Based on the
model, we can compute the optimal practice schedules which maximize long-term
learning for different multi-task motor training. However, we have a lot more to go to
reach our goals.
First, we do not know the neural basis of our multi-task motor learning model. To
better understand the mechanism of multi-task motor learning and complete our
computational models, we need to locate neural substrates of each component in the
multi-task motor learning such as the fast and slow processes.
Second, our model does not account for the effects of uncertainty in learning. It has
been consistently shown that under high uncertainty, the rate of learning increases in
classical conditioning and word learning.
Third, the optimal schedules found by our model are for average subjects and
should be computed in advance of the actual training. To make the optimal schedule
effective in practice, it should be able to compute the optimal schedule of individual
subject on-the-fly during the training.
１ １ ３
Therefore, here we are proposing three studies as future work to cover weaknesses
of current studies and advance to our goals of better understanding the mechanisms of
multi-task motor learning and optimization of multi-task motor learning.
6.2. Future work
6.2.1. Neural substrates of multi-task motor learning
In Chapter3, we proposed the parallel 1-fast N-slow model of motor learning, in
which a single common fast-learning fast-forgetting process and multiple context-specific
slow-learning slow-forgetting processes learn in parallel. Although behavioral experiment
data as well as computational simulation analysis well supported our model, it is unclear
which neural substrates are loci of the fast and slow processes. Therefore, to further
confirm the model and to better understand the mechanism of motor learning, we
designed new experiments in collaboration with Dr. Imamizu at ATR in Japan.
In the first experiment, we will localize neural substrates of both the fast and slow
processes using fMRI. Subjects will practice two visuomotor adaptation tasks intermixed
by over-trained baseline trials in the fMRI scanner. From subject’s behavioral data, the 1-
fast N-slow model will predict the activities of the single fast process and two slow
processes, one per task. Then, we will analyze fMRI data using these activities of the fast
process and slow processes as regressors. Brain areas correlated with the fast and slow
processes will be identified using a generalized linear model (GLM) to explain Blood
Oxygen Level Dependant (BOLD) responses. Based on previous neuroimaging
researches, we hypothesize that the activity of PPC will correlate with the fast process,
１ １ ４
and the activity of cerebellum will correlate with the slow processes.
In the second experiment, we will investigate how the neural substrates of the fast
and slow processes are arranged using rTMS. We will use rTMS to induce virtual lesion
on either the fast or slow processes based on the localization result from the fMRI
experiment. Subjects will practice the same two visuomotor adaptation tasks with over-
trained baseline trials. Subjects will be assigned to one of three groups: rTMS on the fast
process group, rTMS on the slow process group, sham rTMS group. Based on our
previous study, we hypothesize that rTMS on the fast process will improve long-term
retention performance and rTMS on the slow processes will impair long-term retention
performance.
So far, we designed the visuomotor adaptation experiment and, we are conducting
the first fMRI experiment at ATR in Japan now.
6.2.2. Effects of randomness in practice schedules on motor learning
It has been shown that humans and other primates such as monkeys can
simultaneously adapt to two opposite motor adaptation tasks given sufficient contextual
cues, sufficient trials, and proper schedules (Krouchev and Kalaska, 2003; Wada et al.,
2003; Shelhamer et al., 2005; Choi et al., 2008). In random schedules, subjects can adapt
to two opposite force-fields, and several visuo-motor rotations simultaneously (Wada et
al., 2003; Osu et al., 2004; Choi et al., 2008). However, in alternating schedules, either
with alternating blocks of trials or with alternating trial by trial, subjects cannot adapt to
two opposite force-fields (Gandolfo et al., 1996; Karniel and Mussa-Ivaldi, 2002) or
adapt very slowly (Osu et al., 2004).
１ １ ５
Current computational models of motor learning (Wolpert and Kawato, 1998;
Smith et al., 2006; Lee and Schweighofer, 2009), in which motor learning solely depends
on motor errors observed and the states of internal models, cannot explain differences in
performances between the random and alternating schedule because both schedules
provide the same number of practice trials per each task with similar frequency across
trials.
We hypothesize that different levels of randomness between the alternating and
random schedules lead different adaptation speeds: Repetition of the tasks in alternating
schedules, such as ABABAB… will quickly be predictable, resulting in low novelty and
decrease adaptation speed. In contrast, pseudo-random schedules, such as
ABBABABAAB… are unpredictable, resulted in continuous novelty and increase
adaptation speed.
In this study, to account for different adaptation speeds based on randomness of
schedules, we will extend our 1-fast N slow model with Kalman filters so that it can
modulate the learning rates with Kalman gains.Difference in the learning rate has been
explained with Kalman filters (Dayan et al., 2000) in classical conditioning, in which
Kalman filters modulatelearning rates based on thecovariance ofprediction errors, which
is related to model uncertainty. Under high randomness in schedules, the model has high
uncertainty about next trial. Then, Kalman filters weight external observations more than
internal predictions, and it leads to faster adaptation speeds. In contrast, under low
randomness in schedules, the model has low uncertainty about next trial. Then, Kalman
filter weight internal predictions more than external observations, and it leads to slower
adaptation speeds.
１ １ ６
6.2.3. Adaptive optimal schedules in multi-task motor learning
In Chapter4, we showed how to find the optimal schedules in multi-task motor
learning based on our 1-fast N-slow model and the optimal control theory. First we
estimated model parameters for all tasks in one group of subjects and derived the optimal
schedules which maximize long-term retention using the optimal control theory. Then, we
could use the optimal schedules in practice in another group of subjects.
However, these optimal schedules are optimal for average subjects, and not
necessary optimal for individual subject. Also, in practice, it is difficult to estimate model
parameters for every task to practice and derive the optimal schedules in advance.
In this study, we will show how to provide optimal schedules for each individual in
practice. For each individual, model parameters can be different. To account for subjects’
individual differences, we will use a Kalman filter: First, we will assume that our 1-fast
N-slow model with the average parameters can account for individual subject’s learning.
Based on the average parameters, we will compute the optimal schedule, and present a
task according to the schedule. Each trial, the model will predict subject’s performance
and observe the prediction error between the model prediction and subject’s performance.
Based on the error, the model will use Kalman filter to correct estimates of internal states
and model parameters. Note that we want to estimate not only the states but also
parameters of the model. Therefore, we will use either extended Kalman filter or other
methods of parameter estimation using Kalman filter. After updates, with these corrected
parameters, we will compute the optimal schedule again using the optimal control theory
and present the task that maximally increases global predicted long-term performance.
１ １ ７
This will be the first systematic and possibly highly effective method to schedule
motor tasks of different difficulties for different individuals.
１ １ ８
Bibliography
(2005) The practice of theoretical neuroscience. In: Nat Neurosci, 2005/11/25 Edition, p
1627.
(2011) Focus on computational and systems neuroscience. Nat Neurosci 14:121.
Abbott LF (2008) Theoretical neuroscience rising. Neuron 60:489-495.
Anguera JA, Seidler RD, Gehring WJ (2009) Changes in performance monitoring during
sensorimotor adaptation. J Neurophysiol 102:1868-1879.
Anguera JA, Reuter-Lorenz PA, Willingham DT, Seidler RD (2010) Contributions of
spatial working memory to visuomotor learning. J Cogn Neurosci 22:1917-1930.
Bach-y-Rita P, Balliet R, eds (1987) Recovery from stroke. Stroke rehabilitation: the
recovery of motor control. Chicago,: Year Book Medical Publishers, Inc.
Bo J, Seidler RD (2009) Visuospatial working memory capacity predicts the organization
of acquired explicit motor sequences. J Neurophysiol 101:3116-3125.
Boyd LA, Winstein CJ (2004) Providing explicit information disrupts implicit motor
learning after basal ganglia stroke. Learn Mem 11:388-396.
Brashers-Krug T, Shadmehr R, Bizzi E (1996) Consolidation in human motor memory.
Nature 382:252-255.
Brown RM, Robertson EM (2007) Inducing motor skill improvements with a declarative
task. Nat Neurosci 10:148-149.
Brunel N, van Rossum MC (2007) Lapicque's 1907 paper: from frogs to integrate-and-
fire. Biol Cybern 97:337-339.
Butefisch C, Hummelsheim H, Denzler P, Mauritz KH (1995) Repetitive training of
isolated movements improves the outcome of motor rehabilitation of the centrally
paretic hand. J Neurol Sci 130:59-68.
Callan DE, Schweighofer N (2010) Neural correlates of the spacing effect in explicit
verbal semantic encoding support the deficient-processing theory. Hum Brain
Mapp 31:645-659.
１ １ ９
Cauraugh JH, Kim SB (2003) Stroke motor recovery: active neuromuscular stimulation
and repetitive practice schedules. J Neurol Neurosurg Psychiatry 74:1562-1566.
Choi Y , Qi F, Gordon J, Schweighofer N (2008) Performance-based adaptive schedules
enhance motor learning. J Mot Behav 40:273-280.
Cowan N (2001) The magical number 4 in short-term memory: a reconsideration of
mental storage capacity. Behav Brain Sci 24:87-114; discussion 114-185.
Criscimagna-Hemminger SE, Shadmehr R (2008) Consolidation patterns of human motor
memory. J Neurosci 28:9610-9618.
Cross ES, Kraemer DJ, Hamilton AF, Kelley WM, Grafton ST (2009) Sensitivity of the
action observation network to physical and observational learning. Cereb Cortex
19:315-326.
Dayan P, Abbott LF (2001) Theoretical Neuroscience: Computational and Mathematical
Modeling of Neural Systems. Cambridge, Massachusetts: The MIT Press.
Dayan P, Kakade S, Montague PR (2000) Learning and selective attention. Nat Neurosci
3 Suppl:1218-1223.
Della-Maggiore V , Malfait N, Ostry DJ, Paus T (2004) Stimulation of the posterior
parietal cortex interferes with arm trajectory adjustments during the learning of
new dynamics. J Neurosci 24:9971-9976.
DiCiccio TJ, Efron B (1996) Bootstrap Confidence Intervals. Statistical Science 11:189-
228.
Diedrichsen J, Hashambhoy Y , Rane T, Shadmehr R (2005) Neural correlates of reach
errors. J Neurosci 25:9919-9931.
Ethier V , Zee DS, Shadmehr R (2008) Spontaneous recovery of motor memory during
saccade adaptation. J Neurophysiol 99:2577-2583.
Fugl-Meyer AR, Jaasko L, Leyman I, Olsson S, Steglind S (1975) The post-stroke
hemiplegic patient. 1. a method for evaluation of physical performance. Scand J
Rehabil Med 7:13-31.
Fusi S, Asaad WF, Miller EK, Wang XJ (2007) A neural circuit model of flexible
sensorimotor mapping: learning and forgetting on multiple timescales. Neuron
54:319-333.
Gandolfo F, Mussa-Ivaldi FA, Bizzi E (1996) Motor learning by field approximation.
Proc Natl Acad Sci U S A 93:3843-3846.
１ ２ ０
Gilbert PF, Thach WT (1977) Purkinje cell activity during motor learning. Brain Res
128:309-328.
Graydon FX, Friston KJ, Thomas CG, Brooks VB, Menon RS (2005) Learning-related
fMRI activation associated with a rotational visuo-motor transformation. Brain
Res Cogn Brain Res 22:373-383.
Green AM, Kalaska JF (2011) Learning to move machines with the mind. Trends
Neurosci 34:61-75.
Hanlon RE (1996) Motor learning following unilateral stroke. Arch Phys Med Rehabil
77:811-815.
Haruno M, Wolpert DM, Kawato M (2001) Mosaic model for sensorimotor learning and
control. Neural Comput 13:2201-2220.
Hatada Y , Miall RC, Rossetti Y (2006) Long lasting aftereffect of a single prism
adaptation: Directionally biased shift in proprioception and late onset shift of
internal egocentric reference frame. Exp Brain Res 174:189-198.
Hawkins KA, Sullivan TE, Choi EJ (1997) Memory deficits in schizophrenia: inadequate
assimilation or true amnesia? Findings from the Wechsler Memory Scale--revised.
J Psychiatry Neurosci 22:169-179.
Hikosaka O, Nakamura K, Sakai K, Nakahara H (2002) Central mechanisms of motor
skill learning. Curr Opin Neurobiol 12:217-222.
Hinder MR, Walk L, Woolley DG, Riek S, Carson RG (2007) The interference effects of
non-rotated versus counter-rotated trials in visuomotor adaptation. Exp Brain Res
180:629-640.
Hintzman DL, Block RA, Summers JJ (1973) Contextual associations and memory for
serial position. Journal of Experimental Psychology 97:220-229.
Hodgkin AL, Huxley AF (1957) A quantitative description of membrane current and its
application to conduction and excitation in nerve. Journal of Physiology 117:500-
544.
Howard IS, Ingram JN, Wolpert DM (2008) Composition and decomposition in bimanual
dynamic learning. J Neurosci 28:10531-10540.
Huang VS, Shadmehr R, Diedrichsen J (2008) Active learning: learning a motor skill
without a coach. J Neurophysiol 100:879-887.
１ ２ １
Imamizu H, Kawato M (2008) Neural correlates of predictive and postdictive switching
mechanisms for internal models. J Neurosci 28:10751-10765.
Imamizu H, Kuroda T, Yoshioka T, Kawato M (2004) Functional magnetic resonance
imaging examination of two modular architectures for switching multiple internal
models. J Neurosci 24:1173-1181.
Imamizu H, Kuroda T, Miyauchi S, Yoshioka T, Kawato M (2003) Modular organization
of internal models of tools in the human cerebellum. Proc Natl Acad Sci U S A
100:5461-5466.
Imamizu H, Sugimoto N, Osu R, Tsutsui K, Sugiyama K, Wada Y , Kawato M (2007)
Explicit contextual information selectively contributes to predictive switching of
internal models. Exp Brain Res 181:395-408.
Imamizu H, Miyauchi S, Tamada T, Sasaki Y , Takino R, Putz B, Yoshioka T, Kawato M
(2000) Human cerebellar activity reflecting an acquired internal model of a new
tool. Nature 403:192-195.
Immink MA, Wright DL (2001) Motor programming during practice conditions high and
low in contextual interference. J Exp Psychol Hum Percept Perform 27:423-437.
Joiner WM, Smith MA (2008) Long-term retention explained by a model of short-term
learning in the adaptive control of reaching. J Neurophysiol 100:2948-2955.
Kantak SS, Sullivan KJ, Fisher BE, Knowlton BJ, Winstein CJ (2010) Neural substrates
of motor memory consolidation depend on practice structure. Nat Neurosci
13:923-925.
Karniel A, Mussa-Ivaldi FA (2002) Does the motor control system use multiple models
and context switching to cope with a variable environment? Exp Brain Res
143:520-524.
Kawashima R, Roland PE, O'Sullivan BT (1995) Functional anatomy of reaching and
visuomotor learning: a positron emission tomography study. Cereb Cortex 5:111-
122.
Keisler A, Shadmehr R (2010) A shared resource between declarative memory and motor
memory. J Neurosci 30:14817-14823.
Kim S, Hwang J, Lee D (2008) Prefrontal coding of temporally discounted values during
intertemporal choice. Neuron 59:161-172.
１ ２ ２
Kim S, Lee JY , Schweighofer N (2010) In search of the optimal schedule for multi-task
motor adaptation. In: Society for Neural Control of Movement. Naples, Florida.
Kojima Y , Iwamoto Y , Yoshida K (2004) Memory of learning facilitates saccadic
adaptation in the monkey. J Neurosci 24:7531-7539.
Kording KP, Tenenbaum JB, Shadmehr R (2007) The dynamics of memory as a
consequence of optimal adaptation to a changing body. Nat Neurosci 10:779-786.
Krakauer JW (2006) Motor learning: its relevance to stroke recovery and
neurorehabilitation. Curr Opin Neurol 19:84-90.
Krakauer JW, Shadmehr R (2007) Towards a computational neuropsychology of action.
Prog Brain Res 165:383-394.
Krakauer JW, Ghilardi MF, Ghez C (1999) Independent learning of internal models for
kinematic and dynamic control of reaching. Nat Neurosci 2:1026-1031.
Krakauer JW, Ghez C, Ghilardi MF (2005) Adaptation to visuomotor transformations:
consolidation, interference, and forgetting. J Neurosci 25:473-478.
Krouchev NI, Kalaska JF (2003) Context-dependent anticipation of different task
dynamics: rapid recall of appropriate motor skills using visual cues. J
Neurophysiol 89:1165-1175.
Kwakkel G, Wagenaar RC, Twisk JW, Lankhorst GJ, Koetsier JC (1999) Intensity of leg
and arm training after primary middle-cerebral-artery stroke: a randomised trial.
Lancet 354:191-196.
Lee JY , Schweighofer N (2009) Dual adaptation supports a parallel architecture of motor
memory. J Neurosci 29:10396-10404.
Lee TD, Magill RA (1983) The locus of contextual interference in motor-skill acquisition.
Journal of Experimental Psychology: Human Perception and Performance 9:730-
746.
Lee TD, Magill RA, Weeks DJ (1985) Influence of practice schedule on testing schema
theory predictions in adults. J Mot Behav 17:283-299.
Lee TD, Swanson LR, Hall AL (1991) What is repeated in a repetition? Effects of
practice conditions on motor skill acquisition. Physical Therapy 71:150-156.
Li CS, Padoa-Schioppa C, Bizzi E (2001) Neuronal correlates of motor performance and
motor learning in the primary motor cortex of monkeys adapting to an external
force field. Neuron 30:593-607.
１ ２ ３
Lin CH, Fisher BE, Winstein CJ, Wu AD, Gordon J (2008) Contextual interference effect:
elaborative processing or forgetting-reconstruction? A post hoc analysis of
transcranial magnetic stimulation-induced effects on motor learning. J Mot Behav
40:578-586.
Magill RA, Hall KG (1990) A review of the contextual interference effect in motor skill
acquisition. Human Movement Science 9:241-289.
Marr D (1982) Vision: a computational investigation into the human representation and
processing of visual information: W. H. Freeman.
Medina JF, Nores WL, Mauk MD (2002) Inhibition of climbing fibres is a signal for the
extinction of conditioned eyelid responses. Nature 416:330-333.
Miall RC, Jenkinson N, Kulkarni K (2004) Adaptation to rotated visual feedback: a re-
examination of motor interference. Exp Brain Res 154:201-210.
Miall RC, Christensen LO, Cain O, Stanley J (2007) Disruption of state estimation in the
human lateral cerebellum. PLoS Biol 5:e316.
Nixon SJ, Kujawski A, Parsons OA, Yohman JR (1987) Semantic (verbal) and figural
memory impairment in alcoholics. J Clin Exp Neuropsychol 9:311-322.
Nozaki D, Kurtzer I, Scott SH (2006) Limited transfer of learning between unimanual
and bimanual skills within the same limb. Nat Neurosci 9:1364-1366.
Oppenheim A V , Ronald WS (1999) Discrete-Time Signal Processing, Second Edition:
Prentice Hall.
Osu R, Hirai S, Yoshioka T, Kawato M (2004) Random presentation enables subjects to
adapt to two opposing forces on the hand. Nat Neurosci 7:111-112.
Pyle WH (1919) Transfer and interference in card-distributing. Journal of Educational
Psychology 10:107-110.
Rall W (1959) Branching dendritic trees and motoneuron membrane resistivity. Exp
Neurol 1:491-527.
Schaal S, Schweighofer N (2005) Computational motor control in humans and robots.
Curr Opin Neurobiol 15:675-682.
Schmidt RA, Lee TD (1999) Motor control and learning: a behavioral emphasis.
Champaign, Ill: Human Kinetics.
１ ２ ４
Schultz W, Dayan P, Montague PR (1997) A neural substrate of prediction and reward.
Science 275:1593-1599.
Schweighofer N, Doya K (2003) Meta-learning in reinforcement learning. Neural Netw
16:5-9.
Schweighofer N, Doya K, Fukai H, Chiron JV , Furukawa T, Kawato M (2004) Chaos
may enhance information transmission in the inferior olive. Proc Natl Acad Sci U
S A 101:4655-4660.
Seidler RD, Noll DC (2008) Neuroanatomical correlates of motor acquisition and motor
transfer. J Neurophysiol 99:1836-1845.
Shadmehr R, Wise SP (2005) The computational neurobiology of reaching and pointing:
a foundation for motor learning. Cambridge, MA: MIT Press.
Shadmehr R, Krakauer JW (2008) A computational neuroanatomy for motor control. Exp
Brain Res 185:359-381.
Shadmehr R, Smith MA, Krakauer JW (2010) Error correction, sensory prediction, and
adaptation in motor control. Annu Rev Neurosci 33:89-108.
Shea JB, Morgan RL (1979) Contextual interference effects on the acquisition, retention,
and transfer of a motor skill. Journal of Experimental Psychology 5:179-187.
Shelhamer M, Aboukhalil A, Clendaniel R (2005) Context-specific adaptation of saccade
gain is enhanced with rest intervals between changes in context state. Ann N Y
Acad Sci 1039:166-175.
Shidara M, Kawano K, Gomi H, Kawato M (1993) Inverse-dynamics model eye
movement control by Purkinje cells in the cerebellum. Nature 365:50-52.
Smith MA, Ghazizadeh A, Shadmehr R (2006) Interacting adaptive processes with
different timescales underlie short-term motor learning. PLoS Biol 4:e179.
Sunderland A, Tuke A (2005) Neuroplasticity, learning and recovery after stroke: a
critical evaluation of constraint-induced therapy. Neuropsychol Rehabil 15:81-96.
Tanaka S, Honda M, Hanakawa T, Cohen LG (2010) Differential contribution of the
supplementary motor area to stabilization of a procedural motor skill acquired
through different practice schedules. Cereb Cortex 20:2114-2121.
Thoroughman KA, Shadmehr R (2000) Learning of action through adaptive combination
of motor primitives. Nature 407:742-747.
１ ２ ５
Thoroughman KA, Taylor JA (2005) Rapid reshaping of human motor generalization. J
Neurosci 25:8948-8953.
Todorov E (2007) Optimal control theory. In: Bayesian Brain: Probabilistic Approaches
to Neural Coding (Doya K, Ishii S, Pouget A, Rao RPN, eds), pp 269-298: MIT
Press
Tong C, Wolpert DM, Flanagan JR (2002) Kinematics and dynamics are not represented
independently in motor working memory: evidence from an interference study. J
Neurosci 22:1108-1113.
Tseng YW, Diedrichsen J, Krakauer JW, Shadmehr R, Bastian AJ (2007) Sensory
prediction errors drive cerebellum-dependent adaptation of reaching. J
Neurophysiol 98:54-62.
Tsuitsui S, Lee TD, Hodges NJ (1998) Contextual interferences in learning new patterns
of bimanual coordination. Journal of Motor Bahavior 30:151-157.
Wada Y , Kawabata Y , Kotosaka S, Yamamoto K, Kitazawa S, Kawato M (2003)
Acquisition and contextual switching of multiple internal models for different
viscous force fields. Neurosci Res 46:319-331.
Wechsler D (1987) Wechsler Memory Scale-revised. New York: Psychological
Corporation.
Wigmore V , Tong C, Flanagan JR (2002) Visuomotor rotations of varying size and
direction compete for a single internal model in motor working memory. J Exp
Psychol Hum Percept Perform 28:447-457.
Williams KS (1992) The nth Power of a 2 x 2 Matrix. Mathematics Magazine 65:336-336.
Wilson D, Baker LL, Craddock JA (1984) Functional test for the hemiparetic upper
extremity. The American Journal of Occupational Therapy 38:159-164.
Winstein CJ, Merians AS, Sullivan KJ (1999) Motor learning after unilateral brain
damage. Neuropsychologia 37:975-987.
Wolf SL, Blanton S, Baer H, Breshears J, Butler AJ (2002) Repetitive task practice: a
critical review of constraint-induced movement therapy in stroke. Neurologist
8:325-338.
Wolf SL, Winstein CJ, Miller JP, Taub E, Uswatte G, Morris D, Giuliani C, Light KE,
Nichols-Larsen D (2006) Effect of constraint-induced movement therapy on upper
extremity function 3 to 9 months after stroke: the EXCITE randomized clinical
trial. JAMA 296:2095-2104.
１ ２ ６
Wolpert DM, Kawato M (1998) Multiple paired forward and inverse models for motor
control. Neural Netw 11:1317-1329.
Wolpert DM, Ghahramani Z (2000) Computational principles of movement neuroscience.
Nat Neurosci 3 Suppl:1212-1217.
Wolpert DM, Goodbody SJ, Husain M (1998) Maintaining internal representations: the
role of the human superior parietal lobe. Nat Neurosci 1:529-533.
Zarahn E, Weston GD, Liang J, Mazzoni P, Krakauer JW (2008) Explaining savings for
visuomotor adaptation: linear time-invariant state-space models are not sufficient.
J Neurophysiol 100:2537-2548.
１ ２ ７
Appendix A.
Supplementary materials for Chapter 2
A.1. Relationship between parameters of parallel models and
equivalent serial models in single task adaptation trials
Here we show analytically how we find parameters of serial models from parallel
models so that both serial and parallel models behave identically in single task adaptation
trials.
Since, in the single task adaptation trials, all models with fast and slow processes
are equivalent to the 1-fast 1-slow model, we compare only the parallel and serial 1-fast
1-slow models (Smith et al., 2006). The z-transform transfer function (Oppenheim and
Ronald, 1999) of the parallel 1-fast 1-slow model, ( )
parallel
G z is given below:
( )
( )
( )
f f
f
f
X z B
G z
E z z A
= =
- (s23)
( )
( )
( )
s s
s
s
X z B
G z
E z z A
= =
- (s24)
2
( ) ( )
( )
( ) ( ) ( )
( ) ( )
f s f s s f parallel
f s
f s f s
B B z A B A B
Y z
G z G z G z
E z z A A z A A
+ - +
= = + =
- + +
(s25)
where ( )
f
G z and ( )
s
G z are the z-transform transfer functions of the fast process
and slow process.
１ ２ ８
Similarly, the z-transform transfer function of the serial 1-fast 1-slow model,
( )
serial
G z is:
( )
( )
( )
f f
f
f
X z B
G z
E z z A
= =
- (s26)
( )
( )
( )
s s
s
f s
X z B z
G z
X z z A
= =
- (s27)
2
(1 )
( )
( ) ( ) ( ) ( )
( ) ( )
f s s f serial
f f s
f s f s
B B z A B
Y z
G z G z G z G z
E z z A A z A A
+ - = = + =
- + +
(s28)
For the parallel and serial models to have identical system responses, two transfer
functions should be equal. Therefore:
parallel parallel serial serial
f s f s
A A A A + = + (s29)
parallel parallel serial serial
f s f s
A A A A = (s30)
(1 )
parallel parallel serial serial
f s f s
B B B B + = + (s31)
parallel parallel parallel parallel serial serial
f s s f s f
A B A B A B + = (s32)
Using the equations (s7) through (s10), and the conditions of parameters, which are
s f
A A > ,
s f
B B < , we can find parameters of the serial model given parameters of the
equivalent parallel model:
serial parallel
f f
A A = (s33)
serial parallel
s s
A A = (s34)
parallel parallel parallel parallel
f s s f serial
f parallel
s
A B A B
B
A
+
= (s35)
１ ２ ９
1
parallel parallel
f s serial
s serial
f
B B
B
B
+
= - (s36)
A.2. Sensitivity analysis for parameters of the parallel 1-fast N-slow model
Here we find valid ranges of parameters for the parallel 1-fast N-slow model.
Since our model has four parameters and all parameters are related to each other, it is
difficult to find the valid ranges of parameters either by computing analytically or by
simulating the model with all possible combinations of parameters. Instead, we combine
both methods: First, we compute analytically ranges of parameters for the model to hold
basic assumptions of the model. Then, we simulate experiments with changing
parameters in those ranges and see if the model reproduces each of experiment results.
A.2.1. Ranges of parameters to hold basic model assumptions
For the parallel 1-fast N-slow model, we have following basic assumptions:
1 0
s f
A A ≥ > > (s37)
0
f s
B B > > (s38)
Furthermore, in a block of same task trials:
,
( ) ( ) ( )
f s k
y i x i x i = + (s39)
( 1) ( ) ( )
f f f f
x i A x i B e i + = + (s40)
, ,
( 1) ( ) ( )
s k s s k s
x i A x i B e i + = + (s41)
where,
,
( ) ( )
T
s k s k
x i i = x c . We can rewrite s2 to s4 in a vector-matrix form.
( 1) ( ) i i + = + x Ax b (s42)
１ ３ ０
where,
,
( )
( )
( )
f
s k
x i
i
x i
=
x ,
'
'
f f f f f
s s s s s
A B B A B
B A B B A
- - -
= =
- - -
A ,
f
k
s
B
f
B
=
b .
Then, after n trials,
1
1
( ) (0)
n
i n
i
n
- =
= +
∑
x Α b A x (s43)
We can further simplify s21 using the initial condition of (0) [0 0]
T
= x , and the
generic form of n th order of 2 x 2 matrix (Williams, 1992):
i i i
β α
α β
α β β α
- - = +
- -
A I A I
A (s44)
where, α and β are eigenvalues of A:
' ' 2 ' '
4 ( )
2 2
f s s f f s
B B A A A A
β
+ - +
= + (s45)
' ' 2 ' '
4 ( )
2 2
f s s f f s
B B A A A A
α
+ - +
= - (s46)
Then, s21 becomes:
1 1
( )
1 1
n n
n
α β β α
α α β β β α
- - - -
= +
- - - -
A I A I
x b (s47)
From s24, for ( ) n x to converge to a finite value and not to oscillate, α and β
should be less than 1 and greater than 0.
1 0 β α > > > (s48)
From s24 and s26:
' ' 2 ' '
4 ( )
0
2 2
f s s f f s
B B A A A A + - +
- > (s49)
１ ３ １
The s27 turns into:
1
f
s
f s
B
B
A A
+ < (s50)
Using s28 with s15 and s16, we can find looser but more practical boundaries to
choose parameters, s29:
, 0.5
f f s s
B A B A < < (s51)
We can have another condition for parameters by assuming that, when n goes
infinity, ( )
f
x ∞ should be less than
,
( )
s k
x ∞ . Then, from s25 and s26:
'
'
( 1 ( ))
1 1
( )
( 1 ( )) 1 1 ( 1)( 1)
f f s
k
s s f
B A B
f
B A B
α β
β α
α β α α β β β α β α
- + - + - - - -
∞ = + =
- + - + - - - - - -
A I A I
x b
(s52)
Since
' '
f s
A A α β + = + , from s23 and s24, s29 becomes:
'
'
,
(1 ) ( ) (1 )
( )
(1 ) ( ) (1 ) ( 1)( 1) ( 1)( 1)
f s f f s s
k k
s f s k s f f
B A x B A B
f f
B A x B A B β α β α
- ∞ - -
∞ = = =
- ∞ - - - - - -
x
(s53)
Therefore, to be
,
( ) ( )
f s k
x x ∞ < ∞ :
1
1
s s
f f
A B
A B
- <
- (s54)
In summary, so far, we derive two conditions, s28 and s32, for the parameters of the
1-fast N-slow model to meet basic assumptions of the model: The s28 guarantees that the
model neither diverse nor oscillate with a constant input. The s32 guarantees that with
sufficient number of trials, the slow process will take over the fast process.
１ ３ ２
A.2.2. Ranges of parameters to reproduce experimental data
For parameters which meet s28 and s32, we further find valid ranges of
parameters for each experimental paradigm in simulations: (1) Spontaneous recovery, (2)
anterograde interference, (3) dual-adaptation, and (4) savings in wash-out paradigm. We
use same schedules described in the main text (see Figure 3, 4, 5 and 7), and vary each
parameter systematically by a factor of
0.25
10 within the range given by s15, s16, s28
and s32. Figure A.1 shows these ranges of parameters.
Figure A.1. Ranges of parameters of the parallel 1-fast N-slow model, which are given
by s15, s16, s28 and s32. Subplots are arranged along the x and y axes with changes of
1
f
A - and 1
s
A - respectively. In each subplot, blue areas show ranges of
f
B and
s
B
with given values of
f
A and
s
A .
１ ３ ３
A.2.3. Spontaneous Recovery
To reproduce spontaneous recovery, in the initial error-clamping trials, y should
increase with trials. Figure A.2 shows ranges of parameters to meet this condition. Wide
ranges of parameters can reproduce spontaneous recovery except large
f
A with small
f
B .
Figure A.2. Ranges of parameters of the parallel 1-fast N-slow model to reproduce
simultaneous recovery. Among the ranges of parameters given in Figure A.1, parameters
in red areas reproduce simultaneous recovery but parameters in blue areas do not.
１ ３ ４
A.2.4. Anterograde Interference
To reproduce anterograde interference, errors at the switching trials should be
larger than the error at the first trial. Figure A.3 shows ranges of parameters to meet this
condition. Again, wide ranges of parameters can reproduce anterograde interference
except when
f
A is small,
s
A is large, and
f
B and
s
B are similar.
Figure A.3. Ranges of parameters of the parallel 1-fast N-slow model to reproduce
anterograde interference. Among the ranges of parameters given in Figure A.1,
parameters in red areas reproduce anterograde interference but parameters in blue areas
do not.
１ ３ ５
A.2.5. Dual-adaptation
To reproduce dual-adaptation, y should increase gradually with trials. Figure A.4
shows ranges of parameters that show more than 10 percent of performance improvement
at the last 10 trials compared to the first 10 trials. Most of combinations of all four
parameters can dual-adaptation.
Figure A.4. Ranges of parameters of the parallel 1-fast N-slow model to reproduce
dual-adaptation. Among the ranges of parameters given in Figure A.1, parameters in red
areas reproduce dual-adaptation but parameters in blue areas do not.
A.2.6. Savings in Wash-out Paradigm
To reproduce savings in the wash-out paradigm, y should increase faster in the
readaptation trials than in the first adaptation trials. Figure A.5 shows ranges of
１ ３ ６
parameters that show more than 10 percent of performance improvement at the first 10
trials of readaptation compared to the first 10 trials of first adaptation. Wide ranges of
parameters can reproduce savings in the wash-out paradigm. However, either small
s
A
or large
f
B with small
s
B and large
s
A cannot reproduce savings in wash-out
paradigm.
Figure A.5. Ranges of parameters of the parallel 1-fast N-slow model to reproduce
savings in wash-out paradigm. Among the ranges of parameters given in Figure A.1,
parameters in red areas reproduce savings in wash-out paradigm but parameters in blue
areas do not.
１ ３ ７
Figure A.6. Ranges of parameters of the parallel 1-fast N-slow model to reproduce all
four experimental data: spontaneous recovery, anterograde interference, dual-adaptation,
and savings in wash-out paradigm. Among the ranges of parameters given in Figure A.1,
parameters in red areas reproduce savings in wash-out paradigm but parameters in blue
areas do not.
In summary, we have shown the ranges of parameters, with which the 1-fast N-
slow model can reproduce each of four experimental data: spontaneous recovery,
anterograde interference, dual-adaptation, and savings in wash-out paradigm. Figure A.6
show the ranges of parameters, which allow the model to reproduce all four experimental
data. Since all four parameters are related to each other, it is difficult to find general
relationship between parameters in these ranges. Instead, we have practical guidelines for
１ ３ ８
parameters to reproduce data: (1) For large
f
A (1 0.01)
f
A - <
,
f
B
should be greater
than
0.01
. (2) For small
f
A (1 0.1)
f
A - >
and large
s
A
,
(1 0.01)
s
A - <
,
f
B
and
s
B
should not be close. However, it should be noted that here we use specific schedules for
simulations, and for different schedules, resulting ranges of parameters can be varied.
１ ３ ９
Appendix B.
Pontryagin’s maximum principle
Pontryagin’s maximum principle is one of two fundamental bases for optimal
control theory with dynamic programming. Both the maximum principle and dynamic
programming lead to the same solution. The main differences between the maximum
principle and dynamic programming are: (1) the computational complexities of the
solutions based on the maximum principle grow linearly with the state dimensionality
and it avoids the curse of dimensionality. (2) The maximum principle applies to
deterministic problems (Todorov, 2007).
The Pontryagin’s maximum principle uses Lagrangian multipliers to find sequences
0 1 1
( , ,... )
n- u u u and
0 1 1
( , ,... )
n- x x x minimizing J subject to constraints
1
( , )
k k k
f
+
= x x u . Since, we have n constraints, we needs n Lagrange multipliers
1
( ,... )
n
λ λ . The Lagrangian L is:
{ }
1
1 1
0
1
1 1
0
( , , ) ( , ) ( ( , ) )
( ) ( , ) ( ( , ) )
n
T
k k k k
k
n
T
n k k k k k k
k
L J f
h x l f
λ λ
λ
- + +
=
- + +
=
= + - = + + - ∑
∑
x u x u x u x
x u x u x
We also define the Hamiltonian, H and, for convenience, rearrange the terms in
L :
( )
1 1
( , , ) ( , ) ( , )
k T
k k k k k k k k
H l f λ λ
+ +
= + x u x u x u
{ }
1
( )
0 0 1
0
( ) ( , , )
n
T T k T
n n n k k k k k
k
L h H λ λ λ λ
- +
=
= - + + - ∑
x x x x u x
１ ４ ０
Then, the differential changes in L by changes in x and u are as follows:
1
( ) ( )
0 0
0
( ( ) )
T T
n
T T k k
n n n k k k
k
dL h d d H d H d λ λ λ
- =
∂ ∂
= - + + - +
∂ ∂
∑ x
x x x x u
x u
For the optimal sequences of
*
x and
*
u , L
∂
∂x
and L
∂
∂u
should be 0 .
Therefore, we first choose λ so that 0 L
∂
=
∂x
:
* ( )
1
*
( , ) ( , ) , 0
( )
k T
k k k k k k
n n
H l f k n
h
λ λ
λ
+
∂
= = + ≤ <
∂
=
x x
x
x u x u
x
x
and then, with the
*
λ above, we find
*
u to satisfy
*
0 L
λ λ =
∂
=
∂u
:
* ( ) *
1
arg min ( , , )
k
k k k
H
+
=
u
u x u λ
In summary, we can find the optimal control sequence
*
u using the sequence of
the Lagrange multipliers
*
λ , which can be computed with system dynamics and cost
functions.
1
1
* ( )
1
( , )
( , ) ( , )
arg min ( , , )
k k k
T
k k k k k k
k
k k k
f
l f
H
+
+
+
=
= +
=
x x
u
x x u
λ x u x u λ
u x u λ
Abstract (if available)
Abstract
Although recent computational modeling research has advanced our understanding of motor learning, previous studies focused on single-task motor learning and did not account for multiple task motor learning which is the norm in sports, music, professional skill development, and neuro-rehabilitation. In this dissertation, we took the combined approach of theoretical analysis, computational modeling, and behavioral experiments to understand the mechanisms of multi-task motor learning, and based on this understanding, to optimize multi-task motor learning. We first suggested a parallel architecture of motor memory in multi-task motor learning: By examining systematically how possible architectures account for experimental results, we showed that the human brain engages a fast-learning-fast-forgetting learning process in parallel with multiple slow-learning-slow-forgetting learning processes. We then investigated how practice schedules and the integrity of short-term memory affect long-term learning: Based on our model, we found that for healthy individuals with intact short-term memory, random practices schedule lead to better long term learning than blocked practice schedules. However for individuals post-stroke with deficits in short-term memory, the effect of practice schedules in long-term learning was mitigated. We finally derived optimal schedules for multi-task motor learning by applying optimal control theory to our computational model of multi-task motor learning. We found that alternating schedules are optimal only if tasks have equal difficulties. If differences in difficulties between tasks increase, our algorithms provide optimal schedules that have the potential to enhance long-term learning in multi-task motor learning.
Linked assets
University of Southern California Dissertations and Theses
Conceptually similar
PDF
Computational model of stroke therapy and long term recovery
PDF
Virtual surgeries as a tool for studying motor learning
PDF
Computational models and model-based fMRI studies in motor learning
PDF
Iterative path integral stochastic optimal control: theory and applications to motor control
PDF
Computational principles in human motor adaptation: sources, memories, and variability
PDF
Experimental and computational explorations of different forms of plasticity in motor learning and stroke recovery
PDF
Design of adaptive automated robotic task presentation system for stroke rehabilitation
PDF
The representation, learning, and control of dexterous motor skills in humans and humanoid robots
PDF
Machine learning of motor skills for robotics
PDF
Towards learning generalization
PDF
Learning reaching skills in non-disabled and post-stroke individuals
PDF
Deficits and rehabilitation of upper-extremity multi-joint movements in individuals with chronic stroke
PDF
Data-driven autonomous manipulation
PDF
Learning controllable data generation for scalable model training
PDF
Quickly solving new tasks, with meta-learning and without
PDF
Impact of enhanced efficacy expectation on motor learning in individuals with Parkinson’s disease
PDF
Artificial Decision Intelligence: integrating deep learning and combinatorial optimization
PDF
Learning affordances through interactive perception and manipulation
PDF
Model-based approaches to objective inference during steady-state and adaptive locomotor control
PDF
Optimizing task assignment for collaborative computing over heterogeneous network devices
Asset Metadata
Creator
Lee, Jeong-Yoon
(author)
Core Title
Modeling motor memory to enhance multiple task learning
School
Viterbi School of Engineering
Degree
Doctor of Philosophy
Degree Program
Computer Science
Publication Date
04/29/2011
Defense Date
03/17/2011
Publisher
University of Southern California
(original),
University of Southern California. Libraries
(digital)
Tag
computational model,motor learning,OAI-PMH Harvest,optimal schedule,stroke rehabilitation
Language
English
Contributor
Electronically uploaded by the author
(provenance)
Advisor
Schweighofer, Nicolas (
committee chair
), Sanger, Terence (
committee member
), Schaal, Stefan (
committee member
), Winstein, Carolee J. (
committee member
)
Creator Email
ethielee@gmail.com,jeongyol@usc.edu
Permanent Link (DOI)
https://doi.org/10.25549/usctheses-m3807
Unique identifier
UC153331
Identifier
etd-Lee-4057 (filename),usctheses-m40 (legacy collection record id),usctheses-c127-462400 (legacy record id),usctheses-m3807 (legacy record id)
Legacy Identifier
etd-Lee-4057.pdf
Dmrecord
462400
Document Type
Dissertation
Rights
Lee, Jeong-Yoon
Type
texts
Source
University of Southern California
(contributing entity),
University of Southern California Dissertations and Theses
(collection)
Repository Name
Libraries, University of Southern California
Repository Location
Los Angeles, California
Repository Email
cisadmin@lib.usc.edu
Tags
computational model
motor learning
optimal schedule
stroke rehabilitation