University of Southern California Dissertations and Theses

The symbolic working memory system

(USC Thesis Other)

The symbolic working memory system

PDF

Download a page range

Download transcript

Copy asset link

Request this asset

Transcript (if available)

Content The Symbolic Working Memory System
by
Nader Noori
A Dissertation Presented to the
FACULTY OF THE USC GRADUATE SCHOOL
UNIVERSITY OF SOUTHERN CALIFORNIA
In Partial Fulllment of the
Requirements for the Degree
DOCTOR OF PHILOSOPHY
(COMPUTER SCIENCE)
December 2013
Copyright 2013 Nader Noori
Dedication
to Alan Baddeley
ii
Acknowledgements
My profound gratitudes rst and foremost go to my research advisors Professor Laurent
Itti and Professor Michael Arbib. Professor Itti's unconditional support allowed me to
invest almost entire my time during last ve years on the same topic which in turn
resulted to an enriched research experience beyond my wildest imaginations. Professor
Itti's renaissance attitude towards research was truly inspiring for me to set no limit
in enjoying both experimental work like an experimentalist and theoretical work like a
theoretician.
Speaking of theoretical work there is no one to whom I wish to express my gratitude
more than Professor Arbib. Professor Arbib's colossal contributions to the eld of theo-
retical neuroscience has provided a shining and bright lamppost under which a vast area
is now available for generations of scientists in search for their lost keys. I have no doubt
that I would end up searching for my own lost key under his bright lamppost no matter
where in this world I had started searching. I happened to have an oce at the end of the
same hallway of Professor Arbib's oce. This gave me a unique opportunity for which I
feel humbled and honoured to spend countless hours in his oce at the other end of the
hallway.
Besides my advisors, I wish to thank the rest of my thesis committee: Prof. Lisa Aziz-
Zadeh, Prof. Fei Sha, and Prof. Justin Wood, for their encouragement and insightful
comments, in particular, Prof. Aziz-Zadeh who was always responsive, welcoming and
receiving even for short-noticed meetings.
My very special thanks go to my wife, Pardis and my daughter Kiana. As much as
I enjoyed engaging in this research and seeking refuge in iLab, my wife, Pardis, had to
iii
bear with uncertainties and hardship of living a student life for a long period of time.
She trusted me in all my decisions and stood by me and I can't thank her enough for
all her support throughout all these years. My beloved daughter, Kiana, who started
her elementary school when I started my research at ISI, now in her middle school, has
become my indispensable source of knowledge for ecient communication.
There is one person outside my intellectual ring and immediate family circle whom I
need to thank as an intellectual mentor and as a family member: Dr. Behanm Salemi. I
am thankful for his friendship and support which started from the rst day I met him on
March 12, 2002. He and his wife Atusa, not only supported me in every step of my work
but also have provided great support for my wife and my daughter too.
I also need to thank Dr.s Hartmut Neven and Prof. Pedro Szekeley at ISI who gave
me the opportunity to enrol in PhD program in Computer Science department of USC. In
particular I would like to thank Prof. Szekeley for oering generous research assistantship,
giving excellent advices and bearing with me for more than two years. He is an excellent
advisor, a great computer scientists and a great photographer.
I would also like to thank Victor Barres and Brad Gasser for not only their moral
support but also for their stimulating and thoughtful interactions. I had the chance to
enjoy countless hours of discussion and exchanging ideas with Victor Barres which added
a lot of color to my intellectual life in the basement of Hedco Neuroscience Building. I
also need to thank all my fellow labmates in iLab with whom I shared a wonderful time
between February 2008 to present day on November 1st 2013.
Finally I am so grateful for the opportunity that I found in this beautiful land in
living in peace and enjoying my freedom. Although I was born Iranian and proud of
my Iranian heritage, I chose to pledge my allegiance to the
ag of the United States of
America and to the Republic for which it stands. My wholehearted allegiance comes from
my subscription to what has founded this great nation in seeking and promising liberty
and justice for all.
iv
Table of Contents
Dedication ii
Acknowledgements iii
List of Figures viii
List of Tables x
Summary xi
I Working Memory: a Review of Models 12
Chapter 1: `The Working Memory' versus `a working memory' 13
1.1 The Working Memory and its Centralized Executive System . . . . . . . 13
1.2 Decentralized Executive Models for `working memory' in Sensorimotor Sys-
tems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
Chapter 2: A Spatial Registry Hypothesis: Towards a Sensorimotor Ac-
count for Executive Functions of WM 27
2.1 Evidences of entanglement of visual-spatial processing and executive task 28
2.1.1 Insight from studies of individual dierences . . . . . . . . . . . . . 28
2.1.2 Overlaps between neural correlates of wm manipulation and visual-
spatial functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
2.1.3 Behaviour Evidences: Interference of executive tasks and visuospa-
tial cognition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
2.2 The Spatial Registry Hypothesis (SRH) . . . . . . . . . . . . . . . . . . . 36
II Exploring Involvement of Visual-Spatial Resources in Executive
Tasks 42
Chapter 3: Tracing Operational Features of Executive Tasks in Eye Move-
ments 43
3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
3.2 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
v
3.2.1 Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
3.2.2 Experiment 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
3.2.3 Experiment 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
3.2.4 Experiment 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
3.3 Outlook . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
Chapter 4: Tracing Impacts of Executive Tasks on Visual-Spatial STM 69
4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
4.2 Experiment 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
4.2.1 Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
4.3 Experiment 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
4.4 General Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
Chapter 5: Dierential Impact of Visual Presentation on Forward and
Backward Recall 85
5.1 Hypothesis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86
5.2 Experiment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
5.3 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
III SWMS: A Sensorimotor Account for a Symbolic Working Mem-
ory System in Intellectual Domain 94
Chapter 6: SWMS: The Symbolic Working Memory System 95
6.1 The Symbolic Interface: the blueprint for a bridge . . . . . . . . . . . . . 98
6.2 Modal and Symbolic Representations in Utility Systems . . . . . . . . . . 100
6.3 A Schema-Driven Distributed Execution Model in Dierent Levels of Analysis102
6.4 Randomly Accessible Working Memory and Location Registry Systems . . 108
6.5 Utilization of Random Access Working Memory for Dual Counting . . . . 112
6.6 Behavioral and Neural Correlates of the Randomly Accessible Working
Memory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
6.7 Serially Accessible Working Memory and Sensory-Articulatory Circulative
System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
6.8 Utilization of Serially Accessible Working Memory for Dual Counting . . 128
6.9 Neural and Behavioral Correlates of Serially Accessible Working Memory 129
Chapter 7: The Case of Immediate Serial Recall from SWMS Perspective142
7.1 Bidirectional recall a challenge for computational models . . . . . . . . . . 142
7.2 Assessment of pure strategies for serial recall tasks . . . . . . . . . . . . . 144
7.2.1 Forward recall with serial access schema . . . . . . . . . . . . . . . 144
7.2.2 Forward recall with random access schema . . . . . . . . . . . . . . 145
7.2.3 Backward recall with serial access schema . . . . . . . . . . . . . . 146
7.2.4 Backward recall with random access schema . . . . . . . . . . . . . 147
7.3 Computational Modeling of LRS for Immediate Serial Recall . . . . . . . 149
7.3.1 Elements of the computational model . . . . . . . . . . . . . . . . 149
7.3.2 Simulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152
vi
Chapter 8: The Case of Concurrent Counting from SWMS perspective 165
8.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165
8.2 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171
8.2.1 Experiment 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171
8.2.2 Experiment 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179
8.2.3 Experiment 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182
8.3 General Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187
References 195
vii
List of Figures
1 Evolution of Central Executive in Baddeley's model of Working memory . 10
2 A simple schema program for explaining a low level behavior in frogs. . . 11
3.1 Schematic view of the paradigm for the mental sorting task. . . . . . . . . 48
3.2 Sample of eye movements during mental sorting of a sequence of ve digits. 50
3.3 Impact of initial presentation method on share of horizontal gaze shifts. . 53
3.4 Gaze shifts compare to presenting items in random locations. . . . . . . . 57
3.5 Comparing directional distribution of gaze shifts during a mental sorting
task for dierent pairs of presentation methods. . . . . . . . . . . . . . . . 59
3.6 Pairwise comparison of Pearson's skewness measure for four pairs of stim-
ulus types across all subjects. . . . . . . . . . . . . . . . . . . . . . . . . . 67
3.7 The average dierence between probability of gaze shifts for symmetric
stimuli. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
4.1 Schematic view testing location change detection sensitivity during mental
tasks. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
4.2 Impact of task engagement rate on visuospatial STM . . . . . . . . . . . . 76
4.3 Spatial selectivity of the impact of mental sorting on visual-spatial STM . 80
5.1 Schematic view of the paradigm for Forward-or-Backward recall task. . . 87
5.2 Impact of presentation method on error rates in forward and backward recall. 90
5.3 All subjects' data for impact of initial presentation method on recall error
for forward versus backward recall . . . . . . . . . . . . . . . . . . . . . . 92
6.1 A pseudo-tabular depiction of The Symbolic Interface (SI) . . . . . . . . . 136
6.2 A depiction of the Symbolic Working Memory System (SWMS) . . . . . . 137
6.3 A depiction of the Location Registry System (LRS) . . . . . . . . . . . . . 138
viii
6.4 A depiction of the Sensory-Articulatory Circulative System (SACS). . . . 139
6.5 The drum metaphor for sequence of operations in Serially Accessible Work-
ing Memory (SAWM) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140
6.6 Maintenance through rehearsal in SAWM system with noisy identity func-
tion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141
7.1 Binding and recall in 1-dimensional model of LRS. . . . . . . . . . . . . . 157
7.2 Two population of solutions with separate central spatial tuning factors . 158
7.3 Model's prediction of positional error . . . . . . . . . . . . . . . . . . . . . 159
7.4 Distribution of tness for two population of answers. . . . . . . . . . . . . 160
7.5 Comparing quality of predictions of positional errors in backward and for-
ward recall. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161
7.6 Prediction of 1-dimensional LRS model for movement error in forward
recall of six items . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162
7.7 Performance of the over-t probabilistic model for forward recall (OPM-F)
and backward recall (OPM-B). . . . . . . . . . . . . . . . . . . . . . . . . 163
7.8 Displacement error for over-t probabilistic model . . . . . . . . . . . . . 164
8.1 Schematic view of the triple-counting paradigm. . . . . . . . . . . . . . . . 172
8.2 Sequences of counting events in dierent paradigms for concurrent counting.191
8.3 Execution time for LBTC and IBTC . . . . . . . . . . . . . . . . . . . . . 192
8.4 Execution times for IBTC for dierent categories of items. . . . . . . . . . 193
8.5 Execution time for IC-LBTC and LBTC . . . . . . . . . . . . . . . . . . . 194
ix
List of Tables
3.1 Linear relationship between skewness of gaze shift distributions during
sorting stimuli of four dierent categories. . . . . . . . . . . . . . . . . . . 64
3.2 Between categories
e
R
2
values for the anti-symmetric regressor for skewness
of horizontal gaze shift skewness. . . . . . . . . . . . . . . . . . . . . . . 64
7.1 Parameters of 1-dimensional LRS in recall task. . . . . . . . . . . . . . . . 153
7.2 An over-t probabilistic model (OPM) for both forward (OPM-F) and
backward (OPM-B) recall. . . . . . . . . . . . . . . . . . . . . . . . . . . . 154
8.1 Mean SE of error measures for all three experiments. . . . . . . . . . . . . 175
8.2 Mean SE of execution times for all three experiments. . . . . . . . . . . . . 190
x
Summary
The capacity of the brain in maintenance of task-relevant information over a short period
of time is known to be crucial for performing a wide range of activities from low-level
perception-action routines in lower animals to high-level intellectual tasks in human.
Working memory is the common term that has been used among dierent communi-
ties to refer to the manifestation of this feature across dierent domains. However,
despite this common agreement, dominant theoretical paradigms for describing work-
ing memory management systems in the domains of low-level/perception-action and and
high-level/intellectual functions follow drastically dierent principles: embedded and dis-
tributed in the low-level domain, disembodied and centralized in the high-level domain.
Given that the human cognitive system functions at both levels in dierent contexts si-
multaneously this question arises whether indeed there are two types of working memory
systems running in parallel under two dierent operational principles in human brain or,
a more parsimonious account can explain all dierent manifestations of working memory
in all domains.
In attempt to achieve a more parsimonious account, a theory is developed for func-
tioning of working memory in the context of high-level and intellectual domain which
accounts for information management during those tasks that feature symbolic infor-
mation processing. This eort was partially motivated by theoretical inconsistencies and
biological/evolutionary implausibility of standard models of working memory in cognitive
psychology with centralized executive paradigm.
The proposed framework demonstrates how novel assemblage of embedded schemas
in existing sensorimotor systems may supply a system for management of symbolically
xi
represented sensory and motor information serving intellectual tasks. In the proposed
framework, strategic and evolutionarily-constrained reuse of sensorimotor resources for
management of respectively spatially-organized and temporally-sensitive information sup-
port random access and serial access schemas for management of symbolic information.
Through grounding access schemas for management of symbolic information in sensorimo-
tor systems we are able to predict ramications of working memory management during
the performance of mental tasks at behavioral and neural levels. A detailed example in
applying this methodology in well-studied cases of forward and backward recall tasks will
be presented with additional computational modeling and the results of simulations.
Our systematic approach in mapping spatial/temporal characteristics of sensorimotor
systems onto access modes provides a symbolic interface to other frameworks and archi-
tectures for describing the symbolically-intelligent mind. Proposed framework provides
for the rst time a neurally-grounded and sensorimotor-based account for management
of symbolic information with embodied cognition prospects with opportunities for exper-
imental validations and applications.
Proposed theory oers ample opportunities for experimental validations and predicts
novel sources of working memory for symbolic tasks. Moreover, through providing a
mechanistic account for management of working memory, the proposed theory, denes a
practical framework for optimization of working memory resources for performing mental
tasks which has potential educational consequences in measuring IQ and optimal design-
ing of mental operations.
Mechanistic specications of the model about details of interactions between function-
ing of visuospatial systems as the main supplier of random access to symbolic working
memory in normal subjects has allowed a number of predictions that are validated in
experimental studies. Chapters 3,4,5 and 8 of this dissertation report the results of four
experimental studies under four dierent experimental paradigms. These studies have
been able to generate signicant result in support of the proposed theory.
xii
Introduction
The term `working memory' is often used to refer to the crucial function of the brain
in temporary maintenance and manipulation of task-relevant information. This function
enables complex behaviors from low-level action-perception routines in lower animals
such as avoiding obstacles in a walking kitten (McVea and Pearson, 2009) to high-level
intellectual tasks such as solving arithmetic problems in a college student (Alloway, 2009;
Logie et al., 1994; McLean and Hitch, 1999).
Despite a general agreement on a practical denition for working memory among
dierent domains and research communities, theoretical accounts for describing its func-
tioning are drastically dierent. On one side and in the domain of high-level and intel-
lectual tasks, this view is widely accepted that the `Working Memory' is a disembodied
construct serving cognition under a centralized executive system { namely the Central
Executive (CE){ (Baddeley, 2012; Baddeley and Hitch, 1974). On the other side, in the
domain of low-level functions of the nervous system, working memories are described as
embedded mechanisms serving sensorimotor functions which are collectively managed in
a distributed or decentralized fashion and in an embodied context (Arbib, 1992). This
dierence re
ects in some way the dierence between the contexts in which management
of information takes place: high-level cognition may involve situations with no direct
reference to the body and its sensory and motor apparatuses, often with single thread of
running operation (e.g. calculating how much I would save by trading in my old gasoline
car for a new electric car), while in the domain of action-perception functions, the body
often works as an actively involved reference (e.g. when I change lane while driving my
car, the information about nearest cars I gathered over the shoulder a few seconds ago,
state of controllers of the car such as the steering wheel and gas pedal are all registered
trough active engagement of my sensory and motor apparatuses).
1
This comparison poses a host of fundamental questions about the relationship between
these two views of working memory given that in human there are contexts for tempo-
rary maintenance of information both in high-level and low-level functions: Whether
indeed there are two types of working memory systems running in parallel under two
dierent operational principles, one for high-level intellectual tasks and one for low-level
perception-action routines. Or if a CE-operated Working Memory can account for all
aspects of temporary information management across all domains simultaneously. Alter-
natively, whether the CE-operated WM is indeed one of working memory systems along
others serving only intellectual domain. In this case under what circumstances this mem-
ory system has evolved, particularly when it seems that symbolic mental tasks are very
recent in our culture of problem solving. Is it possible to use one of these two paradigms to
get to the other one either in explaining underlying mechanisms or functioning principles?
Before any attempts to answering these questions and many more relevant questions
we rst examine some clues from a high level point of view and evaluate relevance of
each of these general models of working memory in answering these questions: 1. what
are biological justications of principal assumptions of each model|either intellectual or
sensorimotor, should be explained as a biological organism's behavior and on the platform
of nervous system. 2. how successful these models have been in addressing those questions
that once motivated them in their native domain function? The hope is that the result
would help evaluate the relevance of above questions and induce a priority for addressing
them.
First, we will discuss in a historical context how the idea of the Centralized Executive
(CE) for the Working Memory was inspired by the architecture of digital computers to lay
out a plan for functioning the working memory beyond a passive storage of information.
We argue that several parallels between symbolic manifestations of high-level cognition
and symbolic processing features of digital computers facilitated the in
uence of digital
computers on cognitive scientists in proposing the idea of Central Executive (gocognitive,
2
2010; Miller, 2003) in parallel with Central Processing Unit (CPU) in Von Neumann's
architecture (Von Neumann, 1982).
The CE which is responsible for management of the content of WM is hypothesized
to perform its job by controlling limited attentional resources (Baddeley, 1992, 1996).
This hypothesis has some attractive features which have helped establish a popularity
for CE-operated WM in the cognitive psychology community. First, it endorses the
notion of information processing as the monumental concept in cognitive science, second it
piggybacks on the popularity of the idea of central executive in the literature of executive
control and third, it relates to the legacy concept of attention in psychology (James,
1890-2011). Yet these attractive features have not been able to cover inability of CE-
operated models of WM in directly addressing the issue of control of information. While
CE in executive control theories is used to explain control of behavior and successful in
explaining the unitary mode of behavior (Norman and Shallice, 1986), CE in WM models
is supposed to explain the control of storage units (see Figure 1). After four decades and
creating an ALL-IN-ONE homunculus concept from CE this goal has proved to be elusive
(Baddeley, 2012; Repovs and Baddeley, 2006) .
The concept of Central Executive of Working Memory also has inspired cognitive
neuroscientists for pinpointing its neural substrates. Yet, more than two decades of neural
studies have not converged to an unequivocal support nor consistent evidence for the idea
of a central executive ( see (Postle, 2006) for a review of evidences). These evidences
suggest that centralize functioning paradigm not only has no evolutionary justication
but also has no solid and consistent biological support either.
Neural mechanistic accounts are more ubiquitous in studies and modeling of lower
and more basic functions, such as sensorimotor functions of the brain, where experi-
mental methodologies are more in
uenced by biological studies rather than psychological
studies, and theoretical accounts are in
uenced by control theoretic formalisms (Arbib
et al., 1987; Jeannerod, 1997; Wiener, 1948) rather than symbolic processing formalisms
3
(see Figure 2). In the realm of sensorimotor functions of the brain, both experimental ev-
idence and theoretical frameworks support the idea of working memory as the temporary
maintenance of relevant information and until is related to the behavior (Arbib et al.,
1987). However, neither experimental evidence nor theoretical frameworks support the
idea of functioning under a central executive system (Arbib, 1992; Monsell and Driver,
2000).
From early studies of neural mechanisms in lower organisms, control theoretic models
were examined to give an explanation for command control mechanism towards explain-
ing the unitary mode of behavior without relying on the concept of a central executive
unit (Kilmer et al., 1969). Thus, the concept of working memory in theoretical context
of control theories relies on embedded executive mechanisms rather than a centralized
executive unit.
In contrast with the idea of CE of WM which is built around a metaphor from ma-
chines to explain behavior of biological organisms (gocognitive, 2010), in an opposite way,
control theoretic models originally were inspired by mechanisms of control in animals in
hope for learning from animals to apply to machines (Wiener, 1948). So, in the domain
of low-level functions of the brain, theoretical models are inspired and stayed very close
to evolutionary constraints and biological facts.
In terms of success in answering native domain questions that motivated the model,
control theoretic model with the concept of embedded executive model have enjoyed
tremendous success not only on the theoretical front (Shadmehr and Mussa-Ivaldi, 2012)
but also on practical front and in lending support to devising and engineering of practical
solution for a range of application from physical rehabilitation (Hidaka et al., 2012) to
robotics (Schaal and Schweighofer, 2005).
So from the perspective of biological justication, CE-based models do not seem to
enjoy a strong support (gocognitive, 2010) compared to embedded mechanisms that are
backed by control theoretic concepts. And at least according the most prominent pro-
ponents of CE models WM these models have been struggling with theoretical problems
4
the way the started some four decades ago (Baddeley, 2012). We will elaborate on some
of those theoretical issues in our review of the background in detail.
So it should be clear that in authors' evaluation of the posed questions about function-
ing models of working memory, embedded and distributed execution models in low-level
domain appear to be more relevant to a general view of functioning of working memory
compare to centrally operated models of high-level and intellectual domain.
Our eort in this work is giving a view of memory management in the domain of
intellectual tasks which, rst, will allow a formal description of behavior at high level
in terms of symbolic processing concepts which conform with psychologist's observations
and, second, will allow a neural mechanistic account consistent with functioning princi-
ples of rudimentary and low-level functions of the brain. Our key assumption here for
building our framework is that underlying information management mechanisms in senso-
rimotor systems are directly involved and responsible for management of information for
intellectual tasks. This assumption is based upon reusing and, bending those information
management mechanisms that have appeared earlier in our evolution, towards achieving
recently gained capacity of performing intellectual tasks.
This view opens to the possibility of an embodied experience of performing intellec-
tual tasks. The assumption of direct involvement of neural mechanisms of sensorimotor
systems suggests that management of symbolic information for an intellectual task is not
exclusively a mental experience and it involves embodied experience situated in senso-
rimotor systems. This stretches the mechanistic aspect of high-level cognitive tasks in
manipulation of symbols to the most basic features of our experience of the physical world
which we are shared with lower organisms.
However, we argue that it is important to formulate working memory management
mechanisms in terms of a symbolic process language. First, symbolic processing concepts
are suitable for description mental aspect of the behavior in intellectual tasks. Second,
this symbolic representation of the mental state can be used for reconstruction of the
entire experience, if, a. operational aspects in the symbolic level are closely mapped onto
5
the supporting sensorimotor mechanisms and b. mechanistic aspects in sensorimotor
functions are closely mapped onto supporting neural mechanisms. The key step here is
mapping symbolic processed to brain mechanisms which are biologically justied. This
is the insight: once symbolic processes of working memory management are translated
into sensorimotor mechanisms then our knowledge about neural basis of sensorimotor
mechanisms are enough to explain `mechanistic aspects of information management' of
symbolically performed intellectual tasks.
Our proposed framework accommodates for such mappings from a mental descrip-
tion onto a sensorimotoric description and then onto a neural mechanistic description.
These mappings are conceptualized using a structural approach described in four levels of
analysis with dierent degrees of abstraction. We refer to this structure as the Symbolic
Interface or SI for short (see Figure 6.1).
A third and very important benet of a full description of the behavior in terms of
symbolic processing concepts is providing the opportunity to connect to any model of
cognitive dynamics through a symbolic interface. There are a number of formalisms that
give a description of symbolically-intelligent mind which in turn rely on a realistic model
for management of information (Anderson, 1996; Derbinsky and Laird, 2010; Laird et al.,
1987; Meyer and Kieras, 1997). Providing a symbolic interface for memory management
will facilitate connecting the model of working memory as an AS-IS module to those
formalisms.
In our framework at the most abstract and symbolic level, functioning of the work-
ing memory system is described using access schemas. For this reason we refer to this
level as the access level or symbolic level. An access schema is a description of memory
management in terms of a limited number of operations which are supported by a utility
sensorimotor system. Access schemas give rise to the concept of integrated manage-
ment mechanism: instead of assuming a bipartite system of `storage' `versus' `execution'
we apply the concept of `working memory access' which has both concepts of storage
and management of information integrated. This integrated perspective of management
6
of memory at the symbolic level is a re
ection of integrated or embedded functioning
mechanism of sensorimotor systems that support those symbolically represented working
memory (wm) management operations.
To account for human behavior in performing intellectual tasks we propose two types
of wm access schemas and thus we describe two dierent types of sensorimotor systems
that can aord operations of those access schemas. A random access is proposed which
allows
exible variable binding to an address space. The random access is supported
by those sensorimotor systems that utilize spatial encoding and access mechanism for
supporting object oriented action praxes. A serial access mode is also proposed which
augments working memory space with less
exible and yet crucial working memory sys-
tem. This serial working memory system is supported by sensorimotor systems with
temporal encoding characteristics which are specialized in communication praxes.
Supporting utility sensorimotor systems are describe at two levels 1. a generic level 2.
an instance level (see Figure 6.1). In the generic level which is closer to the access level,
sensorimotor systems are described in a rather generic form using canonical mechanisms
that help explain the functioning of generic systems with respect to functions of the
sensorimotor systems in supporting their original praxes. We respectively refer to those
generic systems that support random access and serial access as the Location Registry
System (LRS) and the Sensory-Articulatory Circulatory System (SACS) (see Figure 6.2).
Presentation of generic model of sensorimotor systems helps promote a more abstract
description which is easier to relate to symbolic concepts at the access level and yet
provides the opportunity to support those generic mechanisms with actual instances of
sensorimotor mechanisms. This latter stage takes place at the instance level. Sensori-
motor systems at the instance level are actual service providers and are responsible for
supporting those functions that are described in generic systems. We argue that dierent
instances of sensorimotor systems that are equipped with designated symbolic represen-
tation may satisfy the conditions of a given generic sensorimotor system. Finally at the
7
fourth and the neural level those mechanisms of actual systems are associated to brain's
neural resources.
We refer to the entire of this structure which is described in four levels and organized
around assemblage of two types of access modes, as the Symbolic Working Memory
System or SWMS (see Figure 6.2). We give several examples of intellectual tasks to
explain dierent concepts with dierent degrees of detail.
We demonstrate the importance and the power of our proposed framework by applying
our analysis to forward and backward recall tasks. In particular, we explain how serial
access schemas impose strong limitations on recalling a list of items in backward order,
because these schemas are inherently sequential. Such limitations do not apply to random
access systems, which can access memory items in any order. We use the result of
this analysis to explain the inadequacy of those systems which support serial access (for
example with support of phonological systems) in backward recall, and the advantage of
those systems that support random access (for example with support of ocular resources).
Furthermore, we present the result of the simulation of a simple computational model
of our location registry system (LRS) in an immediate recall task to show that a location-
based random access strategy can account for human performance in backward recall.
Moreover, we show that forward recall is also possible using such a strategy with a per-
formance level comparable to serial-access strategies for forward recall. This, in particular
is very important in explaining serial forward recall in young children, which unlike adults,
do not appear to utilize their phonological system as the provider of serial encoding and
access to the content of working memory in forward recall tasks (Cowan, 1993; Gathercole
and Hitch, 1993; Hitch et al., 1989).
Finally in our general discussion we will discuss dierent aspects of our proposed
model in contrast with existing paradigms to give a view of what our model can oer in
addressing some of persisting issues have been long facing centralized executive models.
In particular we highlight experimental validation opportunities which can distinguish
our model from standard models. We also discuss theoretical consequences and practical
8
opportunities that our model would oer by assuming sensorimotor mechanisms as the
backbone of working memory management system in high-level cognitive domain.
9
Central
Executive
Phonological
Loop
Visual-Spatial
Sketchpad
Three-Component Model 1974
A
B Central
Executive
Visual-Spatial
Sketchpad
Phonological
Loop
Episodic
Buffer
Episodic
LTM
Language Visual
Semantics
Mutli-Component Model 2012
Fluid Systems
Crystalized Systems
Figure 1: From original three-component model (shown in panel A (Baddeley and Hitch, 1974))
to multi-component model (shown in panel B (Baddeley, 2012)). the relationship between the
Central Executive (CE) and storage units has not changed. In the original three-component model,
CE was was given the role of management of information within and between phonological loop and
visuospatial sketchpad. Yet this role is still unknown. As three-component model evolved to multi-
component model Baddeley postulated a role for CE in regulating executive attention resources
which still lacks enough detail to account for management of information in slave storage subunits
of CE. Baddeley's description of CE as `the most important but least understood component' of
his model (Repovs and Baddeley, 2006) is an acknowledgement of theoretical challenges facing CE-
dependent models in giving a satisfactory account for the functioning of human working memory.
10
Figure 2: Analyzing a behavior in terms of mechanisms: schema program for frog approach
behavior. Panel A demonstrates a \naive" schema for the behavior of frogs in approaching small
moving objects and avoiding large moving objects. Panel B takes into the account the result of
lesioning the pretectum in frog's behavior to correct functioning mechanisms (From (Arbib et al.,
1987)). In this sense schema theoretic description of the behavior provides a structure for gathering
knowledge and incremental improvement and tuning of details. As new evidences emerge form
testing predictions of the model there might be opportunities for improving the model. In this
sense Schema theory serves as a language that allows
exible and systematic improvement of the
model.
11
Part I
Working Memory: a Review of Models
12
Chapter 1
`The Working Memory' versus `a working memory'
1.1 The Working Memory and its Centralized Executive
System
Historically, Working Memory (WM) as a term, was popularized by cognitive psycholo-
gists to refer to the system that helps temporary maintenance of task relevant information
to make it accessible to cognitive processes associated with high-level intellectual tasks
(Baddeley, 1992; Baddeley and Hitch, 1974). The program for studying Working Mem-
ory which is marked by Baddeley and Hitch's seminal work (Baddeley and Hitch, 1974)
shares with its predecessor program marked by publication of Atkinson and Shirin's
seminal work (Atkinson and Shirin, 1968) in rejecting a unitary model of memory ()
and accepting two modes of memory based on temporal accessibility of information: a
long-term memory (LTM) versus a short-term memory (STM). The dichotomy of STM
versus LTM, in part, was established by the report of the neuropsychological case of
patient H.M. (W.B. and MILNER, 1957).
Another in
uence on initial theories of working memory came from George Miller's
seminal work in using Shannon's information theory (Shannon and Weaver, 1948) in
estimating the capacity of immediate memory in encoding sensory channelled information
(Miller, 1956). Miller's work provided an opportunity to look at the relationship between
sensory apparatus and cognitive status estimated by capacity of learning and performing
13
tasks in the light of information processing concepts. This eort was one of dening
events of `cognitive revolution' and was the start of an era of in
uence of information
processing formalism on study of human memory.
However, Baddeley and Hitch's program was explicitly dierent from its predeces-
sors in several ways. First Baddeley and Hitch assumed a double dissociation between
LTM and STM. This double dissociation was established as the result of the report of
a neurpsychological case of sever limitations in verbal short-term memory and yet in-
tact long-term memory (Shallice and Warrington, 1970). This double dissociation helped
imagining working memory as a separate construct rather than a functional stage of
memory function. Moreover, Baddeley and Hitch accepted a multi-modal image of STM
by including a visuospatial short-term storage in addition to phonological-verbal memory
which was previously at the focus of research in the studies of the human memory.
Baddeley and Hitch also had an emphasis on the concept of process. Craik and Lock-
hart (Craik and Lockhart, 1972) underscored this matter that the duration of maintaining
information in STM is not the only factor that determines what establishes a represen-
tation in the long-term memory. It was shown that the type of the process performed on
the information is also a key factor in formation of long-term memory of the material.
This was in contrast with Atkinson and Shirin's assumption that duration of passive
maintenance of information in STM (or WM) guarantees a representation in long-term
memory.
However, the most important dierence in Baddeley's program is associated to in-
tegrating the concept of control of information to give the image of a self governing
construct which thereafter was referred to as the Working Memory. Two modal storage
units are capable of storing task relevant information in a symbolic way and need to have
mechanisms for exchanging those representations. Baddeley and Hitch moved towards a
model of working memory with its own executive functioning system for managing those
storage units. This management mechanisms is manifested in assuming a central execu-
tive unit at the center of their model and in charge of control of information within and
14
between two storage units. While storage units were in charge of holding information in
rather a passive and static form, the CE was assumed to be in charge of all unknown
management processes.
The introduction of the concept of Central Executive was a crucial step towards
picturing an independent machinery which maintains information in a short-term state
available to cognitive processes needed for other high-level cognitive tasks. Initially the
CE was nothing but a place holder for very important controlling functions. The the-
oretical formulation of functioning principle of the central executive (CE) and the way
its functioning is explored via engaging subjects in symbolic information processing tasks
ties Baddeley's model very closely to that branch of information processing formalisms
that are inspired by Turing's computational model and some of technical and theoretical
adaptations.
The in
uence of Baddeley's model from symbolic processing formalisms is deep and
yet very subtle. The most apparent aspect of this in
uence is re
ected in the structure
of the model which resembles the dichotomy of control and storage in Von Neumann's
architecture for digital computers. Von Neumann's architecture proved to be crucial in
materialization of Turing's general computing machine. In Von Neumann's architecture
functions of control and storage is given to separate units. Structural relationship be-
tween the Central Processing Unit (CPU) and the primary memory is very similar to the
relationship between the Central Executive (CE) and storage units. This analogy between
digital computers and Baddeley's concept of Working Memory (gocognitive, 2010) can be
also traced in the way that functioning of CE is believed to be relevant to general picture
of cognition and where it can be observed. Baddeley has used concurrent task paradigm
to isolate the role of each of components of its model in dierent tasks and cognitive pro-
cesses. It is widely assumed that engaging the CE is possible through engaging subjects
in particular tasks known as executive working memory tasks that feature algorithmic
symbolic processing and beyond simple maintenance of information.
15
A review of the literature shows that whenever the functioning role of CE is the sub-
ject of scrutiny tasks such as mental arithmetic, random number generation, concurrent
counting, backward counting, digit forward or backward span tasks, n-back recall tasks,
and reordering digits or characters, are used along other type of tasks. So, the picture
that has emerged from functioning of CE is indeed as the result of engaging subjects in
processing some symbolically represented information in an algorithmic way which all
rely on processing of information stored in modal storage systems. An important aspect
of accepting two modalities{which previously had not been suciently emphasized on{is
that it implies a symbolic representation of tasks items in each of those storage mode
unless task items are exactly in the same mode as task items. A list of items which
are presented visually may be maintained in WM via their symbolic and arbitrary (thus
symbolic) phonological traces.
This view is very close to what is associated to the concept of intelligence in general
(Pylyshyn, 1984, 1989) and human intelligence in particular (Carroll, 1996). Previous
works in theories of computation by Turing and others gives a view of intelligence as
exemplied by a process that involves manipulation of symbols (Newell and Simon, 1976;
Newell et al., 1972; Pylyshyn, 1984, 1989; Turing, 1936, 1950) in a way that is comparable
both in human and machine (Newell and Simon, 1976; Newell et al., 1972; Pylyshyn, 1984,
1989; Turing, 1950).
This symbolic intelligence has been the subject of even an older trend in psychology
and in measuring human intelligence in late nineteenth century (Galton, 1883; Herrnstein
and Murray, 2010) and early twentieth century (Herrnstein and Murray, 2010; Spearman,
1904) inspired by statistical analyses of performance of school students.
Baddeley's emphasis on relating and exploring functioning of CE of his WM model
with tasks that feature symbolic processing has created a parallel exchange with eorts
for measurement of cognitive performance in the context of individual dierences (Engle,
2002; Engle et al., 1999, 1991) and measurements of
uid intelligence (Ackerman, 1986;
Ackerman and Heggestad, 1997; Ackerman et al., 2005). Moreover this emphasis has put
16
Baddeley's theories in the forefront of studies of human behavior in abstract problem
solving (Hambrick and Engle, 2003; Robbins et al., 1996) and mathematical cognition
(Berch, 2008; Bull et al., 2008; Kaufmann, 2002; Logie et al., 1994; Mabbott and Bisanz,
2008; McLean and Hitch, 1999; Wu et al., 2008) which has resulted to a more detailed
image of CE in relationship with intellectual tasks with symbolic processing nature.
We argue that Baddeley and Hitch with their focus on short-term representation
of task relevant material in sensory modes serving intellectual tasks featuring symbolic
processing in practice have selected an important type of tasks that are highly dependent
on cortical sensorimotor resources in management of their information. Moreover, we
argue that two storage systems of Baddeley and Hitch's model are indeed proxy for two
categories of sensorimotor systems which are working based on two dierent principles.
While Baddeley and Hitch have focused on the modality of in our program we focus on the
management aspect of working memory and nature of sensorimotor encoding in terms of
augmenting the information with spatial or temporal attributes. These are indeed those
aspects of Baddeley and Hitch's model that we share.
In the next section we will discuss that as much as we disagree with Baddeley's
description of functioning of working memory in this realm we agree with him on the
importance of the domain of problems that he and his colleagues have tried to isolate. We
believe that they have isolated a set of functions in the brain that heavily relies on cortical
and sensorimotor resources of the brain which are better described in a decentralized and
embedded executive models and moreover indeed could have emerged from sensorimotor
resources of the brain already in place serving perception-action routines.
The most problematic aspect of Baddeley and Hitch's program which |not-surprisingly|
is accepted across dierent domains is the centralized description of functioning of the
working memory. The fact that models of working memory have separated along the na-
ture of representation and not on executive functioning model is a testimony to arbitrary
nature of their assumption on one side and the homunculus nature of their central exec-
utive on the other hand which is indeed a black box which can handle any question just
17
because it gives no specication and leaves every thing in hands of unknown mechanisms
of an ALL-IN-ONE component.
This xation on intellectual tasks with symbolic processing features has originated
from the occupation of psychologists in the early twentieth century with understand-
ing what constitutes intelligence, measured by the ability to perform intellectual tasks
(Herrnstein and Murray, 2010; Spearman, 1904, 1927-2006). Earlier psychologists had de-
scribed these capacities using concepts of energy or power (Carroll, 1996, p.8) (Spearman,
1927-2006, p.137).
The idea of separating control and storage was channelled to theories of cognitive psy-
chology by Baddeley and Hitch's seminal three-component model for Working Memory
(Baddeley and Hitch, 1974) and marked a paradigm shift in theoretical descriptions of
cognition in terms of controlled processing of information. Baddeley and Hitch assumed
a similar functional separation between the act of control and storage in humans' Work-
ing Memory. The control was assigned to a central executive unit, named the Central
Executive (CE), which controls upon two slave storage units, named phonological loop
and visuospatial sketchpad. The functional separation of control and storage has been
the most in
uential aspect of Baddeley and Hitch's proposal, which has been preserved in
all revisions of their model by Baddeley (Baddeley, 1992, 2012, 1983) and has in
uenced
all major theories of working memory in cognitive psychology (see (Miyake and Shah,
1999)).
Yet, the idea of separating control and storage did not prove as fruitful for modeling
human Working Memory as it did for digital computers. This is mainly related to unspec-
ied functional roles of CE in controlling storage units or the content of working memory.
Baddeley's description of CE as `the most important but least understood component' of
his model (Repovs and Baddeley, 2006) is an acknowledgement of theoretical challenges
facing CE-dependent models in giving a satisfactory account for the functioning of human
working memory. Baddeley himself in several occasions has phrased the theoretical status
18
of the CE as a homunculus (Baddeley, 2012, 1996, 2002, 2003b; Repovs and Baddeley,
2006) or a rag bag of unanswered questions (Baddeley, 1996; Repovs and Baddeley, 2006).
What is known about the executive role of the central executive, for the most part,
is elaborated by Baddeley and colleagues. Inspired by Norman and Schallice's idea of
the Supervisiory Activating System (SAS) (Norman and Shallice, 1986), Baddeley has
proposed that CE plays a role in controlling limited resources of executive attention
(Baddeley, 1996; Repovs and Baddeley, 2006). These roles are postulated to be `the
ability to focus, to divide and to switch attention and the ability to relate the content
of working memory to long-term memory' (Repovs and Baddeley, 2006). This view of
function of CE in Working Memory is in parallel with concept of a central executive
in models of executive control. The realm of executive control the appealing aspect of
a central executive is related to its easily explanation of the mode of the behavior. In
this sense the CE of the Working Memory may explaining the data from concurrent task
performance however it still does not specify how it exerts its controlling role on storage
units.
Baddeley's the most specic assertion about the role of CE in controlling its slave
storage units relates CE to manipulation of information within the stores, so that simple
representation and maintenance are independent of the CE (Repovs and Baddeley, 2006).
This latter specication provides a non-trivial distinction between roles of CE in dierent
tasks based on their information manipulation requirement. Accordingly, working mem-
ory tasks that draw on executive resources or CE which are known as working memory
executive tasks come in contrast with simple maintenance of information which are sup-
posedly non-executive and apparently independent of the CE. This distinction in turn
can be used for designing experimental paradigms for underpinning CE's corresponding
neural structure which is of interest to cognitive neuroscientists.
Neurobiological studies of working memory started independently and by discovering
sustained single unit activity of neurons in prefrontal cortex (PFC) of performing monkeys
during the delay period of a delayed saccade task (Funahashi et al., 1989; GoldmanRakic,
19
1987). This sustained activity was interpreted as the evidence for maintenance of task-
relevant information in the brain, which bore resemblance to the concept of WM in
cognitive psychologist terminology (Goldman-Rakic et al., 1992). This parallel between
two elds formed a common ground for shaping future cognitive neuroscience studies of
working memory.
In the coalition of cognitive psychology and neuroscience, multi-component model
of working memory has inspired a host of neuroimaging and neuropsychologcial studies,
which in turn have produced a sheer volume of neural evidence hardly reconcilable with
the idea of separate structures for control and storage (see (Postle, 2006) for a detailed
list of these studies). In particular, no clear evidence of a neural structure exclusively
involved in WM executive tasks has been found yet. Every neural structure known to
be engaged in WM executive tasks also is shown to be engaged in maintenance of some
domain-specic information too (see (D'Esposito et al., 2000) and (Curtis and D'Esposito,
2003) for a review focused on the role of PFC, see (Olson and Berryhill, 2009) for a review
focused on the role of parietal regions).
Lack of supporting evidence for dissociation between neural substrates of maintenance
versus executive functions has invited two speculations among cognitive neuroscientists:
1. maintenance of information is more than storage and involves both storage and exec-
utive resources together (D'Esposito et al., 2000; D'Esposito, 2007; Jonides et al., 2008b)
2. the act of control is an emergent property of the brain and not exerted from neural
structures separable from storage. Along the former draw, one may assume that `control'
and `storage' are structurally separate, yet the goal of isolating neural structures asso-
ciated exclusively with the CE may not be achieved by employing WM executive tasks
versus maintenance tasks.
The latter draw, however, dismisses a clear distinction between neural substrates of
storage and control (O'Reilly and Frank, 2006; O'Reilly et al.; Postle, 2006). This idea has
recently invoked some attempts to replace the concept of central management of working
memory with an integrated, decentralized or distributed execution model for intellectual
20
tasks (Chatham et al., 2011; Frank et al., 2001; Hazy et al., 2006; O'Reilly, 2006). How-
ever, these attempts are rather bottom-up in their approach; they build upon speculated
neural encoding features of dierent brain regions involved in intellectual working mem-
ory tasks to achieve a distributed execution model for management of information. Yet,
a possible cost for following a bottom-up approach is isolating form psychological and
broader evolutionary contexts in theorizing from very low-level perspective.
In this work we present an alternative distributed executive model for working memory
in a rather top-down manner which is framed rst, in a psychological context for the
description of the behavior and second, in an evolutionary context for explaining WM
management mechanisms. In the next section we will discuss the importance of keeping in
relationship with psychological context in description of working memory in intellectual
tasks. We will also discuss the importance and benets of theorizing about functional
principles of working memory in an evolutionary contexts in both next section and in our
general discussion.
1.2 Decentralized Executive Models for `working memory'
in Sensorimotor Systems
Distributed executive paradigms have been long applied to analyzing mechanisms of
lower-level functions in biological organisms, in particular those functions that are more
directly related to action-oriented behavior and action-perception routines. In this do-
main, control-theoretic formalisms have the most in
uence. In fact, Norbert Wiener's
magnum opus, Cybernetics, was inspired by observations of interplays of perception and
action in biological organisms.
Cybernetics is a system theoretic formalism of control in animals and machines based
on the concepts of closed loops of information and feedback (Wiener, 1948). In this
formalism the act of control is inseparable from the
ow of information within an organ-
ism's sensorimotor system and between the organism and its environment. In this sense,
21
Wiener's idea of regulation of information towards control of actions is in contrast with a
centralized paradigm, exemplied by Von Neumann's architecture. Wiener's control the-
oretic approach has suitable theoretical elements for mechanistic description of low-level
biological functions, yet some adaptations and extensions proved to be crucial in applying
those concepts to analyzing and modeling sophisticated behavior in living organisms in
terms of neural mechanisms.
The schema theory is one of theoretical extensions of Wiener's proposal, introduced
by Michael Arbib as a general and formal tool for analyzing neural basis of behavior
in living organisms (Arbib, 1992). Schema theory is the result of amalgamation of the
previous concepts of schema in cognitive psychology (Bartlett, 1932; Piaget and Cook,
1952) and neurobiology (Frederiks, 1969) with the systems-theoretic concept of control
in Cybernetics (Wiener, 1948). Schema theory is a generalization of previous attempts
by McCulloch and his colleagues to determine the mode of behavior without relying on
a central executive or executive control (Arbib, 1992; Kilmer et al., 1969).
Arbib's schema theory gives a description of behavior in an `active organism' which
seeks information from the world to pursue its chosen course of action, supported by
cooperative computation of modules which encapsulate what is known about how per-
ception maps onto action (Arbib, 1992). The relationship between behavior, mechanisms
and neural resources is an intrinsic feature of schema language. This formalism provides
tools for analyzing functioning of the system at dierent levels of abstraction in an inter-
connected manner (Arbib, 2003; Jeannerod, 1997). According to Arbib \... schema theory
complements neuroscience's well-established terminology for levels of structural analysis
... with a functional vocabulary, a framework for analysis of behavior with no necessary
commitment to hypotheses on the localization of each schema ..., but which can be linked
to a structural analysis whenever appropriate. Schemas provide a high-level vocabulary
which can be shared by brain theorists, cognitive scientists, confectionists, ethologists ..."
(Arbib, 1992).
22
Arbib and his colleagues have applied a schema theoretic analysis to a range of func-
tions, starting from their neurethological studies of simple behavior in lower animals such
as frogs and toads (Arbib, 1980; Arbib and Liaw, 1995) (see Figure 2), to complex behav-
ior such as language comprehension and language production in humans (Arbib, 2005;
Arbib and Lee, 2008; Arbib et al., 1987).
Later on, the concept of `working memory' was added to schema theoretic analyses of
more complex behavior wherein maintenance of task-connected information is necessary
for explaining the behavior. This concept explicitly has appeared in Arbib's work as the
capacity of maintenance of task-relevant information as long as necessary and relevant
to behavior (Arbib, 1987). In these analyses and for computational purposes, a neural
network which is capable of maintaining representations of task-relevant information rep-
resents the `working memory'. Typically, behavior of such network is determined by the
relevant biological context and through adjusting some network's relevant parameters.
Hence, speculation about functioning of working memory and examining its features in
these scenarios is often carried out with reference to a target behavior with biological and
adaptive relevance to the organism. In this sense, a behavior of biological or adaptive
value signies the function of maintenance of information in the system. Accordingly, de-
scription of memory management is often presented in the context of another function or
target behavior with fundamental biological relevance rather than a separate construct.
This biological context can be established even in the case of high-level functions such
as human language (for example adaptive value of communicating information about
events and intentions). Yet this approach is not easily applicable or at least has not been
applied to the case of working memory in the domain of intellectual tasks, for that the
biological context in this domain often is not clearly present. Most important intellectual
skills, such as mentally comparing or adding multi-digit Arabic numerals, have become
relevant to our life no later than several hundred years ago (Herrnstein and Murray, 2010,
p.26), not likely enough time to shape humans' working memory management system in
the domain of intellectual tasks. A same argument applies to almost every intellectual
23
skill related to our symbolic problem-solving culture. It is not obvious that a particular
intellectual skill could have driven evolution of our intellect and could have paved the
way for forming a legacy memory management system.
While none of these intellectual tasks may feature an immediate biological relevance of
their own, one's ability to perform each of them might be indicative of an important char-
acteristics of her or his mind (Carroll, 1993; Spearman, 1904). The anecdote of `having
genius,' which has been used to refer to exceptional intellectual abilities of some individu-
als, later was replaced by a concept driven from signicant statistical correlation between
performance on dierent intellectual tasks (Galton, 1883; Spearman, 1927-2006). This
statistical correlation suggests that intellectual skills rely on the same set of resources,
which aect performance of all intellectual tasks. The notion of general intelligence is
directly associated the performance in learning and performing mental skills. It has been
argued that working memory is not equivalent to general intelligence (Ackerman et al.,
2005) as other factors such as interests, motivations and personality may play a role in
determining one's performance (Kanfer and Ackerman, 1989). Yet one may still think of a
core intellectual capacity re
ecting typical performance which interacts with motivational
and personal factors (Ackerman and Heggestad, 1997). Maybe
uid general intelligence
which is shown to strongly correlated with WM and less correlated with STM (Engle
et al., 1999) is a better estimation of this mental capacity. Nevertheless, one may think
of using psychological concepts for establishing a working context for working memory
in the domain of intellectual tasks rather than a pure biological context. This context
allows talking about individuals' capacity in performing intellectual tasks, independently
of particular instances of intellectual skills.
In this context, working memory can be dened in close relationship with the intellec-
tual capacity (probably estimated by
uid intelligence), as a proxy for all resources that
collectively act as a general-purpose system that enables performing intellectual tasks by
facilitating cognitive process. This psychological context bears similarity to the original
motivations behind the independent study of working memory as a separate functional
24
construct that enables mental capacity which can be described in terms of information
processing concepts.
Yet, we argue that neither intellectual capacity nor its associated working memory
are manifestations of a separate and dedicated construct and hence may not properly be
explained in terms of a dedicated and disembodied system.
In contrast, we argue that our intellectual capacity is not a legacy of particular mental
skills and indeed has emerged from the aggregation of a number of capacities associated
with rudimentary functions with biological adaptive values. In particular, the capacity
of maintenance and management of information in sensorimotor systems, which have
enabled crucial functions such as ecient object manipulation or eective communication,
are utilized towards management of information in the intellectual context and thus have
helped emerge the intellectual capacity. This implies that management of information
for intellectual tasks is shared between a number of systems each of which describable in
terms of control theoretic concepts and in a distributed fashion.
Despite arguing against a single and dedicated system in charge of memory manage-
ment for performing intellectual tasks, we argue that it is practically useful to think of
working memory as a collective term which sums up all mechanisms and functions that
enable mental capacity { or perhaps manifestation of
uid intelligence. This allows talk-
ing about what contributes to individual dierences in intellectual skills in terms of a
collective concept. Thus we share our target of study with the cognitive psychology of
working memory, yet we rely on the ndings about rudimentary functions of the brain as
building blocks of our theoretical framework.
In our view rudimentary functions are those functions with clear biological and adap-
tive relevance, which should be sought as the source of resources and mechanisms for
working memory manipulation in intellectual tasks. While we emphasize on retaining a
psychological perspective in exploring and describing human cognition in the intellectual
domain, we maintain that a complete picture should include neurobiological mechanisms.
Exploring mechanisms of working memory management can be informed by evolutionary
25
trends and by focusing on those rudimentary functions with adaptive importance. Evolu-
tionary arguments provide a unique and indispensable source for understanding existing
features of mechanisms in the living organism and thus understanding foundations of
psychological features of the mind (Tooby and Cosmides, 1992, 2000).
The present work is an attempt for devising a framework for management of working
memory in the high-level intellectual domain, by assembling working memory manage-
ment mechanisms in sensorimotor systems that serve rudimentary functions. In our view,
memory management for intellectual tasks is made possible by collaboration of several
utility systems, each of which accommodating for a particular working memory manage-
ment schema necessary to serve a particular praxis of crucial biological relevance. So, we
explicitly bring the evolutionary precedence of working memory management mechanisms
into account and hence, as the result, we propose a neural mechanistic account which is
also evolutionarily and biologically plausible.
26
Chapter 2
A Spatial Registry Hypothesis: Towards a Sensorimotor
Account for Executive Functions of WM
This chapter rst reviews evidences that suggest a role for visual-spatial system in ex-
ecutive functions of the working memory. In the second section of this chapter, these
evidences are used as a basis for proposing a hypothesis for a functional role for visual-
spatial system in working memory management. We will refer to this hypothesis as the
Spatial Registry Hypothesis (SRH). This hypothesis present a view of a semi-autonomous
subsystem for management of working memory which works next to other subsystems to
provided a functionality for the working memory components which can be better de-
scribed with the concept of `access' rather that storage. The Spatial Registry Hypothesis
describes a component which allows random access to the content of working memory.
We will discuss that because of this feature of the visual-spatial system in organiz-
ing task relevant information in the space for parallel processing and via random access
this system makes a suitable candidate for supporting random access to the content of
working memory in intellectual tasks. The only step needs to be taken is using a sym-
bolic representation for task relevant items in visual-spatial system. We will generalize
on these feature to present our Symbolic Working Memory System (SWMS), as an alter-
native theory for working memory system in the realm of intellectual tasks based in Part
III. SWMS extends the assumption of utilization of sensorimotor systems via symbolic
27
representation to explain a serial access in addition to random access which is mostly
described in this chapter.
2.1 Evidences of entanglement of visual-spatial processing
and executive tasks
In section 1.1 it was brie
y mentioned that neurobiological studies have not been able
to establish a dedicated neural substrate to the CE. It was mentioned that every region
involved in executive tasks is also shown to be implicated in maintenance of information
in either verbal mode or visuospatial mode. In particular, neural evidences collected
across dierent studies suggest that the fronto-parietal network associated to executive
tasks is also engaged in spatial tasks in general and working memory visual-spatial tasks
in particular.
All these evidences suggest a functional entanglement between visuospatial process-
ing/abilities and executive functioning of the working memory system manifested mainly
during performance of executive tasks. Here these evidences are are organized and pre-
sented in three categories in the following subsections.
2.1.1 Insight from studies of individual dierences
Statistical analysis of individual dierences in performing intellectual tasks has a long
tradition in psychological studies which is tied to the development of major statistical
tools and methods in late nineteenth century and early twentieth century (Fienberg, 1992;
Spearman, 1904, 1927-2006). Statistical models describing trends in data of individual
dierences in large batteries of tests have played a critical role in development of theories
of intelligence. It has been long argued whether these trends can be associated to a
general capacity (G) (Spearman, 1927-2006) or multiple factors should be regarded to
explain these individual dierences (Carroll, 1996).
28
In search for viable latent variables with mechanistic or functional relevance to human
behavior that in the meantime can explain the variety of observations in factor analyses
some researchers have looked at oshoots of working memory frameworks. A milestone
in this regard was achieved by development the concept of the working memory span by
Daneman and Carpenter (Daneman and Carpenter, 1980) that would allow measuring
the amount of temporarily maintained information (Turner and Engle, 1989) which would
also show high correlation with performance in important cognitive functions such as
reading comprehension, language comprehension (Daneman and Merikle, 1996), spelling,
acquisition of logic and following directions. Working memory span in these studies is
measured in complex tasks known as span tasks that combine both storage function and
processing concurrently. Span tasks typically are performed in a continuous sequence of
stages and at each stage subject engages in a processing tasks in addition to maintaining
an increasing list of items. Dierent processing tasks have be used in dierent span tasks
from reading and comprehension of sentences (Daneman and Carpenter, 1980) to arith-
metic operations (Turner and Engle, 1989). Working memory span is typically dened
as the maximum number of stages that performance in processing phase meets a certain
threshold which in the meantime is equal to the number of maintained items.
The importance of WM span measures is related to the fact that they predict the par-
ticipants' performance on dierent cognitive tasks and intelligence test better than simple
word or digit span ( referred to as short-term memory span) (Daneman and Merikle, 1996;
Engle et al., 1999, 1991). This dierence between predicting power of short-term memory
span and WM span is proposed to be associated to dierential involvement of controlled
attention WM span and STM span tasks (Kane et al., 2001). According to this view WM
capacity equals to STM capacity plus controlled attention ability (Miyake et al., 2001).
This view is highly compatible with the standard model of working memory in that
WM span tasks refer to those tasks that draw on CE in addition to phonological loop while
STM span is a measure that maps onto the capacity of phonological in passive mainte-
nance of information. Given the parallel between visuospatial sketchpad and phonological
29
loop proposed by the standard model, some researchers entertained this idea that a similar
relationship between simple and complex working memory spans should be also observed
in the case of maintenance and processing visual-spatial information (Miyake et al., 2001;
Shah and Miyake, 1996).
Shah and Miyake (Shah and Miyake, 1996) devise a simple spatial span task (maintain-
ing spatial orientations indicated by arrows) and complex spatial span task ( maintaining
spatial orientations along with performing mental rotation). They also tested their sub-
jects on set of spatial abilities and contrary to their original prediction they found that
STM spatial span can predict spatial abilities as well as WM spatial span. Miyake et al
evaluated the extent to which spatial STM span and spatial WM span can be distin-
guished in terms of their predictive power with respect to a larger battery of spatial tests
in the context of individual dierences and included abilities in Spatial Visualization,
Spatial Relations and Visuospatial Perceptual Speed. As the result of their evaluation
the reached to this conclusion that in the spatial domain, `the distinction between WM
and STM span tasks may not be as clear-cut as in the verbal domain' (Miyake et al.,
2001).
These results have been reproduced in the context of large scale voxel-based-symptom
mapping on populations of patients with brain damage (Barbey et al., 2013; Gl ascher
et al., 2010, 2009). Barbey et al in a recent lesion mapping study administered a battery
of neuropsychological tests including the Wechsler Adult Intelligence Scale (WAIS) and
n-back task. They applied a latent variable modeling to obtain error-free scores of
uid
intelligence and working memory followed by voxel-based lesion-symptom mapping to
elucidate their neural correlates (Barbey et al., 2013). Their latent variable analysis
revealed a strong correlation between working memory manipulation tasks (measured by
performance in sorting and arithmetic) and verbal/numeric working memory (measured
by performance in digit forward and backward) which also mapped onto a substantial
overlap between underlying brain regions mostly distributed in left hemisphere at the
site of inferior parietal cortex and posterior regions of superior temporal cortex. These
30
two factors also showed strong correlation with Spatial Working Memory (measured by
performance in spatial forward and backward span) and overlap in underlying regions
mostly focused on right posterior parietal cortex (PPC).
We will give a more detailed review of neural evidences that suggest a strong cor-
relation between functioning working memory in executive tasks and spatial abilities in
2.1.2.
2.1.2 Overlaps between neural correlates of wm manipulation and visual-
spatial functions
The contrast between executive tasks and passive maintenance (or non-executive) (Repovs
and Baddeley, 2006) which conceptually establishes a functional distinction between the
Central Executive and storage components based on behavioral evidences, also, can be
utilized for investigating into neural substrates of the Central Executive. In particular,
neuroimaging techniques which allow dierentiate brain activities in functioning subjects
based on their engagement in executive tasks versus non-executive tasks would allow
isolating neural substrates of the CE by contrasting between indices of brain activities.
Yet, these studies up until now have not been able to generate an unequivocal result.
One of diculties in this regard is related to unspecied roles of the CE in management
of storage components in the standard model. Neither the standard model nor other CE-
based working memory models have specied when and why and for what aspect of
the information management within and between storage components CE's controlling
role in needed. This means that executive functions of CE with respect to control of
slave storage systems is unspecied and thus controlling for executive functions is not
guided by the standard model. Later adoption of the concept the Supervisory Attentional
System (SAS) (Norman and Shallice, 1986) by Baddeley (Baddeley, 1996) for explaining
functioning of the CE not only did not add any detail to this picture but also shifted the
argument towards the concept of a central executive in executive control literature with
the dierent emphasis in control of behavior in a broader context. This latter context
31
has been traditionally associated to the functioning of prefrontal cortex and theoretically
had been proposed to account for single mode of behavior and limitations in attending
to or switching between several tasks.
An example of an alternative `function' which would be used towards giving a mech-
anistic account for management of information in the working memory is `the switching
function'. Yantis and his colleague have entertained this idea that `switching' in a broad
context and in the meaning of the ability to `recongure' resources for matching required
condition is a general and fundamental capability of the brain that can function more or
less with a similar functional principle across dierent domain. Such a switching mecha-
nism can be used to describe function of working memory in term of conditional access
to the working memory content. Yantis and his colleagues have studied common neu-
ral substrates of this switching mechanism in dierent contexts and by comparing neural
substrates of switching spatial attention in within-subject studies fMRI studies (Chiu and
Yantis, 2009; Greenberg et al., 2010; Shomstein and Yantis, 2006; Tamber-Rosenau et al.,
2011). Their ndings point to the superior parietal lobule (SPL), the superior region of
the Posterior Parietal Cortex (BA7). Of particular interest is the overlap between neural
substrates of switching attention between working memory representations of the sym-
bolic type and internal domain and switching spatial attention in the external domain at
SPL (Tamber-Rosenau et al., 2011).
Koenig et al. showed that patients who have sustained damage to their superior
parietal lobule (SPL) generally lose their capacity for mental operations that need rear-
rangement of information and thus they concluded that SPL is critical for manipulation
of information in working memory (Koenigs et al., 2009). Their proposal is consistent
with what Yantis and his colleagues have proposed.
Interestingly, SPL is a part of the association cortex in the posterior parietal cortex
(PPC) and sits at the junction of several sensory processing regions, with projections to
motor area of the brain. In fact, a host of studies have shown involvement of this region
in a number of visual-spatial tasks including in saccadic eye movements (Quintana and
32
Fuster, 1993), visuospatial attention (Grin and Nobre, 2003; Kanwisher and Wojciulik,
2000), visuospatial short-term memory (D'Esposito et al., 1998; Olesen et al., 2004; Smith
and Jonides, 1997) and visuomotor functions (Ferraina et al., 2009).
Implication of the same region during executive working memory tasks has captured
the attention of other researchers too. Osaka et al in the study of group dierences in
working memory tasks noticed that tasks that are known to be more demanding in terms
of executive resources engage the SPL to a greater extent. Moreover, the noticed that
activation of SPL can predict participants' performance during executive demanding task.
As the result the suggested a role for SPL in focusing executive attention (Osaka et al.,
2007b). Implication of SPL along other regions in PPC has been mentioned in a host of
other neuroimaging studies (D'Esposito et al., 1999, 1995; Henson et al., 2000; Olson and
Berryhill, 2009; Osaka et al., 2007a; Smith and Jonides, 1997; Zago and Tzourio-Mazoyer,
2002). Yet, because of the emphasis on the role of preforntal regions in the literature of
executive control and because of in
uences
In the context of studying neural correlates of intellectual tasks SPL is frequently
referred to as an active region. Involvement of SPL has been reported in deductive
reasoning (Fangmeier et al., 2006; Knau et al., 2003) , mental calculation with abacus
(Hanakawa et al., 2003), spatial imagery and deductive reasoning (Hanakawa et al., 2003),
in numerical comparison (Pesenti et al., 2000) mental arithmetic (Knops et al., 2009).
In search for a shared underlying neural mechanism in high-level and low-level cogni-
tive functions some researchers have compared activation signals during both sensorimotor
and intellectual tasks in within-subject neuroimaging studies. SPL has been the region
of interest in some of these studies. In one of these studies Knops et al. showed that
BOLD signals induced by eye movements in SPL are similar to that of mental arithmetic
operations (Knops et al., 2009). They showed this by using a classier trained for dis-
tinguishing the source of BOLD signals as the rightward or the leftward saccades in an
eye movement experiment to reliably classify the mental operations as the addition or
the subtraction operation based on their induced BOLD signal in a dierent experiment.
33
Therefore they argued that brain mechanisms involved in eye movements are recruited
for mental arithmetic. They propose this eect is mediated by the role of shared spatial
encoding mechanism with a role in representation of numbers and in the meantime eye
movements. Their account is evolutionary plausible and yet restricted to representation
of numbers only while the same region is shown to be engaged in working memory tasks
with non-numeric content.
2.1.3 Behaviour Evidences: Interference of executive tasks and visuospatial
cognition
In search for possible contributions of the CE in functioning of the visual system some
researchers have studies the impact of performing executive tasks on the performance of
visual-spatial tasks. Studied visual-spatial functions range from simple recognition tasks
to high-level functions of ocular system such as visual search. The eect of concurrent
performance of an executive task on the performance of perceptual task is often compared
to the impact of concurrent maintenance of executive task material on the performance
of same perceptual tasks.
A common nding of all these studies has been that compared to simple maintaining
of working memory material, using the same material in executive paradigms signicantly
impairs the function of visual system. These observations have started speculations about
possible roles of executive attention resources in dierent aspects of visual-spatial pro-
cessing to an extent which some times is contradictory to what Baddeley as assumed
about roles of the CE.
A particular case is related to the impact of executive tasks on the capacity of mainte-
nance of spatial information. In one the earliest studies of this kind, Phillips used mental
arithmetic as an executive task along with spatial span task and reported that the stud-
ied executive tasks impairs the capacity of visuospatial working memory measured by
the spatial span of subjects. He took this nding as the evidence for engagement of
executive resources in maintenance spatial information (Phillips, 1983). He argued that
34
maintenance of spatial information would rely on a mental imagery process which in turn
relies on executive resources of the CE. This interpretation would con
ict with Baddeley's
view of independence of simple maintenance of visuospatial information in visuospatial
sketchpad from the CE.
Logie, Baddely and Zucco (Logie et al., 1990) in a dierent study compared the impact
of executive tasks on spatial span with that of a task that heavily relied on mental imagery
and noticed that the mental imagery tasks impairs the spatial span to a larger degree and
accordingly refuted Phillips' argument. In the meantime they acknowledged the impact
of executive task on the spatial span and attributed that to the load on CE.
More recently, some researchers started examining those visual tasks that are of inter-
est to perception community. Han and Kim (Han and Kim, 2004) studied the in
uence
of backward counting and mental sorting task on the eciency of the visual search and
reported the signicant impact of engaging in executive tasks. In a dierent study He and
McCaley through an additive factors analysis demonstrated that this eect is not due to
the impact of executive load on the target identication aspect of the visual search (He
and McCarley, 2010). Peterson, Beck and Wong in a study of visual search concurrent
with a dual counting task noticed that attending to the counting task although aects
the eciency of the search, it does not aect the probability of revisiting targets after
dierent saccade lags. In particular, they noticed that similar to the case of simple visual
search, the chance of of revisiting an already inspected target the highest after one sac-
cade lag. They argued that performing the executive task did not aect the visual-spatial
working memory of subjects because returning to an already visited target relies on the
spatial short-term memory of visited target which seems to be unaected.
Controlling for engagement of other components of the standard model of WM demon-
strates that only loading spatial component of visual-spatial working memory can neg-
atively aect the eciency of the visual search. Han and Kim compared the eciency
of a single visual search task with eciency of a similar visual search when subjects
maintained a list of letters and digits and did not observe any signicant dierence.
35
Woodman and Luck engaged the visual working memory concurrent with a visual
search tasks and noticed that the load on visual working memory does not aect the
eciency of the visual search (Woodman et al., 2001). However, in a dierent study
when they included a very simple spatial element to visuospatial working memory task
they observed a signicant impact on the eciency of the visual search (Woodman and
LUCK, 2004).
These evidences suggest that visual search is aected by executive tasks more or less
in the same way that spatial component of visual-spatial.
Other behavioral studies also have shown that engaging in executive tasks signicantly
aects those functions of visual system which is associated to deployment of visual-spatial
attention. For instance it has been shown that active monitoring of the content of working
memory will induce larger attentional blink with larger working memory set size (Aky urek
et al., 2007). In comparison with a similar study which needed only passive maintenance
of items in working memory (Aky urek and Hommel, 2006), it became clear that the eect
of set size in this example is related to active operation (monitoring of items) on working
memory content.
In a dierent study Fougnie and Marios showed that engaging in active manipula-
tion of working memory content in a mental sorting task induces inattentional blindness
measured by the decrease in the likelihood of detecting an unexpected visual stimulus
(Fougnie and Marois, 2007b).
2.2 The Spatial Registry Hypothesis (SRH)
In this section we will lay out a hypothesis for explaining the intimate relationship be-
tween visual-spatial processing and performance of executive tasks. The common feature
of all executive tasks surveyed in the above review is their need for dynamic memory
manipulation which often goes beyond encoding items in a sequential order for a later
recall. We characterize this common feature as the need for selective or random access to
36
the content of the working memory. From a practical point of view random access to the
content of working memory adds
exibility to management and manipulation of work-
ing memory. From a theoretical point of view random access to the content of working
memory relies on a mechanism for storing working memory in an address space which is
accessible to a parallel mode of processing.
Here we propose a role for visual-spatial system in providing access to representa-
tions of task relevant items in a non-sequential or rather a random way. We assume
that due to limitations of phonological-verbal short-term memory system in supporting
random access, utilization of visual-spatial system is necessary for eective management
of working memory items during executive working memory tasks. Thus, we argue that
symbolic representations of task items in visual-spatial short-term memory system will
help utilize the power of this system in random access to its content. The capacity of
selectively attending to representations of objects in the visual-spatial system is a crucial
utility of this system and fundamental for its function in guiding object-oriented actions
or behavior.
This also suggests that the capacity of storing information is not the only factor that
determines which storage modality will be used for encoding a symbolic representation of
task relevant items. It is also of practical importance to provide an ecient way for access
to stored information. Eciency of encoding and accessing information maintained in a
modal system is determined by underlying mechanisms which support maintenance of
information in that particular system. While underlying mechanisms in some systems
allow a natural way for sequential encoding and access to information some others have
advantage to selective and non-sequential access. This is the requirement of the task that
determines which of dierent systems would be a better support or utility system for
management of information.
A key assumption is that visual-spatial system provides a
exible access to symbolic
representation of task relevant items while this function can not be aorded by the phono-
logical loop to the same extent or degree of
exibility. In contrast, when the task requires
37
a sequential encoding of information it seems that the phonological loop can be used as
a reliable source for maintenance of information. In Chapter 6 we discuss general char-
acteristics of dierent sensorimotor systems in supporting random and serial access to
the working memory. In Section 6.7 we will discuss implausibility of using a pure serial
access strategies in handling some executive tasks. These arguments justify the crucial
role of systems with the capacity of random access to their content which in turn explains
dependency of executive tasks on resources associated to those systems.
In the rest of this section we explain how characteristics of visual-spatial systems in
maintaining limited amount of information about identity and location of objects over a
short period of time can be utilized for random access to symbolic representation of items.
Thus we assume that involvement of visual-spatial system is through associating some
symbolic representations which may transiently appear in the working memory system
associated to visual-spatial system. We assume that indeed random access is piggybacking
on the very same system dedicated to supporting object-oriented actions via maintaining
a short-term representation of items and their location. We will refer to this hypothesis
as `the Spatial Registry Hypothesis' or SRH.
We assume that the capacity of the visual-spatial working memory system in parallel
access to items through shift of attention to spatial locations associated to the object of
interest is the key mechanism for supporting random access to the content of the working
memory during intellectual tasks. Utilization of visual-spatial working memory is made
possible by symbolic representation of task relevant items in the visual system. This
system serves performance of intellectual tasks through a seemingly an address-based
random-accessible working memory system.
The crucial characteristics of parallel access to object representations through an ad-
dress space is not limited to visual-spatial system. There are other sensorimotor systems
which can provide a similar capacity. Examples of such systems can be occulomotor sys-
tem, or a kinesthetic system that helps proper conguration of body parts in space using
proprioception and muscle movements (think of a profoundly blind individual's ability for
38
performing tasks in space without any visual reference). In chapter 6.4 a generic formal-
ism will be presented by which a description of functioning of all sensorimotor systems
that can support random access is explained.
The core functional mechanism which provides a basis for describing functioning of
visual-spatial working memory system in random access is the capacity of registering in-
ternal object representations with internal representation of spatial locations.This registry
mechanism provides spatial addressing for random access to items of working memory. A
spatial selective attention with a handle for top-down selection facilitates moving in this
address space. The enables as the selection of symbolically represented items through the
spatial addresses.
The support of a mechanism for using the address space in a programmatic way
would make make mental operations independent of constant monitoring of the content
of working memory. This in turn would allow learning to solve problems in the symbolic
mode based on schemas that rely on addresses. We assume that the Spatial Registry
system is supported by such programming system which allows a sequential shift between
addresses in an algorithmic way. An operational schema (OS) denes the sequence of
shifts between registry locations.
For example, imagine the case of a concurrent mental head-counting of adults and
children in a party. As your gaze shifts to a person in the living room, rst your visual
system becomes engaged in identifying whether the person at focus is an adult or a child.
In the next step, one of two running counts that matches the identied category should
be increased by one. The challenge is keeping track of two numbers and associating them
to categories. A spatial registry strategy is associating the existing count of adults n
a
to
locationl
a
(e.g., left side in visual eld ) and the existing count of children n
c
to location
l
c
(e.g., right side of visual eld). Identifying the next child will trigger a shift of spatial
attention to l
c
, to fetch the current count of children. Once the increment operation is
applied on the current count the result will replace (by rst deletion and then insertion)
the old count.
39
Note how attention shifts might be used both for perception of the external world and
for selection of WM items, which, under the SRS hypothesis, might give rise to con
icts in
some situations, which in turn provides ways to test the hypothesis (see Chapter 8 where
we report the eect of congruency of shift of spatial attention or target detection and
shift of selective attention in internal domain during triple-counting of visual targets).
The operational schema can be conceptualized as a list of mappings of the current
state onto the next action. Here is a formal representation of an alternative OS for our
head-counting scenario.
OS
1
:fchild)shift to l
c
; adult)shift to l
a
g
OS
2
:fat l
c
)fetch n
c
; at l
a
)fetch n
a
g
OS
3
:f at l
c
& n
c
is retrieved)n
c
!n
c
+ 1
; at l
a
& n
a
is retrieved)n
a
!n
a
+ 1g
OS
4
:fshift the gaze to next person & identify the categoryg
Each of these schemas may include other sub-schemas. For example n
c
! n
c
+ 1 in
OS
3
may include a sequence of operations over internal representation such as deletion
and insertion (binding to space).
We need to add that the spatial registry system is not a unitary system and several
SRS systems might collaborate in running the executive machinery of working memory.
However, what all SRS instances have in common is, rst, their build-in internal space
representation, second, a mechanism to shift the attention to those locations, third, a
binding mechanism which can associate locations with symbolic representations, and,
fourth, a mechanism for deploying the sequence of operations in a programmatic way.
Finally, SRH with its emphasis on the importance of random access to working mem-
ory content during executive tasks and also the role of visual-spatial system in supporting
such function is used as a guideline for for devising experimental paradigms of the studies
reported Part II. The goal of reported studies is isolating involvement of visual-spatial
system in executive aspect of working memory tasks. This executive aspect is related to
40
the role of visual-spatial systems in management of information through facilitating the
random access to working memory.
Our model of Spatial Registry System gives an account for observed eects., we pro-
pose the idea of a location registry system (LRS), in which symbolic items in the working
memory task may be bound to locations that are subject to voluntary access through
shifts of attention. A binding mechanism is utilized to associate a location address to
a symbolic item of the intellectual tasks. These locational addresses mimic a randomly-
accessible memory, in contrast with a serially-accessible memory supported by a speech
perception-vocalization subsystem (also referred to as phonological loop in the WM lit-
erature). An operational schema (OS) denes how the attention shifts between these
registry locations for the management of the memory for an algorithmic operation on
symbolic content of the working memory.
As Ballard and Hayhoe have discussed, the visual system provides such a binding
mechanism through its deictic function for goal-oriented manual actions in natural tasks
(Ballard et al., 1997). We believe that the visual system is not unique for manipulation
of working memory content through a binding mechanism and through schema-driven
shifts of attention. Other perception-action subsystems, including proprioception-muscle
movement, may provide a similar function.
41
Part II
Exploring Involvement of Visual-Spatial Resources in
Executive Tasks
42
Chapter 3
Tracing Operational Features of Executive Tasks in Eye
Movements
3.1 Introduction
Eye movements, when serve vision, help brining the most relevant parts of the visual
scene into the focus of foveal vision for scrutiny of a higher resolution visual processing.
Thus, traditionally, eye movements have been treated as a proxy for visual-spatial atten-
tion and are widely used to probed dierent aspects of attention mechanisms in primates.
However, some times cognitive processes with no immediate visual features hijack oculo-
motor system and take the control of eyes. Here is the idea of a simple experimentation;
verbally read these ve digits to a friend for a possible later recall : 7-1-9-8-4, then af-
ter a few seconds ask your friend to sort the digits into order and observe her/his eye
movements during the sorting process. See Figure 3.2 for sample of eye movements in a
slightly dierent set up during sorting of ve random digits in memory. In this section we
will compare eye movements during sorting versus retaining and analyse the dierence to
show the signicant impact of sorting items form memory compared to just maintaining
them.
Now the question is that why some non-visual processes can hijack eye movements so
easily? What these seemingly irregular eye movements can tell us about the architecture
of our cognition? Is involvement of eye movements in those tasks related to the function
43
of eye movement in shifting attention or something else mediates or modulates those eye
movements? Are there any information in eye movements that can tell us about details of
the background or the non-visual parallel processes? These questions have intrigued some
researchers to look into movements of eyes during non-visual tasks even before Jr. and
Darrow (1962) systematic study of eye movements in vision by Yarbus Yarbus (1967).
It is known that eye movements in visual tasks is not a single faceted phenomenon
and both bottom-up (stimulus driven) and/or top-down (goal-driven) factors may play a
role. Similarly one may expect that non-visual tasks in
uence eye movements dierently.
At least several accounts in dierent contexts have been proposed for non-visual eye
movements. In the context of random number generation it is recently shown that location
of gaze before emitting a number can predict the size of the number Loetscher et al.
(2010). Authors of this report suggest a role for number line during generation of random
numbers. They argue that because of the association between numbers and their spatial
representation selection of random numbers will cause shift in attention which in turn
might manifest itself in movements of eyes. This hypothesis does not assume anything
about underlying mechanism for generating seemingly random numbers and implicitly
explains task correlated eye movements during the process of random number generations
as an epiphenomenon of representation of numbers in space.
Another account which was relatively popular for a period of time was Cerebral Lat-
eral Eye Movement or CLEM hypothesis Gur (1975) suggests engagement of the left
hemisphere as the result of verbal thinking may induce initial rightward non-visual gaze
shifts while spatial thinking and engagement of right hemisphere may result to initial left-
ward gaze shifts. Other researchers who studied eye movement rate (EMR) during mental
tasks questioned this hypothesis and suggested that the memory demand of the task de-
termines EMR so that higher memory demand would result to higher EMR Ehrlichman
and Micic (2012); Ehrlichman and Weinberger (1978); Ehrlichman et al. (2007); Hiscock
and Bergstrom (1981); Lorens and Darrow (1962).
44
Yet, all these accounts are agnostic to the function of eye movements in overt shift
of visual attention. None of these accounts explicitly refer to sensorimotoric features of
ocular system in explaining involvement of eye movements in seemingly non-visual and
non-spatial task.
The proposed Spatial Registry Hypothesis (SRH) in the previous chapter lays out
a framework for utilization visual-spatial system for management of symbolically repre-
sented information during tasks that require fast and
exible access to the content of
working memory. This assumption might explain the eye movements during mental task
as correlates of shift in spatial attention needed for management of symbolic representa-
tions.
We argued that this is the result of a strategic use of visual-spatial system for
exible
variable binding and random and non-sequential access to task relevant items which draws
on the visual-spatial short-term memory system which supports object-centered actions.
It has been previously argued in the literature that gaze can serve a deictic function for
setting a reference for action programs Ballard et al. (1997) which are also amenable to
short-term maintenance for access to action programs even when targets of action are
temporarily out of sight Ballard et al. (2011). Because of the ubiquity of the ocular
system in supporting this spatial registry function in object-centered actions it is likely it
is being reused in a symbolic role and for management of working memory in intellectual
tasks.
The purpose of these experiments is exploring whether the ocular system is serving
symbolic working memory management. We will achieve this goal by inspecting eye
movements for traces of the intellectual task. In particular we use specications of SRH
in utilization of ocular system for memory management. Here are hypotheses about how
ocular system might become engaged in symbolic memory management in intellectual
tasks. We will use these hypotheses to devise parameters of our experiments.
45
Hypothesis 1: Maintaining a list of items in memory for a short-term recall is not
likely to draw on ocular resources and thus eye movement behavior during retaining
period of a recall tasks can be used as the baseline.
Hypothesis 2: Eye movement activities during mental tasks which require manipula-
tion of symbolic information is the result of recruiting ocular resources and are induced by
shifts of attention needed for memory manipulation. Working memory task relevant shifts
of attention induces eye movements which predominantly between registry locations.
Hypothesis 3: Registry locations may be manipulated by visual presentation of task
relevant items once the opportunity of visual presentation of those symbols is present.
3.2 Experiments
The main goal in this study is inspecting non-visual eye movement of human subjects for
traces of working memory management predicted by Spatial Registry Hypothesis. This
not only would allow us explain increased EMR in mental tasks but also would signify
eye movements as a source of information for inspecting underlying cognitive processes
through mental operations needed for manipulation of information in the working mem-
ory. To do so we chose reordering of a list of numbers as the mental task that would rely
on random access to working memory content and compared it we the eect of retaining
a similar list for later recall which according to Noori and Itti is preferably handled by
phonological resources that support serial access to working memory. We analyze abrupt
non-visual gaze shifts as a measure of shift in spatial attention and thus our study is fo-
cused on investigating the impact of dierent features of task requirements and location
registry on abrupt non-visual gaze shifts.
Since we use numeral items in our tasks we need to control for potential impacts of
number processing on eye movements. The biasing impact of number processing which
is known as SNARC (Spatial-Numerical Association of Response Codes) eect Dehaene
et al. (1993) has been reported in the case of speed of motor responses in both manual
46
responses Wood et al. (2008) and eye movement responses Fischer et al. (2004) during in
a range of tasks that involve processing numbers. Recently some researchers have argued
for potential impact of number processing on eye movement patterns during mental tasks
that require number processing such as Random Number Generation (RNG) Loetscher
et al. (2010). SNARC eect is often explained by the role of a mental number line in
representation of numerals for smaller numbers represented at left side and larger numbers
represented at right side of the number line.
Luckily both SNARC and Spatial Registry hypotheses provide opportunity for sepa-
rating their eects. While SNARC eect is believed to be automatic and independent of
presentation features SRH eects depend on the schema for spatial binding of items. So
in our rst experiment we try to manipulate location registry of items through dierent
initial visual presentation of items. We presented a list of items either along the horizontal
or the vertical orientation hoping for in
uencing the spatial binding of working memory
of items for the mental operation and when items were removed from the presentation
screen.
In our next two experiments we establish that even a closer relationship gaze shifts
during mental sorting and an internal representation used for sorting operation. In our
third experiment we manipulate the stimuli for mental sorting tasks by controlling relative
order of list items. We see that
While our rst two experiments demonstrate that eye movement behaviour during the
sorting task - and not during retaining for a later recall- is in
uenced by a visuospatial
memory component, in our third experiment we investigate how this spatial memory
of sorting items is associated to the operational aspect of sorting rather than a simple
priming to the location of items.
47
+
+
1
3
3
3
2 4
5
6
7
8
9
0
0
0
9
6
5
5
7
7
8 8
+
7 2 1 4 5
1 2 3
4 5 6
7 8 9
0 ok <-
+
1 2 3
4 5 6
7 8 9
0 ok <-
2.0 sec
100 ms.
3.5 s. Sorting path
Eye tracking
Recall path
Time
Recall
Manual Response
Manual
Response
Task
Cue
Figure 3.1: Schematic view of the paradigm for the mental sorting task.
3.2.1 Method
3.2.1.1 Apparatus
Stimuli were displayed on a 46-inch LCD monitor (Sony Bravia XBR-III, 1,016 571.5
mm), 97.8cm in front of participants (corresponding eld of view is 54:7

32:65

). A
xed chin rest was used to position the eyes in front of the screen. Eye position was
tracked by an ISCAN RK-464 (ISCAN) in pupil-CR mode (240 Hz) to right eye.
3.2.1.2 Stimulus and General Procedure
Stimuli for all WM tasks (mental sorting and delayed recall) consisted of ve non-
repeating pseudo-random decimal digits. Two main paradigms were used for visual
presentation of list items: simultaneous linear presentation and separate random presen-
tation. In simultaneous linear presentation all ve digits appeared together in a horizontal
or a vertical line-up and remained on the screen for 2 seconds (see Figure 3.1). Items
48
visually spanned 3

of angle view around the center of screen either in a horizontal or a
vertical orientation. For the separate random presentation method, list items were pre-
sented separately each one for 700 milliseconds in random locations within an imaginary
central box spanned 15

degrees of angle of view around the center of screen.
To avoid engaging in the sorting task during the visual presentation, the sorting task
was post-cued after a 3.5 second delay (audio cue). Alternatively, a recall of items could
be cued (dierent audio tone), so that when items were presented subjects did not know
whether they should be recalled in the same order or sorted in an ascending order. One
third of trials in each sorting block were randomly selected to be cued for recalling the
list in the presentation order. Subjects were instructed that their performance in these
recall trials is critical for inclusion of their data in the data analysis. We reasoned that
maintaining ve digits in their original order along with their sorted order exceeds the
WM capacity and hence a high performance in recalling items in their original order
suggests that subjects refrained from sorting until the cue was presented.
All visual presentations of items were followed by a visual mask, followed by a blank
screen with a central cross. During sorting blocks, eye movements were recorded after the
audio cue for starting the mental sorting. There was limitation in duration of the sorting
trials and subjects would signal their readiness for reporting the sorted list by clicking
a computer mouse button. The mouse click would end the recording of eye movement.
Subjects were instructed to repeat the sorted list once and before signalling the report.
At the end of each trial a virtual keypad appeared on the screen through which digits
were selected using a visual pointer which was controlled by a computer mouse. There
was no time limitation for reporting the digits.
During the delayed recall trials eye movements were recorded after the 3.5 seconds
delay period and before displaying the response virtual keypad on the screen. The time
duration between cueing and the displaying the virtual keypad was taken randomly from a
normal distribution with the mean and the variance of the sorting duration for a particular
subject.
49
During the recording period subjects could move their eyes freely; however, they were
advised to keep their eyes open, and, due to limitations of our eye tracking method, to
keep their gaze location within the presentation screen (54:7

32:65

angle of view).
1˚
Figure 3.2: A sample of eye traces (white curved lines) during a the mental sorting task in front
of a blank scree. Black arrows mark gaze shifts. Discontinued downward eye traces are artefacts
of blinking.
3.2.1.3 Analyzing eye movements
A fteen-point display calibration was used to compute the ane transform from the
eye-tracker coordinates to the presentation screen coordinates in the least-square sense.
Small non-linear residual errors in the transformation were corrected by a thin-plate-
spline warping algorithm Bookstein (1989). Eye positions in the screen coordinates were
used to mark local gaze points. A local gaze point is dened by locality of distribution
of eye position in a time window, quantied by the ratio of eigenvalues of covariance
matrix of x and y positions of eye location in that certain time window. A ratio close to
50
1 means that eye locations are distributed more locally while a very small or a very large
ratio means that eye locations are distributed more elongated. Each two consecutive gaze
points dene an abrupt gaze shift which corresponds to a vector in presentation screen.
Figure 3.2 shows a sample of eye traces on the presentation screen during a trial of mental
sorting with gaze shifts superimposed on top of eye traces.
We use distributions of abrupt gaze shifts as the indicator of the eye movement be-
haviour. Abrupt gaze shifts for all trials of a particular condition are pooled together and
normalized for each subject separately.
3.2.2 Experiment 1
In this experiment we tried to manipulate spatial binding features of the putative location
registry system supporting a mental task that requires working memory manipulation.
We reasoned that shared mechanisms for visual-spatial working memory of visual targets
and underlying mechanisms of SRH may help induce the schema for binding working
memory items. So we presented task items visually and before the start of the task was
cued. We manipulated the orientation in which items of working memory appeared.
To prevent subject from starting the sorting process during visual presentation of
items and in the meantime force them to attend to the linear order of items the tasks
was sort-or-recall and where
The goal of this experiment is rst to establish that manipulation of WM in the form
of reordering the memory content can aect the eye movement behaviour in a dierent
way than the retaining similar content for a later recall. We compare the eye movement
behaviour during the mental sorting with that of a retaining period in a delayed recall
task within the same time bounds. To test any possible impact of the sorting task on
the eye movements, we also control for the initial visual presentation by presenting the
sorting list either in the horizontal or the vertical orientation.
51
3.2.2.1 Subjects
Eight female and two male university undergraduate students with normal or corrected
to normal vision, participated for course credit. Participants' ages ranged from 18 to
22 (M=20 years, SD=1.05). Nine subjects reported to be right handed and one subject
reported to be left handed.
3.2.2.2 Procedure
Items of WM task were presented simultaneously and in a linear order. In each block,
items were presented in either the horizontal orientation or the vertical orientation. Each
subject performed four blocks of WM, each of which with a xed WM task and a xed
presentation method. The rst two blocks consisted of the post-cued mental sorting,
each one with a dierent presentation orientation. The last two blocks consisted of the
delayed recall task, each one with a dierent presentation orientation. Half of the subjects
were selected randomly to start with the horizontal presentation. Each block yielded 15
recorded trials for each subject.
3.2.2.3 Results
Figure 3.3 summarizes the result of the rst experiment in terms of average proportion
of horizontal abrupt gaze shifts for all four combinations of tasks and visual presentation
methods. All gaze shifts with a direction of45

below or above the horizontal line
were considered as the horizontal gaze shift. To examine the impact of both task and
presentation factors on the proportion of horizontal gaze shifts, the data was submitted to
a repeated measure two-way ANOVA with the type of mental task and the initial visual
presentation as two main factors each of which with two levels.
A signicant main eect of the presentation orientation was observed atp< 0:05 level
[F (1; 9) = 5:5385;p = 0:043] however the main eect of the task was only marginally
signicant [F (1; 9) = 3:4167;p = 0:0976]. No signicant interaction between factors
observed [F (1; 9) = 0:6667;p = 0:4353].
52
Horizontal
0.6
0.56
0.52
Sorting
Delayed recall
Figure 3.3: The impact of initial presentation orientation of WM stimulus on the share of
horizontal gaze shifts during the delayed recall and the mental sorting task.
Further analysis showed a simple main eect of the presentation method for the
sorting task. A one-way within-subjects ANOVA (correlated samples) with two levels
associated to the horizontal and the vertical presentation methods indicated that the
mean proportion of horizontal abrupt gaze shifts for the vertical presentation (M =
0:511;SE = 0:013) was signicantly less than the average for horizontal presentation
method (M = 0:565;SE = 0:012), [F (1; 9) = 5:93;p = 0:037].
However a similar one-way ANOVA test for the delayed recalled task within similar
time bounds did not show a signicant eect of the visual presentation method on the
proportion of horizontal gaze shifts [F (1; 9) = 0:68;p = 0:43].
53
This result suggests that in terms of the direction of gaze shifts during the sorting task,
the orientation of initial simultaneous visual presentation of the sorting list signicantly
impacted the eye movement behaviour, meanwhile such an impact did not reach to a
signicant level for the delayed recall task within similar time bounds.
3.2.2.4 Discussion
The analysis of gaze shifts in this experiment showed that the initial visual presentation
of the WM items in
uenced the eye movement behaviour during the mental sorting task.
Importantly, such an eect was not signicant for maintaining the same content within
similar time bounds for a the delayed recall. This suggests that engaging in mental sorting
impacts eye movements, and this impact is mediated by a visuospatial working memory
element which is in
uenced by initial stimulus presentation.
There are several possibilities for how spatial short-term memory of list items may
play a role during sorting and not during maintaining for delayed recall. One possibility
is related to the role of space in encoding numbers. It has been proposed that numbers
are some recent cultural inventions in our history and hence it is likely that space plays
a role in their representations Dehaene and Cohen (2007). This hypothesis has received
support from experimental data indicating the impact of number processing on spatial
response codes (known as SNARC eect) Hubbard et al. (2005); Wood et al. (2008). One
common belief is that a linear (horizontal) representation with small numbers associated
to the left and large numbers to the right side is utilized. Based on this assumption,
Knop et al. Knops et al. (2009) tried to distinguished mental addition from mental
subtraction based on the similarity of their induced cortical activation patterns to that of
separately-measured, visually-guided gaze shifts to the right and the left. They reasoned
that since a mental addition results to a larger number and a mental subtraction results
to a smaller number, a mental addition should elicit activation patterns similar to that
of shifts of attention or gaze to the right, while mental subtraction should look similar to
a shift in spatial attention towards the left. Knop et al. realized that a classier trained
54
to distinguish activation patterns in SPL during execution of either rightward or leftward
eye movements could also distinguish activation patterns during mental addition versus
subtraction. They concluded that brain regions involved in eye movements are recruited
for mental arithmetic. This involvement can potentially explain an impact of number
processing on eye movement behaviour; however, a hallmark feature of the impact of
number processing on response codes is that they are generally independent from the
presentation method (dierent symbolic presentation or the location of the presentation),
and even from the task in hand Wood et al. (2008). Thus, an eect mediated by spatial
encoding of numbers would be insensitive to the direction of the initial presentation, or
even to whether the task is sorting or maintaining. Thus, our results cannot be explained
solely by an interaction between numbers and space.
Another possibility is that the sorting task induces gaze shifts to actual presentation
locations of items. Such gaze shifts might be related to retrieving items from the visuospa-
tial short-term memory, or might just be irrelevant shifts to locations previously occupied
by those items Richardson and Spivey (2000).Yet Richardson and Spivey Richardson and
Spivey (2000) showed that even when the spatial location is not relevant to the task at
hand, there is a tendency to look at the locations previously occupied by the items.
Another possibility is that gaze shifts during the sorting task are induced by an
internal mechanism, which, perhaps accidentally, happens to be in
uenced or biased by
the initial presentation method in this experiment. A use of space for transforming the
order of items in the reverse recall task has been discussed in the literature Larrabee and
Kane (1986). In the case of reverse recall one can imagine any internal linear arrangement
of working memory items can helps transform the order of items as long as items are
arranged in a specic direction which re
ects the original order and then recalled in the
reverse direction. Utilization of a specic orientation for reversing the order is up to a
strategy for using space and yet can be in
uenced by priming a specic orientation.
55
Our next experiment is designed to test the viability of an internal transient spatial
working memory account versus accounts for gaze shifts to the location of presentation
of WM items.
3.2.3 Experiment 2
The main goal of this experiment is determining whether the observed biasing of gaze
shifts along the presentation orientation in experiment 1 is related to actual presentation
locations of WM items, or to an internal linear representation in
uenced by the initial
presentation. The latter hypothesis predicts that even when no dominant orientation is
present in the visual presentation phase, gaze shifts would be biased along a direction (e.g.,
horizontal mental line). Hence, we include another paradigm for visual presentation of the
sorting list, in which items are presented separately and in random locations. Presenting
items separately helps encode an order for the sorting list based on the temporal order of
item presentation, rather than an explicit spatial relationship between list items. We thus
present items during of the sorting task within an imaginary box around the center of
screen with a uniform probability. If the observed biasing of gaze shifts during the sorting
task is mediated by a direct sensory short-term memory of item presentations, then one
would expect that the pattern of gaze shift during the random presentation would be
dierent from both the horizontal and the vertical presentations. On the other hand, an
internal spatial working memory representation (e.g., line) would predict a biasing in the
gaze shifts along that representation even when items were presented randomly.
3.2.3.1 Subjects
Fourteen female and two male university undergraduate students with normal or corrected
to normal vision, participated for course credit. Participants' ages ranged from 19 to 22
(M = 20years;SD = 0:89). Fourteen subjects reported to be right handed and two
subject reported to be left handed.
56
3.2.3.2 Procedure
The experiment was administered in three blocks of sorting task each of which with a
dierent visual presentation methods. In two of blocks items presented simultaneously,
either along the horizontal direction or the vertical direction. In the other block items
presented separately and in dierent locations within an imaginary box. The order of
presentation method changed for dierent subjects. Similar to the previous experiment
the sorting task was post-cued. each subject performed 15 trials of the sorting task for
each of the presentation methods.
Proportion of horizontal gaze shifts
0.60
0.55
0.50
0.45
Horizontal
presentation
Vertical
presentation
Random
presentation
Figure 3.4: Proportion of horizontal gaze shifts during a mental sorting task with three dierent
presentation methods.
57
3.2.3.3 Results
Figure 3.4 shows the average proportion of horizontal gaze shifts (up to 45

above
or below the horizontal line) during the mental sorting for all three dierent stimu-
lus presentation methods. To test the signicance of the eect of the visual presenta-
tion method on horizontal gaze shifts during mental sorting, a one-way within-subjects
ANOVA (correlated samples) with three levels for visual presentation methods was con-
ducted. A signicant eect of the presentation method was observed at p < 0:01 level
[F (2; 30) = 8:35;p = 0:00131].
Post hoc comparisons using the Tukey HSD test indicated that the mean proportion
of horizontal gaze shifts for the vertical presentation (M = 0:511;SE = 0:013) was
signicantly less than both the horizontal presentation (M = 0:554;SE = 0:011) and the
random presentation (M = 0:558;SE = 0:0130) at p < 0:01. However, the eect of the
horizontal presentation and the random presentation on the proportion of horizontal gaze
shifts were not signicantly dierent.
This result suggests that in terms of the orientation of gaze shifts during the sorting
task, there is no dierence between initially presenting the sorting list separately and in
random places or presenting it simultaneously and linearly in a horizontal orientation.
However, similar to what was observed in the rst experiment, presenting list items
simultaneously and in the vertical orientation results in more vertical (or less horizontal)
gaze shifts during mental sorting.
To illustrate our ndings in more details, panels of Figure 3.5 show the average dier-
ence between distribution of gaze shifts for dierent presentation methods with a higher
resolution and in all directions. As Figure 3.5.a shows, the average dierence between
gaze shift distributions for the simultaneous horizontal and the separate random presen-
tations falls near zero (marked by bold dashed circle). Figures 3.5 b. and c. demonstrate
the similarity between the distribution of gaze shifts for the horizontal and the random
presentations in contrast with the distribution of gaze shifts in mental sorting with initial
vertical presentation. Both gures show a similar horizontal elongation of distribution
58
orientation
Δprobability
× 10
-4
-8
-4
0
4
8
orientation
Δprobability
× 10
-4
-8
-4
0
4
8
orientation
Δprobability
× 10
-4
-8
-4
0
4
8
Random - Horizontal Horizontal - Vertical Random - Vertical
Figure 3.5: Comparing directional distribution of gaze shifts during a mental sorting task for
dierent pairs of presentation methods. Each graph displays the dierence between normalized
distribution of the direction of gaze shifts for a pair of task. The gray area marks the error bar
associated to error in the mean value.
of gaze shifts for the horizontal and the random presentations relative to the vertical
presentation.
3.2.3.4 Discussion
In terms of the direction of gaze shifts, eye movement behaviour after random presen-
tation is similar to horizontal presentation and dissimilar to vertical presentation. This
result does not favour an account based on a direct involvement of sensory-based visu-
ospatial short-term memory of the sorting list on induced gaze shifts during sorting (e.g,
Hollywood Squares eect Richardson and Spivey (2000), or spatial binding of memory
items for direct access to short-term memory of visual form of numbers presented in the
space); otherwise we should have observed signicant dierences in gaze shifts between
all three presentation modes.
The similarity of gaze shifts between horizontal and random presentations suggests
that even when no particular orientation was present during the visual presentation of
WM items, an internal linear horizontal arrangement has helped the mental sorting. This
might be related to an internal spatial working memory used for bookkeeping and nec-
essary for memory manipulation during the sorting task, something similar to what has
been previously proposed for the case of reverse recall Gerton et al. (2004). However,
59
experiment 2 has not fully ruled out possible irrelevant or epiphenomenal interplays be-
tween the sensory input and the internal spatial working memory, since initial vertical
presentation of WM stimuli signicantly increases gaze shifts along the vertical direction.
This motivates the next experiment, to probe more precisely the extent to which eye
movements are related to procedural aspect of the sorting task.
3.2.4 Experiment 3
The goal of this experiment is to establish a relationship between procedural aspects
of the mental sorting tasks and gaze shifts during the task. In particular, we try to
show that the bookkeeping requirements of the mental sorting task can be traced in the
task-induced gaze shift patterns. This would establish a functional relationship between
the shift in visuospatial attention (indicated by gaze shift proles) and the mechanics of
memory manipulation for the sorting task.
To achieve this goal, we categorize our sorting stimuli based on the relative order of
items in the sorting list. We reasoned that in a spatial bookkeeping mechanism, lists
with the same relative order of items would be bound to space in a similar way. If
this binding is used in an order-sensitive procedure, then subsequent use of space would
be only sensitive to the order of items. For example imagine these two lists of items:
s
1
=h3; 1; 2i ands
2
=h8; 4; 5i. The relative orders of items in these two lists are identical
and can be represented with this canonical list: h2; 0; 1i. Thus, if a spatial bookkeeping
mechanism were used to store and access the WM items during sorting, such mechanism
could be exposed by controlling the relative order of list items, and this in turn would be
re
ected in the gaze shifts. Thus we hypothesized that sorting lists with dierent relative
order of items would result to dierent gaze shift proles.
However, relating sorting lists to gaze shift proles requires some knowledge about
how a potential underlying spatial bookkeeping mechanism would serve the sorting task,
something which does not seem obvious at rst. Nonetheless, for some lists, this relation-
ship would be independent of the details of the sorting algorithm, for example one would
60
expect that items of lists with
ipped relative orders are bound to space in a symmetric
way, and hence they would result to symmetric use of space during the sorting procedure.
For instance, consider these two lists: s
1
=h3; 1; 2i ands
2
=h5; 4; 8i . The relative order
of items in these two lists can be represented with these two canonical lists: c
1
=h2; 0; 1i
andc
2
=h1; 0; 2i, which are symmetric. We reasoned that if gaze shifts during the sorting
task re
ect an internal spatial bookkeeping mechanism, then independent of underlying
mechanics of the sorting procedure, one would expect some spatial symmetry in gaze shift
patterns for these two symmetric lists. In contrast, we reasoned that if eye movements
were just epiphenomenal and unrelated to sorting (e.g., independent scanning of the pre-
vious visual locations of items), symmetric lists would not yield symmetric eye movement
patterns.
To demonstrate this functional relationship, we used two pairs of symmetric canonical
lists for generating sorting lists. We compare the gaze shifts patterns during the sorting
task and we will examine skewness of gaze shift distribution to measure the symmetric
aspect of gaze shift distributions.
3.2.4.1 Subjects
Ten female and zero male USC undergraduate students with normal or corrected to
normal vision, participated for course credit. Participants' ages ranged from 18 to 21
(M = 18:8years;SD = 1:03). All subjects reported to be right handed.
3.2.4.2 Procedure
We chose four categories of 5-digit lists (categories 1,2,3 and 4) denoted byC
1
;C
2
;C
3
and
C
4
, identied by these canonical strings (respectively):h3; 4; 0; 1; 2i;h2; 1; 0; 4; 3i;h4; 1; 2; 3; 0i
andh0; 3; 2; 1; 4i. Note that category 1 is symmetric of category 2, and category 3 is also
symmetric of 4. Exemplars for each category were generated by using dierent digit val-
ues while preserving relative ordering. For instance,h7; 8; 1; 5; 6i andh6; 9; 0; 2; 5i are two
exemplars of C
1
.
61
We randomly generated strings of ve digits belonging to these four categories as the
stimuli for the sorting task and presented them horizontally on the screen similar to the
simultaneous horizontal presentation method in experiments 1 and 2. We administered
this experiment in four blocks each block including 20 sorting trials. 5 exemplars of each
of stimulus types used for sorting trials plus 10 trials of random recall challenge.
After administrating the fourth block, subjects responded whether they discovered
any pattern in the lists, and none realized that sorting lists fell into four limited patterns.
3.2.4.3 Results
For each subject and each of the categories a distribution of gaze shift vectors along the
horizontal line was formed (for all gaze shifts between 45

above or below the horizontal
line). Pearson's rst measure for skewness (3(meanmode)=) was calculated for each
of these distributions. Skewness measures for each pair of categories were submitted to a
linear regression for evaluating a linear correlation (or anti-correlation) between skewness
measures. Table 3.1 summarizes the result of linear regressions for all possible pairs of
sorting stimulus categories. Skewness for two pairs of stimuli showed a signicant anti-
correlation. For symmetric categories 1 and 2 the linear regression resulted tob
1
=0:82,
with a signicant (anti)-correlation, with skewness in C
1
distribution accounting for 79%
of variability of skewness of C
2
distribution, [r
2
= 0:7934;F (1; 8) = 30:7151;p< 0:0005].
A similar signicant anti-correlation between skewness measures was observed for the
other symmetric categories 3 and 4 with skewness in C
3
distribution accounting for 70%
of variability of skewness of C
4
distribution. The linear regression resulted b
1
=0:89
with a signicant (anti)-correlation [r
2
= 0:7043;F (1; 8) = 19:057;p = 0:0024].
The linear regression for skewness measures for the pair of 1 and 3 (not symmetric of
each other) resulted to b = 0:06 with n.s. correlation [r
2
= 0:0042;F (1; 8) = 0:0341;p =
0:8581]. Likewise, a linear regression for categories 2 and 4 resulted to b = 0:56 with a
n.s. correlation [r
2
= 0:208;F (1; 8) = 2:11;p = 0:1844].
62
For the pair of 1 and 4 the linear regression resulted to b =0:38 with r
2
= 0:113
and F (1; 8) = 1:021;p = 0:34. For the pair of 2 and 3 the linear regression resulted to
b =0:16 with r
2
=0:018 and F (1; 8) = 0:149;p = 0:709.
Taken together these results show that sorting stimuli with symmetric list order (stim-
ulus pairs 1-2 and 3-4) result in signicant symmetric skewness (measured by Pearson's
rst measure for skewness) in distribution of gaze shifts. This means that a bias in gaze
shifts towards a specic direction during the mental sorting task of a specic category
(identied by the relative order of sorting items) predicts a bias towards the opposite
direction during the mental sorting of stimuli with a
ipped relative order. The symmet-
ric relationship between skewness of spatial distributions did not hold for non-symmetric
sorting elements.
Our analysis shows that, only for the case of symmetric sorting lists, an acceptable
linear regression exists. However, not every linear regression can establish the functional
relationship that we proposed in our hypothesis. What indeed tests our hypothesis is
whether this linear regression re
ects the spatial symmetry associated to symmetric cat-
egories. An anti-symmetric regressor ( y =x +) dictates such constraint. Hence
statistical metrics associated to symmetric categories need to be explained based on an
anti-symmetric regressor rather than an arbitrary regressor.
One may recalculate the coecient of determination with regard to the anti-symmetric
regressor to explore the power of this regressor in explaining the trend in between-
categories skewness data. Table 3.2 shows
b
R
2
values with regard to the anti-symmetric
regressor. Our analysis shows that an anti-symmetric regressor can notably explain the
trend in the skewness data only for the case of symmetric sorting stimuli. Negative
b
R
2
values for the case of non-symmetric stimuli shows that the anti-symmetric regressor is
even less suitable than a simple normal noise in explaining the trend in the data. Fig-
ure 3.6 demonstrates the least-squares linear regressor and the anti-symmetric regressor
in the same panel for four pairs of sorting categories. The two panels associated to
symmetric categories demonstrate the similarity of the anti-symmetric and least-squares
63
regressor, while the two other panels show the discrepancy between anti-symmetric and
least-squares regressors for non-related categories.
Table 3.1: Linear relationship between skewness of gaze shift distributions during sorting
stimuli of four dierent categories.
Category Pairs b
1
b
0
R
2
F-statistics t Pr(>jtj) signicance
(C
1
;C
2
) -0.82 0.53 0.793 F (1; 8) = 30:71 -5.54 0.0005 ***
(C
1
;C
3
) 0.065 0.77 0.004 F (1; 8) = 0:034 0.18 0.855
(C
1
;C
4
) -0.38 0.64 0.113 F (1; 8) = 1:021 -1.01 0.34
(C
2
;C
3
) -0.16 -0.31 0.018 F (1; 8) = 0:149 -0.39 0.709
(C
2
;C
4
) 0.56 -0.11 0.208 F (1; 8) = 2:11 1.45 0.18
(C
3
;C
3
) -0.89 -0.39 0.704 F (1; 8) = 19:06 -4.37 0.0024 **
Table 3.2: Between categories
e
R
2
values for the anti-symmetric regressor for skewness
of horizontal gaze shift skewness. The boldfaced values indicate the notable explanation
power of the anti-symmetric regressor
C
1
C
2
C
3
C
4
C
1
- 0.711 -1.485 -0.6279
C
2
0.657 - -1.119 -2.837
C
3
-1.176 -0.564 - 0.556
C
4
-0.273 -1.531 0.603 -
3.2.4.4 Discussion
Our rst two experiments showed that gaze shifts during mental sorting carry informa-
tion about the location of items, in a putative space which maps onto the presentation
space when items are linearly ordered in their visual presentation. In this experiment,
we manipulated the order of items to expose in a more explicit manner the correlation
between gaze shifts and location binding of working memory items in this putative space.
In fact our denition of categories of sorting stimuli is based on spatial ordering in the
presentation space. This use of space allows a bookkeeping of the items in the process of
sorting so that one can dene the termination condition of the sorting tasks based on a
clear condition in this putative space: once every item is smaller than the next item on
the right then terminate the sorting. So this spatial representation can fully capture an
64
algorithm for sorting an array of digits by providing a spatial address to each item which
then can be used for basic operations of a sorting algorithm (e.g., comparing two items
for their relative magnitude and whether they are in the right order in the list).
The result of this experiment shows that a symmetry in the location of items in this
space results in a symmetry in a spatial feature of gaze shifts during mental sorting of
items. This establishes a functional use of space for the mental sorting for bookkeeping
of items.
Synthesizing the results from all three experiments allows us to present a more general
discussion on the role of space during not only mental sorting but also as a general
framework for manipulation of items during intellectual tasks with symbols.
3.3 Outlook
We closely inspected eye movements of human subjects for traces of an intellectual sym-
bolic task that features active memory management as an evidence for involvement of
the visual system in intellectual symbolic tasks with no immediate visual or spatial char-
acteristics.
We detected traces of a mental sorting of ve random digits in eye movements and
in the distributions of gaze shifts when no item was visually available. This impact was
manifested in extra gaze shifts along either the horizontal or vertical directions, only for
the sorting task and not for recalling the items in their original presentation order. When
items were initially presented linearly along the horizontal or vertical direction, task-
induced gaze shifts showed a modulation along the orientation of the initial presentation,
suggesting an in
uence of the memory of spatial locations of symbolic items. However,
when items were presented in random locations, gaze shifts during the sorting tasks were
similar to those of initial horizontal presentation, suggesting that this memory of location
is associated to an internal representation which may be in
uences by linear presentation
of items. In our last experiment, we showed that spatial distributions of gaze shifts
65
re
ect the order of items in a putative linear spatial ordering of items in the unsorted
stimuli, so that items with
ipped order of initial stimuli resulted in symmetric skewness
of distribution of gaze shifts. So, one may say that information about a putative spatial
arrangement of mental stimuli for the intellectual symbolic task can be traced in the
statistical aspects of gaze shifts during the mental task.
Our ndings suggest a role for the visual system (particularly, the occulomoter system)
in mental symbolic tasks, which is tied to a visual-spatial binding of task relevant items.
In particular, our experimental ndings are consistent with what are specied by proposed
Spatial Registry Hypothesis (SRT).
66
C
1
<3,4,0,1,2>
+3 -3 0 +3 -3 0
+3 -3 0 +3 -3 0
+3
0
-3
-3
+3
0
+3
0
-3
-3
+3
0
R
2
=0.704
R
2
= 0.79
R
2
=0.004
R
2
=0.21
R
2
= 0.71
~
R
2
= - 1.531
~
R
2
= -1.485
~
R
2
= 0.603
~
Least Square Regressor
Anti-Symmetric Regressor
C
4
<0,3,2,1,4>
C
2
<2,1,0,4,3>
C
3
<4,1,2,3,0>
Skewness Skewness
Skewness
Skewness
Figure 3.6: Pairwise comparison of Pearson's skewness measure for four pairs of stimulus types
for the sorting across all subjects. Top-left and bottom-right panels are related to symmetric
pairs ((C
1
;C
2
)and(C
3
;C
4
)) with notable explaining power of both the least square and the anti-
symmetric regressors , top-right and bottom-left panels are related to non-symmetric sorting
stimuli for which neither the least square nor the anti-symmetric regressor describes the trend
of the data. Two other combinations of pairs associated to non-symmetric lists also resulted
non-signicant correlations.
67
gaze shift amplitude
4˚ 4˚ 2˚ 2˚
-0.01
0.01
Δprobability
(b)
-0.01
0.01
Δprobability
(a)
left right
Figure 3.7: The graph on each of two panels is obtained by averaging over subtraction of nor-
malized distributions of rightward and leftward saccades for two symmetric stimulus types. The
top panel is associated to the pair of 1 and 2 and the bottom panel is associated to the pair of 3
and 4.
68
Chapter 4
Tracing Impacts of Executive Tasks on Visual-Spatial STM
4.1 Introduction
In this chapter we present the result of our attempts in detecting another aspect of
involvement of visual-spatial system in manipulation of information for intellectual tasks
specied by the Spatial Registry Hypothesis. SRH is explicit in assuming utilization of a
type of short-term memory of spatial locations for registering symbolic representations of
task items. These spatial short-term representations later on can be used as an address
for accessing those symbolic representations. Moreover, it is also explicit that the spatial
short-term memory systems which allows and serves object-centered actions are likely to
be used for this purpose.
Our experiments in Chapter 3 provided evidences of implication of the ocular system
during mental tasks that feature dynamic working memory manipulation. This would
suggest that it is likely that visual-spatial short-term memory linked to the ocular system
is likely to be also implicated during mental tasks which feature memory manipulation.
It has been previously shown that engaging in performance of executive tasks might
impair visual-spatial short-term memory. A common feature of tested executive tasks
(e.g. mental arithmetic or random number generation) is the need for memory manipula-
tion. The standard model of the working memory assumes that manipulation of working
memory draws on executive attentional resources and thus engaging in those tasks would
69
divert limited executive attention resource away from other tasks that may require those
resources and thus might aect the performance of either of those concurrently performed
tasks. This is often interpreted as the evidence for engagement of the CE in performance
of the secondary task along the executive task.
Any change in the memory span as the result of engagement of the CE would be in-
terpreted as evidence for a role for executive resources in maintaining spatial information.
The fact that engaging in executive tasks impairs the capacity of maintenance of visual-
spatial information is at odds with what Baddeley is assumed about the role of CE in
management of storage components of the standard model. In fact, Baddeley maintains
that simple maintenance of verbal or spatial information should not draw on executive
resources unless transformation of information between components is necessary Repovs
and Baddeley (2006).
In an early attempt by Phillips Phillips (1983), studied the impact of engaging in
mental arithmetic on visuospatial short-term memory by measuring spatial span { the
maximum size of a matrix of symbols which can be recalled better than a threshold perfor-
mance { in a dual task paradigm. He noticed that engaging in mental arithmetic reduces
the spatial span and thus concluded that maintaining spatial information is facilitated
through an active mental imagery process which is inhibited by the load of the mental
arithmetic.
Logie, Zucco and Baddeley Logie et al. (1990) compared the eect of both a mental
arithmetic and a mental imagery task, on both visual and word spans, and showed that
the the mental imagery task impairs the visual span to a greater extent, while mental
arithmetic impairs the word span to a greater extent. However, they still observed an
impact of mental arithmetic on visuospatial span and stated that "the impairment in
short-term visual memory resulting from secondary arithmetic re
ects a small general
processing load".
70
However, later on, Baddeley and Repovs summarized the results of many dual-task
studies, pinpointing the role of CE in maintaining and manipulating information in com-
ponents of Baddeley's multi-component model of working memory. They concluded that
simple representation and maintenance of information may be independent from the CE
Repovs and Baddeley (2006). This conclusion includes maintaining spatial information
tested in measuring spatial span which is contrary to Baddeley's previous note on the
impact of mental arithmetic on spatial span.
In a more recent experiment, Fisk and Sharp used the error in recalling order of
spatial sequences as the measure of the capacity of the spatial working memory to study
possible roles of executive system in maintaining spatial working memory Fisk and Sharp
(2003). They were motivated by recent evidences of strong correlations between executive
tasks and spatial working memory tasks emerged from factor analysis studies Miyake
et al. (2001). Fisk and Sharp engaged their subjects in a dual task with random letter
generation as the executive task along with their spatial working memory task. They
found that engagement in the random letter generation task impaired performance on
the spatial task especially recall of the early serial positions. However, contrary to their
expectation the degree of impairment was no greater on the longer lists Fisk and Sharp
(2003).
These ndings motivated us to inspect whether the observed impacts might be indeed
a manifestation of direct engagement of visual-spatial short-term memory in the proce-
dure of memory management during the mental tasks known as executive tasks. In our
experiments we use a measure which is potentially able to detect implications of symbolic
working management on the short-term memory of visual-spatial targets specied by our
Spatial Registry Hypothesis. An SRH-based account relates the impact of the secondary
intellectual task to the interference between spatial short-term memory of visual targets
and spatial short-term memory of symbolic representations of the secondary task items.
Thus this interference might be limited to those short-term memories that are represent-
ing roughly same places in the space. As the result, one may expect a spatial selectivity
71
in the impact of the secondary executive task on the visual-spatial STM which will be
limited to those parts of space being used for the spatial registry.
In the present study, we use the sensitivity of subjects in detecting a change in location
of two dots to measure the ability of retaining visual-spatial information. This requires
retaining an amount of spatial information which is below the capacity of normal subjects
Luck and Vogel (1997). In other words, instead of using a xed threshold for performance
in measuring the span of short-term memory, we used a xed amount of information load
below the normal capacity of our subjects, to measure the eect of a secondary intellectual
working memory task. Although this paradigm does not measure the spatial working
memory capacity in its conventional denition yet, it re
ects the general capacity of
maintaining spatial information over a short period of time. Moreover, this measure tests
the spatial short-term memory in a way that is closer to the use of spatial information
in daily perception-action routines. Finally, this paradigm can be easily used to test the
spatial selectivity of any potential impact on the spatial short-term memory.
Using this sensitivity measurement paradigm, we inspected the in
uence of two dier-
ent symbolic working memory tasks on the short-term spatial information retention. In
our rst experiment, we used a dual-counting task in which two running counts need to
be maintained and updated upon presentation of two distinguishable audio signals. We
use the rate of signal presentation as a parametric feature to change the amount of time
that is allocated to this task. This allows us detect any impact onto spatial short-term
memory caused by decaying information as the result of performing the symbolic working
memory (SWM) task.
In our second experiment, we used mental reordering versus retaining of four random
alphabetical characters (presented auditorily) as our symbolic working memory tasks. We
compared their impact onto retaining spatial information along either the horizontal or
vertical orientation. This allowed us to test the spatial selectivity of the impact of mental
reordering of characters compared to maintaining them.
72
N
+
+
Y
<-
Time
500 ms.
500 ms.
1.0 s.
500 ms.
10.0 s.
VSSTM stimulus
presentation
VSSTM probe
presentation
Audio cue for
VSSTM probe
1000 ms.
VSSTM retaining period
& mental task execution
VSSTM
response
Mental task
response
Stimulus presentation
for the mental task
1 2
4 5
7 8
0
<- N
Ok
Figure 4.1: Schematic view of the experimental paradigm.
4.2 Experiment 1
We asked our subjects to perform a mental dual counting task of two audio signals, while
they were also retaining visual-spatial information. The goal was to test whether this
symbolic working memory task could interfere with retaining visual-spatial information
as simple as the spatial locations of two visual targets. We also aimed to test whether
a possible interference is due to competition over scarce executive resources that might
be needed for both the manipulation of working memory content and the retention of
visual-spatial information.
Mental dual counting involves maintaining two running counters, associated with two
signals, in working memory, and, each time a new signal is perceived, incrementing the
associated counter. The rate of updating of the internal counters can be adjusted by
the rate at which audio signals are presented. This allows manipulation of the total
amount of time that putative executive resources may be free and available for other
tasks (e.g., active retaining of spatial short-term memory), which in turn may aect the
73
sensitivity measure. We chose two dierent rates for presenting audio signals for the dual
counting task. In separate blocks, we asked subjects to ignore versus count the signals
while retaining the visual-spatial information.
4.2.1 Method
4.2.1.1 Apparatus
Visual-spatial stimuli were displayed on a 46-inch LCD monitor (Sony Bravia XBR-
III, 1,016 571.5 mm), 97.8 cm in front of participants (corresponding eld of view is
54:7

32:65

). To control the viewing distance, subjects used a chin rest to maintain
their head position during the experiment. A gray background (0.62 cd/m2) was dis-
played during the experiment. A headphone was used for presenting audio stimuli. Our
stimulus presentation program was developed using iLab Neuromorphic Toolkit (iNT)
and operated on a Linux 64bit machine.
4.2.1.2 Subjects
Fourteen female and one male undergraduate students with normal or corrected to normal
vision, participated for course credit. Participants' ages ranged from 18 to 23 years
(M = 20:9; SD = 1:6).
4.2.1.3 Procedure
Figure 4.1 displays a schematic view of the experimental paradigm. Visual-spatial stimuli
consist of two separately displayed red dots, each one placed randomly on an imaginary
circle at center of screen with a diameter of 3:125

angle of view. Each dot stayed on the
screen for 500 ms and a 500 ms blank screen separated the display of red dots. On the
virtual circle, dots were at least 90

and at most 120

apart. Subjects had to retain the
location of red dots for about 10 seconds during which they were supposed to engage in
a symbolic working memory task.
74
During a 10 second period after the removal of the second dot, audio signals of two
easily distinguishable types were played in a random order either in a slow tempo or a
fast tempo. We used two 250 ms long, 50 Hz tones as audio signals; a soft tone (sine
wave) and a rough tone (square wave). For the slow tempo condition, four signals were
played with a random ISI of 3000 ms to 3600 ms, and for the fast tempo condition 8
signals were played with ISI of 1330 ms to to 2000 ms.
In all trials, subjects were given an initial set of 3 separate digits (each could have
initial value between 0 and 3). In half of the blocks, subjects were asked to ignore audio
signals and to keep repeating three random digits played before the onset of visual targets.
We refer to this task condition as the ignore condition (IC). Under this condition, subjects
had to report these same three digits at the end of the trial. In the counting condition,
subjects were asked to increment the rst digit upon hearing the soft tone, to increment
the last digit unpon hearing the rough tone, and to remember the middle digit unchanged.
All three digits were reported at the end of trial. We refer to this condition as the engage
condition (EC).
The memory of the location of targets was probed at the end of a 10 second retaining
period by presenting two probe targets simultaneously. Probe targets were presented
either on exact same location as the initial targets (with 50% chance), or the location
of one of the probe targets was shifted along the imaginary circle at least by 45

and at
most 60

away from the location of the initial target. Subjects were supposed to respond
whether both probe targets appeared at the same locations as the original stimuli. During
the retaining period, a xation cross remained at the center of the screen. Subjects xated
the xation cross during the SWM task execution period. Subjects reported their three
digits by mouse clicks on a virtual keypad after responding to the visuospatial probe.
We administered the experiment in separate blocks of 20 trials for the engage and
ignore conditions. Each block contained equal numbers of trials with each possible tempo.
Each subject performed two blocks of trials for each engagement condition.
75
Sensitivity ( d')
1.5
Dual
Counting
Ignore
Signals
2.5
Slow Tempo
Fast Tempo
Figure 4.2: Impacts of task condition and audio signal rates on the sensitivity measure.
4.2.1.4 Results
Sensitivity of subjects in detecting matching probes was used to measure the impact of
the symbolic working memory (SWM) task onto the visual-spatial short-term memory
(VSSTM) task. Figure 4.2 demonstrates the mean value of sensitivity (d
0
) in identifying
matching visuosptial probes, for dierent conditions. To determine the signicance of the
impact of task and tempo factors, d
0
values were submitted to a two-way ANOVA with
repeated measures on both factors.
The analysis revealed a main eect of task at signicance levelp< 0:0001 [F (1; 14) =
37:69]. The tempo of audio signals did not show a signicant main eect [F (1; 14) =
0:504;p = 0:49]. No signicant interaction between factors was observed [F (1; 14)0:11;p =
0:74].
Further analysis revealed that sensitivity was higher in identifying identical probe
targets when subjects ignored audio signals (M = 2:46;SD = 1:29), compared to when
76
subjects were engaged in dual counting of audio signals (M = 1:57;SD = 1:21). More-
over, increasing the tempo of audio signals decreased the mean value of d
0
for both task
conditions; however, this change did not reach a signicant level.
4.2.1.5 Discussion
Our results indicates that, rst, the sensitivity measure is suciently sensitive for de-
tecting the impact of a secondary working memory task such as the dual counting, even
though the load on the VSSTM appears to be half of the capacity of visual-spatial short-
term memory in normal subjects Luck and Vogel (1997). Second, the double counting
task can impair the retention of visual-spatial information over a short period of time.
A signicant impact on the sensitivity measure with such a low amount of spatial infor-
mation suggests that the dual counting task, independent of the tempo, can potentially
impact the spatial span too.
One may maintain that this eect is caused by engaging CE in the dual counting task.
However, on the basis of the sensitivity measure, increasing the rate of dual counting
neither showed a main eect nor an interaction with the VSSTM task. Based on this
result, one may come to the conclusion that increasing the tempo indeed does not change
the complexity of the task, and thus does not add to the load on the central executive.
In this sense, the double counting might use a specic amount of executive resources
in lapses associated to each signal presentation event. Garavan has proposed a model
for a self-paced version of dual counting task which consists of ve steps: 1. stimulus
identication, 2. orientation of attention to the associated counter, 3. updating the count,
4. rehearsing the other count, 5. key-press. The rst four steps can be used as a model
for our version of the dual counting task. Previous research suggests that executive
attention does not play a direct role in the rst step He and McCarley (2010). Also, verbal
rehearsing in step 4 is suggested not to be dependent on executive resources Repovs and
Baddeley (2006). However, any load caused by steps 2 or 3 is constant for every turn of
the signal presentation. This means that, according to this model, a higher rate of signal
77
presentation engages the executive resources in dual counting task for a longer fraction
of time.
Yet, one needs to propose how CE may play a role in retaining VSSTM, if one wants
to associate the observed impact to a possible engagement of the CE in steps 2 and 3 of
Garavan's model. One may propose that retention of VSSTM requires active maintenance
through a rehearsing process Awh et al. (1998, 1999), which according to Jonides is a
\controlled sequence of retrievals and re-encoding of items into the focus of attention"
Jonides et al. (2008a). This may also draw on general executive resources needed for
manipulation of information in symbolic working memory tasks. This argument hinges on
this assumption that rehearsing prevents VSSTM traces from decay, so that interrupting
the rehearsal process results in decaying traces of spatial short-term memory. Yet, this
would imply that the more the rehearsing is interrupted, the more the eect of decay is
pronounced. This in turn suggests that adding to the rate of dual counting may aect
the performance on the visual-spatial task, which is not supported by our results.
Without CE as the shared scarce commodity between the SWM task and spatial
information maintaining process, one should consider another source of con
ict between
manipulation of information in the SWM task and retaining spatial information. One
source of con
ict could be that visuospatial short-term memory is indeed used during the
SWM task.
The use of space for the manipulation of information has been previously discussed
for specic SWM tasks such as immediate reverse recall Rudel and Denckla (1974) and
mental sorting of numbers Noori and Itti (2011). One may imagine a use of space as
natural addressing system for the content of SWM, which can be used as a handle to
shift processing to dierent items of working memory Noori and Itti (2011).
Our next experiment explores this matter in the case of a mental sorting task by
measuring the sensitivity in detecting a location change along the horizontal versus the
vertical directions. An account based on a bottleneck in executive resources for the impact
of SWM on retention of spatial information maintains that interrupting the CE would
78
aect VSSTM independently of the spatial location of visual targets. Hence, our second
experiment provides us with another opportunity for testing the role of CE in retaining
visual-spatial information.
4.3 Experiment 2
To test whether the observed in
uence of the SWM task on VSSTM is due to utilization
of space for active manipulation of symbolic working memory content, we examined the
selectivity of the impact of a mental sorting task on VSSTM. In particular, we used two
visual-spatial targets either along the horizontal orientation or the vertical orientation.
Subjects performed a sorting task on a random list of English letters during the visual-
spatial information retaining period.
4.3.0.6 Subjects
Eleven female and three male native English speaking undergraduate students with nor-
mal or corrected to normal vision participated for course credit. Participants' ages ranged
from 19 to 22 years (M = 20:39;SD = 1:4).
4.3.0.7 Procedure
The procedure for this experiment is similar to experiment 1, except for the location of
visual targets and the symbolic working memory task ( see Figure 4.1 ). Visual targets
were two red dots presented either along a horizontal line or a vertical line passing through
the center of screen, each dot on one side of the center, and between 1

:::4:9

angle of
view away from the center. Visuospatial probe targets were presented simultaneously in
the same locations as target stimuli with 50% chance, otherwise, one of probe targets was
displaced by 1:4

, either inward or outward along original presentation direction so that
two probe targets remained on two sides of the center cross along the direction of initial
presentation.
79
Before the onset of the red dots, four randomly selected English letters were presented
aurally to be maintained in the same presentation order (during maintaining trials), or
sorted in alphabetical order (during sorting trials), within a 10 second period. At the
end of the delay period, subjects rst responded to the visuospatial query, followed by
reporting four characters by mouse clicks on a virtual keypad displayed on the screen.
We administered the experiment in separate blocks of 20 trials for the maintaining
and sorting conditions, but each block contained equal number of trials for each dierent
direction for the presentation of visual targets. Each subject performed two blocks of
trials for each task condition.
Sensitivity (d')
Sorting Maintaining
1
2
Horizontal
Vertical
Figure 4.3: Average sensitivity measure for two tasks and two target orientations (experiment
2).
4.3.0.8 Results
Figure 4.3 demonstrates the mean value of sensitivity (d
0
) in identifying matching vi-
suosptial probe targets for dierent conditions of SWM task (Maintaining vs. Sorting)
and visual target orientations (Horizontal vs. Vertical). To determine the signicance of
80
the impact of task and target orientation factors d
0
values were submitted to a two-way
ANOVA with repeated measures on both factors.
The analysis revealed a main eect of the task [F (1; 13) = 8:43;p = 0:012]. No
signicant main eect of target orientation was determined [F (1; 13) = 3:12;p = 0:10]
while the interaction was marginally signicant [F (1; 13) = 4:3;p = 0:058]. A post-
hoc correlated-samples one-way ANOVA revealed a simple eect of the task, only for
horizontal targets [F (1; 13) = 15:26;p = 0:0018], and no simple eect of the task condition
at the level of vertical visual targets was observed [F (1; 13) = 0:03;p = 0:86].
Moreover, further analysis for exploring simple eects of orientation at dierent task
levels showed that under the maintaining condition subjects demonstrated a higher sen-
sitivity in detecting identical horizontal probe targets (M = 2:02;SE = 0:19) compared
to identical vertical probe targets (M = 1:28;SE = 0:29). A correlated-samples one-
way ANOVA revealed that the simple eect of orientation during the maintaining task is
signicant [F (1; 13) = 5:11;p = 0:041].
4.3.0.9 Discussion
As the analysis revealed, compared to maintaining of four characters in their original order
for a later recall, sorting them into an alphabetical order could signicantly in
uence
the sensitivity measure. This result again demonstrates the capacity of the sensitivity
measure in registering the impact of a secondary SWM task on temporary retention of
spatial information. Given our signicant results under the low amount of load on VSSTM
in our location change detection, one would also expect an impact on spatial span tasks
(higher load) due to engaging in a mental sorting task.
However the striking result was that the impact of the sorting task on the sensitivity
measure is only signicant for visual targets that are spanned along the horizontal direc-
tion. Switching task condition did not change the average sensitivity to shift in location
of targets along the vertical direction.
81
The sensitivity to change of location for vertically spanned visual targets was signi-
cantly above chance and, unlike the horizontally spanned targets, switching to the sorting
task did not decrease sensitivity. This is consistent with the nding of the previous ex-
periment, in that the in
uence of SWM task on the retention of spatial information is not
caused by involvement of executive resources in spatial information retention; otherwise,
one would expect an in
uence on the sensitivity for vertically distributed visual targets
too. The initial sensitivity along vertical orientation was lower than along the horizontal
orientation, hence one may raise the point that there was less room for decreasing the
sensitivity along the vertical direction, and detecting a change would need more space.
Controlling for the in
uence of this initial dierence on the sensitivity in location change
detection remains to be tested in a separate experiment, with a setup that can balance
the sensitivities for detecting target locations along the vertical and horizontal directions
during the list maintaining task.
4.4 General Discussion
The goal of this study was to explore the in
uence of intellectual working memory tasks
devoid of visual and spatial features on the ability of retaining visuospatial information
over a short period of time. We used a measure which is dierent from the actual capacity
of spatial memory for holding spatial information. Instead we used the ability to detect
a change in spatial location of one of two simple visual targets. In both experiments we
observed that engaging in active manipulation of working memory content results to a
decrease in the sensitivity of subjects in detecting a change in location of targets which
needs to be explained.
We chose this particular measure to be able to distinguish the predictions of our
Spatial Registry Hypothesis about impairment of the visual-spatial short-term memory
as the result of engaging in a secondary executive working memory task. SRH predicts
an impact of intellectual task on spatial short-term memory of visual targets as the result
82
of interference of the use of same representation for registry of symbolic representation
of working memory task items. This would suggest a selective impact which is also
independent of executive engagement. These two aspects were validated in two separate
experiments.
Theories of working memory in the realm of cognitive psychology |independent of
what they assume about the nature of representation in WM | usually assume a specic
execution model based on separation of storage and execution. As such, a con
ict between
two tasks is either associated with sharing storage or with drawing on limited executive
resources. Given that the WM tasks in our study are devoid of immediate visual features,
one may conclude that the source of con
ict is the the dependency of the retention of
visual-spatial information and WM task on the CE. Yet, one should be clear as to how
CE explains this con
ict rather than | as Baddeley and Repovs have stated (Repovs
and Baddeley, 2006) | using CE as a homunculus which has an undisclosed role in
everything.
As we discussed in our introduction Baddeley's latest account assumes no role for the
CE in retention of spatial information as simple as we tested in our experiment.
We also discussed that the proposal of Awh and his colleagues for engagement of a
rehearsing mechanism in maintaining visuospatial information (Awh et al., 1998, 1999),
and Jonides' account for the dynamic of rehearsing process (Jonides et al., 2008a), may
suggest a role for the CE in retention of visuospatial information. Our experiments were
able to test this hypothetical role and our results did not support it.
Another view of working memory, proposed by Cowan (Cowan, 2001), has recently
gained popularity in the cognitive psychology community. According to Cowan, working
memory content is a part of long-term memory in a heightened state and under attention
(Cowan, 2001). He explains the limitation in the capacity of working memory by the
limitation of the internal attention in covering only four items at a time. The role of CE
in this schema is to dynamically dispatch attention between representations in the long
term memory, to make them available for processing. Accordingly, one might say that
83
attention might be shared between information about the locations of two dots and the
identities of four characters of the sorting task, which would exceed the capacity limit of
4 items proposed by Cowan. Yet, this explanation lacks sucient detail to explain why
adding to the rate of double counting and fastening this juggling of content of working
memory under the watch of attention has no impact, or, in our second experiment, the
eect of sorting characters in memory is limited to the sensitivity in detecting changes
along the horizontal orientation.
84
Chapter 5
Dierential Impact of Visual Presentation on Forward and
Backward Recall
In exploring the spatial selectivity of the impact of a secondary sorting task on the
visuospatial short-term memory we observed that subjects demonstrate an advantage in
retaining visuospatial information along the horizontal direction (the ignore condition of
the second experiment - see our experimental result in section 4.3.
It is important to note that in that study in baseline conditions subjects retained only
some verbal material which presumably can be handled via serial access strategy. Given
the load on visuospatial part of the task in baseline studies, it is optimal to shift the load
of the mental task to phonological resources to have spatial resources free for visuospatial
STM task. This suggests that the higher sensitivity in detecting target change along the
horizontal direction compared to the vertical direction might be the result of an advantage
for the retaining information along horizontal direction. This might be the result of
long exposure to a bias in spatial information of task relevant items along the horizontal
direction. Richness of the information along the horizontal line or the reading habit along
the horizontal line may have some in
uence in forming such bias. Such stimulation bias
along the horizontal line may result in a higher resolution for visual-spatial short-term
memory along the horizontal line.
Such a dierence in spatial STM retaining features may fold on performance of intel-
lectual tasks if one can guide the spatial binding to dierent regions of the space with
85
dierent spatial information retaining features. In our eye tracking studies we observed
that initial visual presentation of task items during sorting tasks modulates eye move-
ments during the mental sorting and when items were not visually present along the
initial presentation orientation. This suggests that initial presentation may prime or be
suggestive for using a the orientation direction for the location registry.
In this experiment we will utilize what we learned from two previous studies to test
whether initial presentation of mental task items in regions of the space for which main-
taining sensory information is is selectively dierent would later on aect the performance
in mental task.
We also discussed that forward and backward recall usually rely on dierent access
strategies and thus dierent sensorimotor systems. In particular we discussed that forward
recall in normal subjects would rely on phonological resources while backward recall would
rely on visual-spatial resources (e.g. ocular resources). This suggests that manipulation of
orientation of initial presentation of task items would dierentially aect the performance
of forward and backward recall.
5.1 Hypothesis
Given above discussion we posit the following hypothesis which helps devise our experi-
mental paradigm in this study:
Horizontal or vertical presentation of items for a forward recall tasks will not aect
the performance while it will have a dierence in performance of the recall in backward
order in particular in favour of the horizontal initial presentation.
This hypothesis dictates two main parameters for control: type of the recall task
(forward or backward) and initial presentation orientation of task items (horizontal versus
vertical).
86
s
s
g
v
q
j
k
l
9 d
d
k
p
z
m
s
g
v
q
k
d
t b j f r m
t
t b
t b j
Recall type
cue
t b j f r m
3 secs
3 secs
Until Response
7 secs
2 secs End of trial
b c d f
g h j k
l m n p
q r s t
v w x z
spc ok <-
click to start next trial
Figure 5.1: Schematic view of the paradigm for Forward-or-Backward recall. Items presented
either along the horizontal direction (shown in the image) or the vertical direction. The order of
recall was postcued.
5.2 Experiment
We measure the performance of our subjects during a forward-or-backward recall task
(see Figure 5.1) in which subjects rst read some items from the screen without knowing
whether they have to be recalled in a forward or a backward order. This constraint
ensures that encoding processes would be identical for both recall directions.
We measure the performance in both backward recall (executive memory task) and
forward recall (active maintenance) by errors in recalling items in their order. The forward
recall, presumably draws only on the phonological loop and thus the presentation method
is not supposed to have any eect on the the performance. However backward recall
requires memory manipulations and therefore is potentially sensitive to task-irrelevant
visual presentation method.
Items for recall tasks were presented visually using a setup similar to previous ex-
periments (see subsection 4.2.1). Two orientations were used for presenting the items :
horizontal and vertical. Presentation orientation was xed for all trials in each block (see
Figure 5.1).
Each trial started with the subject's mouse click. Six random non-repeating English
characters (a, e, i, o, u and y were excluded) appeared on the screen, on a virtual line
87
passing through the center of the screen, one item at the time with a 300 milliseconds
time interval between them. During the horizontal presentation items appeared from
left to right and for the vertical presentation items appeared from top to bottom. Items
were separated from each by 0:42

of angle of view gap. After the presentation, each
item remained on the screen until 3 seconds after the sixth item was presented. Items
then were replaced by a mask of randomly scattered characters across the screen for 50
milliseconds. Three seconds after removal of the mask subject were aurally cued about
the type of the recall. So subjects did not know about the type of recall until at least six
seconds after all items appeared on the screen and three seconds after all items removed
from the screen.
Upon the subject's mouse click a virtual keypad would appear on the screen to help
report the items. Subjects had no time limit for performing the recall task, however,
reporting the items after the mouse click was speeded. Subjects were supposed to report
the recalled string within a 7 second time window. The time constraint was set to force
subjects to nalize the reordering (in the case of backward recall) before signaling the
end of task performance.
After expiration of this time interval the virtual keypad would be replaced be a mes-
sage for starting the next trial. Subjects were allowed to skip a forgotten item by pressing
a place holder on the virtual keypad. Failing to report the proper item in a position was
counted as an error.
The experiment was administered in four blocks. The presentation orientation changed
between two consecutive blocks. Half of the subjects started with the horizontal presen-
tation. Each block contained 10 trials of each type of recall. This left us with 20 trials
for each combination of recall type and presentation orientation per subject. The order
of recall types in a block was chosen randomly.
88
5.2.0.10 Participants
Nine female and seven male undergraduate USC students with normal or corrected to
normal vision, participated for course credit. Participants' ages ranged from 18 to 22
(M=19.2 years, SD=1.03). Fifteen subjects reported to be right handed and one subject
reported to be left handed.
89
Horizontal Vertical
Backward recall
Error rate
0.1
0.3
Forward recall
Figure 5.2: Average error rates in recalling items in their right order for all four combinations
of recall order and initial presentation method. Presentation method has a signicant impact on
the performance of the backward recall while the impact on the forward recall is not signicant.
5.2.0.11 Results
Figure 5.2 displays the average error rates in recalling the items for all combinations of
the presentation orientation and the recall type.
Our result shows that the average error rate for backward recall (for both presentation
methods) was signicantly larger than forward recall. The average error rate for forward
recall was 12:42%1:9%(MSEM) which was signicantly less than 31:2%3:1%(M
SEM) for backward recall paried ttest;n = 16;p< 0:00001).
Our result indicates that when the task was forward recall, switching the presentation
orientation from horizontal to vertical resulted to a change in average error rate of recall
from 11:04% 1:89%(MSEM) to 13:8% 2:2%(MSEM). The average change in
error rates was 2:76%1:12%(MSEM) which statistically is not signicant (ttest;n =
16;p> 0:112 ).
90
However, when the task was backward recall, switching from the horizontal presenta-
tion to the vertical presentation resulted to a signicant change from 26:87%3:07%(M
SEM) to 35:5% 3:33%(MSEM), (paired ttest;n = 16;p< 0:00001). The average
change in error rates in this case was 8:86% 1:6%(MSEM).
91
-0.2
0.2
Change in error rates in forward recall
Change in error rates in backward recall
0.2
-0.2
Figure 5.3: Change in the performance of recall tasks by switching the presentation method from
horizontal to vertical for all sixteen subjects.
repeated measure two-way ANOVA with error rates as the dependent variable and
2( Forward vs Backward) 2 (Horizontal vs. Vertical) as condition. signicant main
eect for Task F (1; 25) = 95:8798;p< 0:000001 and presentation orientation F (1; 25) =
20:0065;p < 0:0004 and signicant interaction between task condition and orientation
condition F (1; 25) = 8:7319;p< 0:0098.
Simple main eect of orientation in backward recall F (1; 15) = 29:17;p < 0:000073
and no signicant simple eect of orientation on forward recall F (1; 15) = 2:8518;p>
0:112
5.3 Discussion
Results of this study clearly support our initial hypothesis in that the horizontal pre-
sentation of items during the backward recall results in better performance compared to
initial vertical presentation.
92
In a previous study selective impact of initial presentation method on the performance
of backward recall was reported by Li and Lewandowsky (Li and Lewandowsky, 1995b).
In that study items of for a forward recall were presented either at the center (on item at
a time) or in random places. They observed that random presentation of items selectively
impacts only the performance in backward recall and not forward recall with lower per-
formance compared to presentation of items at center. Their result suggests that unlike
recalling in forward order, backward recall relies on visual-spatial resources. Their nding
is consistent with our observations in eye tracking study as well as this study.
However, Li and Lewandowsky's result may not be used by itself to predict our ndings
in this experiment. Our location registry hypothesis which assumes items for a backward
recall rely on spatial resources can account for Li and Lewandowsky's ndings and also
can be used to explain our ndings in this study. Yet to account for the nding of
this study based on location registry hypothesis we need to establish that binding or
registry of items along the vertical direction is less advantageous compared to binding
them horizontally. This is consistent with ndings of our previous study (see Chapter
4) in that sensitivity of subjects in detecting location change of visual targets along the
horizontal orientation was signicantly better that that of vertical orientation.
93
Part III
SWMS: A Sensorimotor Account for a Symbolic Working
Memory System in Intellectual Domain
94
Chapter 6
SWMS: The Symbolic Working Memory System
Here we present a working memory management model for performing intellectual tasks
based upon the assumption of accessibility and utilization of evolutionarily older senso-
rimotor systems with the capacity of maintenance of domain-specic short-term mem-
ory representations. We argue that working memory management for intellectual tasks
that feature algorithmic characteristics is enabled by utilization of a number of semi-
independent sensorimotor systems that are capable of maintaining and management of
some active short-term and system-specic representations. This STM-enabled sensorimotor-
supported management of working memory provides rapid, reliable and
exible memory
management that is required for ecient performance of many intellectual tasks.
We refer to those sensorimotor systems that take part in the working memory man-
agement for intellectual tasks and out of the realm of their perception-action functions,
as `symbolic utility systems' or `utility systems' for short. We also refer to a collection
or assemblage of those utility systems functioning in the form of an integrated general-
purpose system for working memory management serving intellectual cognition as the
Symbolic Working Memory System or SWMS. Utilization of this integrated system often
comes with employing several intermediary symbolic representations in utility systems
which refer to items of an intellectual task.
In this section, we present a functional model for the SWMS with respect to what
dierent utility systems can aord the for management of information. The original praxis
95
which has helped shape a particular sensorimotor system determines the characteristics
of that system in the maintenance and management of information. Despite variations of
praxes and varieties of sensorimotor systems their, function can be conceptualized by the
way they provide access to their maintained information. We use the concept of working
memory access as a fundamental concept to dene the functionality of each utility system
in the domain of intellectual tasks.
Towards this, we oer an access-based taxonomy of symbolic utility working memory
systems. We assume two categories of utility working memory systems: 1. randomly
accessible working memory (RAWM) systems and 2. serially accessible working mem-
ory (SAWM) systems. We discuss that this categorization also re
ects the fundamental
characteristics of sensorimotor systems in augmenting sensory or motor information with
spatial or temporal attributes. Random access is provided by those systems with ca-
pacity of adding spatial or location attributions to sensory or motor information, and
serial access is provided by those systems with temporal encoding of sensory or motor
information.
Each access mode has limitations and advantages which is the result of underlying
mechanisms of the supporting utility system. Those limitations and advantages determine
the range operations the might be supported. For example our proposed implementation
of SAWM does not impose a xed capacity in terms of number of maintained items which
allows strategic use of symbolic codes for maximizing the number of stored items, yet,
this access mode allows a unidirectional access which imposes a strict limitation on a
number operations required for some tasks such as reverse recall of a list of items.
The functioning or execution of operations within and between utility systems is
explained using a distributed and embedded executive system. Our focus in this work
is on explaining executive models of utility systems in their independent functioning.
However, an important aspect of SWMS is related to the capacity of using several utility
systems in a coordinated way towards allocating even more resources.This will result to
a higher degree of
exibility.
96
The Symbolic Working Memory System relies on fundamental resources which regu-
late rudimentary functions of the brain and in the meantime its functions and its func-
tioning is fundamental for intellectual tasks. However, there is a strong strategic aspect
to SWMS so that the presence of constituting sensorimotor resources determines only an
upper limit for functioning of symbolic working memory system. The actual performance
level is determined by the type of strategies employed in using very limited resources of
SWMS. It is from this perspective that general intelligence which re
ects many aspects
of cognition appears to be less correlated with WM Ackerman et al. (2005) and
uid
intelligence which re
ects capacity for symbolic processing appears to be more correlated
with WM (Engle et al., 1999). We will discuss evidences that suggest these strategies
might be induced by eduction in early ages and as the result of schooling.
The impact of limitations of SWMS is minimal if those limitation are the result of
poor strategic choices in using utility systems. Yet, sometimes limitations in SWMS
are in fact manifestations of decits in underlying sensorimotor systems. In these cases,
limitations in performing intellectual tasks may coexist with serious limitations in other
aspects of cognition arisen from a decit in basic functions of the brain. For example, the
case of Gerstmann's syndrome (Nielsen, 1938) or Balint-Homes' syndrome (Vallar et al.,
2007), which are described as a constellation of both intellectual and spatial or motor
disabilities, including dyscalculia, nger agnosia and left-right confusion or disruption in
bodily schemas (Nielsen, 1938; Vallar et al., 2007).
The following two subsections are devoted to explaining two important aspects of our
proposed working memory system: representation and process. Although we emphasize
that these two aspects are functionally indivisible, it is useful to present our argument with
respect to traditional perspective of cognitive psychology of working memory. This may
help explain the diculty of separating representation and execution and the rationale
for replacing them with the concept of working memory access.
97
6.1 The Symbolic Interface: the blueprint for a bridge
Our eort in this work is giving a view of memory management in the domain of in-
tellectual tasks which, rst, will allow a formal description of behavior at high level in
terms of symbolic processing concepts which conform with psychologist's observations
and, second, will allow a neural mechanistic account consistent with functioning princi-
ples of rudimentary and low-level functions of the brain. Our key assumption here for
building our framework is that underlying information management mechanisms in senso-
rimotor systems are directly involved and responsible for management of information for
intellectual tasks. This assumption is based upon reusing and, bending those information
management mechanisms that have appeared earlier in our evolution, towards achieving
recently gained capacity of performing intellectual tasks.
This view opens to the possibility of an embodied experience of performing intellec-
tual tasks. The assumption of direct involvement of neural mechanisms of sensorimotor
systems suggests that management of symbolic information for an intellectual task is not
exclusively a mental experience and it involves embodied experience situated in senso-
rimotor systems. This stretches the mechanistic aspect of high-level cognitive tasks in
manipulation of symbols to the most basic features of our experience of the physical world
which we are shared with lower organisms.
However, we argue that it is important to formulate working memory management
mechanisms in terms of a symbolic process language. First, symbolic processing concepts
are suitable for description mental aspect of the behavior in intellectual tasks. Second,
this symbolic representation of the mental state can be used for reconstruction of the
entire experience, if, a. operational aspects in the symbolic level are closely mapped onto
the supporting sensorimotor mechanisms and b. mechanistic aspects in sensorimotor
functions are closely mapped onto supporting neural mechanisms. The key step here is
mapping symbolic processed to brain mechanisms which are biologically justied. This
is the insight: once symbolic processes of working memory management are translated
98
into sensorimotor mechanisms then our knowledge about neural basis of sensorimotor
mechanisms are enough to explain `mechanistic aspects of information management' of
symbolically performed intellectual tasks.
Our proposed framework accommodates for such mappings from a mental descrip-
tion onto a sensorimotoric description and then onto a neural mechanistic description.
These mappings are conceptualized using a structural approach described in four levels of
analysis with dierent degrees of abstraction. We refer to this structure as the Symbolic
Interface or SI for short (see Figure 6.1).
A third and very important benet of a full description of the behavior in terms of
symbolic processing concepts is providing the opportunity to connect to any model of
cognitive dynamics through a symbolic interface. There are a number of formalisms that
give a description of symbolically-intelligent mind which in turn rely on a realistic model
for management of information (Anderson, 1996; Derbinsky and Laird, 2010; Laird et al.,
1987; Meyer and Kieras, 1997). Providing a symbolic interface for memory management
will facilitate connecting the model of working memory as an AS-IS module to those
formalisms.
In our framework at the most abstract and symbolic level, functioning of the work-
ing memory system is described using access schemas. For this reason we refer to this
level as the access level or symbolic level. An access schema is a description of memory
management in terms of a limited number of operations which are supported by a utility
sensorimotor system. Access schemas give rise to the concept of integrated manage-
ment mechanism: instead of assuming a bipartite system of `storage' `versus' `execution'
we apply the concept of `working memory access' which has both concepts of storage
and management of information integrated. This integrated perspective of management
of memory at the symbolic level is a re
ection of integrated or embedded functioning
mechanism of sensorimotor systems that support those symbolically represented working
memory (wm) management operations.
99
6.2 Modal and Symbolic Representations in Utility Systems
Utilization of a sensorimotor system for symbolic working memory management relies on
a set of system-specic representations which may transiently become activated during
the course of task execution. Those representations might be arbitrary representations
of a concept in a specic utility system. For example, for a hearing-abled individual, a
phonological representation is associated to the concept which for a visually-abled person
is graphically represented by `2' or `II' or `'. For a vocally-abled individual, those
visual and phonological representations may associate with a motor representation in the
vocal apparatus which enables producing a similar phonological representation. Those
symbolic representations may or may not make a reference to a meta representation (e.g.,
a representation of 2 which is dierent from its visual or phonological representations
associated to the concept of 2).
The set of designated representations in sensory and motor cortices determines appli-
cability of utility systems in management of task-relevant items, and in part determines
what kind of concepts can be reliably handled using the SWMS. A part of the learning
process in performing intellectual task is forming designated utility symbolic representa-
tions in associated sensory and motor cortices.
Moreover, there are some functional limitations associated with underlying represen-
tation mechanisms. For example, the capacity limitation in a specic utility system is
closely related to underlying representation mechanisms. Some utility systems implement
an object-based representation and their capacity limitation can be measured in terms
of number of items. Representation in some of utility systems rely on buering sensory
traces or motor codes associated to a time interval. The amount of information that can
be encoded in that time interval may give an estimation of capacity limitation in those
systems. We will discuss possible sources of this limitation in both random access and
serial access memory systems.
100
It should be noted that the mode in which items are represented is not determined
by the very same mode in which task items are presented. Depending on the type of
necessary access and characteristics of necessary control actions, co-activation of dier-
ent representations might be necessary for representing a particular task-relevant item.
Presentation of task relevant items may prime or be suggestive for the use of a particular
strategy for performing a task; however, what determines the modality of representations
in SWMS is the strategy for performing the task. A strategy should be devised with
respect to not only the type of operations needed for the mental task but also availability
of utility systems for the operation in a certain condition. For example, maintenance of
a number of digits for a later recall in their presentation order can be handled by utility
systems with serial access capability (e.g., the phonological resources), however, in a dif-
ferent task which requires reporting items in the reverse or backward order utility systems
with capacity of random access are required and thus independent of initial presentation
mode items are required to be represented in a system with random access capacity (e.g.,
visuospatial system).
A major source of con
ict is related to dual function of utility systems in both intel-
lectual domain and perception-action routines. Thus, functioning of utility systems in the
context of their original sensorimotor function may interfere or in
uence their function
in SWMS. The disruptive eect of irrelevant sensory input on the capacity maintenance
of a list of items has been traditionally used to investigate into possible roles of a sensory
mode in internal encoding in WM (Baddeley and Salam e, 1986; Jones and Macken, 1993;
LeCompte, 1996; Quinn, 1996; Salam e and Baddeley, 1986; Wilson and Emmorey, 2003).
Another important aspect of representation of task relevant items in utility systems
is that often several representations of dierent nature are needed to represent one single
item. For the case of RAWM at least two representations of two dierent nature are
involved: a representation of object and a representation of address. These two represen-
tations might be encoded in dierent systems, yet both help represent an item (visual or
haptic) in a specic address (position or location). This schema for encoding task relevant
101
items in working memory helps access an item through an address. Also, depending on
the type of a specic SAWM system two representation, one of sensory nature (auditory
or visual) and the other of motor nature (in vocal apparatus or hand movement) might
be necessary for encoding one single tasks relevant item. This schema for encoding in-
formation in SAWM systems is necessary to access items of the working memory in a
sequence and is critical for serial access.
6.3 A Schema-Driven Distributed Execution Model in Dierent
Levels of Analysis
We propose a distributed execution model for management of information within and
between comprising utility sensorimotor systems in SWMS. In doing so, we build on the
schema theory (Arbib, 1992) and propose our adaptation of schema theory, tailored for
our target behavior. One of the features specic to our adaptation is including a level of
description which can capture symbolic processing aspect of intellectual tasks in rather a
formal way.
So, at the behavior end, we envision utilizing a formal symbolic processing represen-
tation. This would enable integration of our model into any model of cognitive dynamics
(e.g. cognitive architectures (Anderson, 1996; Laird et al., 1987; Meyer and Kieras, 1997)
) with symbolic processing interface. This would be an important investment for that we
do not present a full model for cognitive processing in the symbolic domain. Our model
is limited to management of STM-based working memory content which is a crucial com-
ponent functioning in a bigger context. Note that we are not presenting \The Working
Memory" as a proxy for human cognition. Our goal is limited to devising a short-term
based working memory system that explains only the management of information and
in terms of storage and access together. We argue that this system is a crucial part of
intellectual task that is shared between a range of tasks that are involved with symbolic
content and this justies investing on devising a general purpose machinery which serves
102
symbolic cognition. Symbolic cognition needs other components in addition to a memory
management system.
Two incentives motivated us for providing a formal symbolic processing interface to
our model. First, an important aspect of human behavior can be captured in symbolic
processing formalisms (the mental aspect) which later on can be used to reconstruct
the behavior in the larger and embodied context. Second the practical importance of
providing an interface for other models of cognitive dynamics which gives the outlook of
a bigger picture of cognitive functioning.
With this reasoning we break the task into two steps: rst, arranging a scaold for
organizing our model in dierent levels of abstraction with a formal and abstract level at
one end and a level for neural mechanisms of the brain at the other end (see Figure 6.1),
second, occupying this scaold with our proposed models associated to two access modes
(see Figure 6.2).
We refer to the scaold of the rst step as the Symbolic Interface of (SI). The scaold
structure is inspired by this feature of the schema language in presentation of the model
in dierent levels of abstraction.
SI is our proposed way to build a formal model for description of functioning of
a working memory system with symbolic processing functions that contains details of
functioning mechanisms in dierent levels of abstraction and in an inter-connected way.
In fact we leveraged the
exibility of schema theory in supporting dierent levels
of abstraction (Arbib et al., 1987) and adapted a four-level framework presented with
specic levels of abstraction at each level (see Figure 6.1). We dedicated a level to the
description of formal aspects of behavior in terms of abstract symbolic processing concepts
of WM management. The concepts at abstract level are not arbitrary mathematical
notations and are biologically constrained to neural mechanism that are presented at the
second level which in turn are a generic description of neural mechanism that support a
biologically relevant praxis. Evolutionary arguments inform our modeling eort at the
second and third level of this formalism. Presented models at the second level also can
103
be seen as general description of all possible systems that alternatively can serve the
same information management functions at the abstract and symbolic level. All possible
instances of presented generic systems can be presented at the third level of analysis
where they can be supported with neural models at the fourth level. In this way our
symbolic interface at one end gives an abstraction of access models which is a suitable
language for symbolic processing description of information management in intellectual
tasks and at the other end deals with mechanisms of regulating those access models in
terms of neural basis of sensorimotor systems.
In this paper we organize our analyses in the form of a hierarchy of four levels of
abstraction to accommodate for dealing with symbolic aspects of target behavior at one
end, and neural mechanisms at the other end. At the highest level of abstraction (the
lowest level in the diagram in Figure 6.2), the schema formalism may be presented with
no explicit reference to the underlying mechanism. Yet, in devising characteristics of
concepts and operations at this level their relationship with supporting utility systems is
considered.
To separate implementation details from conceptual details and yet having them in
close relationship, we organize our discussion about supporting sensorimotor mechanisms
in two levels between the abstract level and the neural level. The level next to abstract
access level contains a generic formulation of those sensorimotor systems that support dif-
ferent access modes in abstract level. Those sensorimotor systems related to our presented
taxonomy are described in terms of generic mechanisms available to all implementations
of such systems. As the result the description of utility systems at this level is made with
respect to both actual sensorimotor systems and description of access modes.
This paper is mostly focused on more detailed analysis at the access level and generic
sensorimotor level. Yet we supplement our analysis with a brief review of the neural evi-
dences through reviewing evidences of instances of those generic utility system. A more
104
detailed review of neural evidence needs a comprehensive comparative study of litera-
ture in several domains including action-perception, working memory and mathematical
cognition.
For the purpose of clarity we refer to schemas at dierent levels with dierent terms.
From the abstract level to the neural level they are referred to as: access schemas, opera-
tional schemas, sensorimotor schemas and neural schemas. We may refer to all instances
as executive schema collectively.
Our discussion at the abstract level is mainly related to and promoted using the con-
cept `working memory access'. The concept of `access' captures both concepts of `storage'
and `operation' together and in an inseparable way. It not only covers the function of
storage of information but also determines or describes the way the stored information in
a particular utility system might be used in the process and during intellectual tasks. The
concept of storage by itself gives a static view of availability of the information. The only
dierence between static storage units might be in in the modality of storage (Baddeley
and Hitch, 1974) or state of representation (Cowan, 1999). In contrast, the concept of
working memory access re
ects the range of operations that can be supported by a utility
system in maintenance of information. The concept of working memory access, at the
conceptual level, enables analysing the procedure of intellectual tasks from a more formal
symbolic processing perspective.
From a top-down perspective, access models can be used to distinguish dierent types
of utility systems in terms of their symbolic processing function. However, from a bottom-
up perspective, access models are abstraction of dierent endowed sensorimotor mecha-
nisms that are utilized for intellectual tasks.
We assume two major types of access modes: random access and serial access. This
categorization of access modes gives a basis for a taxonomy of utility sensorimotor systems
at sensorimotor level of our analysis. In the generic level, two types of generic sensorimotor
systems in direct association with those access modes are presented. At this level, we
give an abstraction sensorimotor systems with respect to shared mechanisms between
105
those systems that serve random access and serial access to their content. These generic
utility systems will be referred to as as Location Registry System (LRS) and Sensory-
Articulatory Circular System (SACS). We argue that LRS utilises a location or space
representation towards providing random access to its content and SACS utilizes the
temporal aspect of encoding information in a pair of sensory and articulatory system for
serial access to their content. Executive model for these two systems is presented in terms
of interaction of components the system and in a schema theoretic language.
As it will be discussed, our proposed random and serial access in utility systems are
not simply some abstract mathematical concepts. A number of important features in our
access schemas are indeed abstraction of real systems that support these access modes.
For example, we will discuss how our assumptions about the nature of representation
which are determined by the original praxis of the sensorimotor system will aect our
description of access schema.
At the third level, or instance level, we relate those LRS and SACS to dierent
instances of sensorimotor systems. We argue that sensorimotor systems that match char-
acteristics of LRS and SACS are not unique. Although visuospatial systems which serve
object manipulation praxes and the auditory sensory-vocal articulation loop (or phono-
logical loop) are prevalent instances of LRS and SACS, other sensorimotor systems can
aord the same functions (Wilson, 2001).
The importance of access modes at the abstract level and in description of access
schemas makes our discussion at all levels mostly focused around two dierent systems.
Yet, it should be noted that implementing mixed strategies which employ both access
modes will add to the
exibility of the SWMS in handling working memory manage-
ment of intellectual tasks. Employing mixed strategies in turn requires between-system
interactions and careful handling of cross-talking between system. In this sense, one may
think that maybe the concept of a central executive is needed as a proxy for handling the
cross-talk between systems. However, from a theoretical perspective, there is no reason
to assume that the nature of interaction between systems is dierent from the interaction
106
between sensory systems and motor systems for forming a utility system. So, we argue
that just in the same way that some of sensory and motor systems join together and form
an assemblage that supports random or serial access they may join together to form an
assemblage of even more systems. We argue that the same theoretical principles that
can be used to describe sensorimotor utility systems can apply to description of their
assemblage.
Execution Schemas can be used for designing strategies for performing mental tasks
and in the meantime examining the behavior and also the dynamics of brain systems
in performing the task. In this sense, access schemas serves as a language which helps
translate an algorithmic procedure into cognitive behavior with respect to limitations of
supporting systems for implementation of the procedure. Thus access schema by default
can perform as a description cognitive architecture of humans' working memory.
In our view, limitations in intellectual cognitive functions in part arise from encoding
limitations of each engaged sensorimotor system. Therefore we argue that limitations of
the working memory system can not be simply explained by one single magic number.
Each of the utility systems may have a magic number attached. These system-specic
magic numbers might be helpful to give an estimation of limitations of working memory
system. A better estimation is possible with respect to a particular task and when the role
of each utility system in management of temporary information is known. Therefore, we
argue, a proper description of limitations of working memory system should be made with
reference to particular subsystems engaged in the task. Executive Schemas are supposed
to provide a reference to resources engaged in a given intellectual tasks and hence are
suppose to help identify the source of limitations with respect to the performance strategy.
This gives a drastically dierent view of limitations of working memory system compared
to traditional models of ascribing all limitations one unitary source. As we will explain
in the following sections, each of sensorimotor systems have an inherent limitation in
encoding and accessing stored information. The limitation in performing mental tasks
then emerges as the result of limitations of engaged sensorimotor systems and limitations
107
in co-ordination between them. Limitation of sensorimotor systems in terms of number
of items that can be maintained is only one of the sources of limitation.
In the following two sections we discuss two dierent symbolic working memory sys-
tems and suggest some of possible sources of limitations for each of those working memory
systems. We show how dierent assumptions about operational schema for dierent util-
ity system may result to a dierent method for estimation of their capacity limitation.
6.4 Randomly Accessible Working Memory and Location
Registry Systems
The capacity of random access to the content of working memory is a crucial element of
a dynamic and
exible memory management system. In general, random access schemas
allow direct and rapid access to the content of memory by utilizing an address space which
is operated under a parallel access mechanism. This address space also accommodates
more complex processing schemas such as variable binding.
We describe characteristics of working memory systems in the intellectual domain
that allow random access to their content through associating or attributing a secondary
representation which is subject to voluntary and parallel access. We give our description
with regard to biologically relevant praxes that rely on an address-based and parallel
access to tasks-relevant items in a short-term memory system. A generic sensorimotor
system that features minimum requirements for supporting those praxes will be used
for devising management schemas of our proposed random accessible working memory
system.
Object-centered actions in the organism's physical environment rely on spatial or loca-
tion representations that allow a common representation between perceptual experience
and motor interaction with the physical environment (Paillard, 1991). In performing
object-centered actions sensory inputs need to register with motor system in order to
guide sensory driven actions purposefully and with regard to the location of the sensory
108
signal. The source of the sensory signal may be a mate or food or a predator and thus
it is crucial to react or act with respect to the exact registered location in the sensory
system. In this context space or location serves as an index that can separate and in the
meantime dene a relationship between dierent stimuli and their relationship with the
body or the organism's actuators at each moment. Moreover, objects or stimuli in the
environment with biological relevance to the organism need to be processed in parallel to
grant instantaneous higher processing priority to the stimulus of higher relevance to the
behaviour. The pop-up eect or stimulus-driven saliency which drives visual attention
to guide foveal vision towards a conspicuous target is postulated to be the result of a
parallel processing of visual stimulus across the space (Itti et al., 1998).
Also there are object-centered praxes with high adaptive value that rely on mainte-
nance of location of a number of task-relevant objects in the space, in particular when
those objects are still present and biologically relevant to the behavior and are no longer
reachable to the sensory apparatus. A short-term memory system that is capable of active
maintenance of objects of interest and their locations relative to actuators in the same
physical space provides a great adaptive advantage in reacting to previously detected ob-
jects which are not detectable at the moment and based on their assigned or attributed
location.
Our description of a generic sensorimotor system which serves random access to the
content of working memory is given with respect to above sensorimotor systems that allow
maintenance of location or spatial information of task relevant items for object-centered
actions. We refer to this generic description as the Location Registry System of LRS for
short. LRS is a characterization of those sensorimotor systems that feature the following
properties: 1. an embedded representation for locations that can be available to a parallel
access 2. the capacity of binding active representation of specic type (objects of binding)
to location representations 3. the capacity of voluntary shift the focus of operation to
registry locations through which objects of interest can be accessed 4. the capacity of
serial planning for shifting the focus of operations between those locations.
109
Figure 6.3 is a schematic illustration of LRS. In LRS, in addition to two dierent types
of representation for location and objects a dedicated mechanism is needed for binding
active object representations and their associated location representation. This binding
mechanism is crucial for accessing objects through activated spatial representations. This
binding mechanism imposes a limitation in terms of number of items objects that can be
reliably bound to spatial addresses. This binding limitation imposes a limitation on the
capacity of LRS in terms on number of items that can be maintained simultaneously.
Maintenance of active representations and binding requires an active mechanism that
helps sustain activations without which items would be eliminated from LRS quickly.
This sustaining mechanism in fact helps control the content of working memory and is
in
uenced by a planner which determines which items need active maintenance.
In the meantime a selection mechanism which is in
uenced by the same planning
mechanism helps shift the target of active operations across the space representation.
This selection mechanism acts similar to a selective attention mechanism which helps
access to objects of interest through shift in spatial attention to the registry location of
the target object.
In spite of their strict capacity limitation, LRS systems play a crucial role in man-
agement of working memory for intellectual tasks. These systems provide the capacity of

exible variable binding which allows learning procedures based on access schemas which
point to addresses rather than content of working memory. Functioning of this vital fea-
ture is demonstrated in the following section where we apply a random access strategy
for performing a dual-counting task. So one may conceptualize the memory manipulation
in an LRS as a sequence of shift of attention to control the content of working memory.
A mechanism that can encode the sequence of shifts in attention with operational con-
tingencies can encode the procedure of memory manipulation needed for an intellectual
task.
We posit that instances of LRS support the capacity of random access to WM content
during intellectual tasks through their dedicated mechanism for associating an address
110
to objects working memory items. Thus we use above characterization of the Loca-
tion Registry Systems for postulating a random access schema during intellectual tasks.
These access schemas constitute operational features needed for our proposed Randomly
Accessible Working Memory (RAWM) component.
Our description of RAWM is rather abstract and includes a limited number of ad-
dressed memory space with some operations which are dened with close relationship
with what was described in LRS model. Above mentioned capacities of a Location Reg-
istry System can aord the following basic operations which may be used to describe
symbolic processing aspect of intellectual tasks at the symbolic level of our analysis and
using access schemas.
1. at : returns the active address or the address of current operation
2. shift to : changes the address or location of operations
3. bind or assign : registers a symbol or an object with current location
4. remove : unbinds or removes the object or symbol which is registered with current
location
5. fetch : returns the registered symbol or object at current location
This is not an exhaustive list of available operations aorded by every instance of LRS
at symbolic level, yet, achieving these functions qualies a system for being considered
as a Random Accessible Working Memory (RAWM) system.
The limitation of the Location Registry System in maximum number of objects that
can be registered with space results to a limitation in maximum number of items that can
be encoded and maintained in randomly accessible working memory systems. The actual
value for this maximum number depends on the instance of LRS which supports random
access schema. In the case of visual-spatial working memory system as the actual instance
of LRS for supporting random access working memory management this number is around
four items (Luck and Vogel, 1997; Todd and Marois, 2004). This number matches the
estimation of working memory capacity in a wide range of mental tasks (see (Cowan,
2001) as a comprehensive review of evidences for this estimation).
111
The notion of active address or the address of current operation denes the character-
istics of RAWM system in applying working memory management operations to a limited
number of items. This limitation is directly associated to limitation of selection mecha-
nism in LRS which in turn might be a manifestation selective attention mechanisms in
giving higher priority to one or a limited number of objects for processing at a moment.
A powerful feature of RAWM systems is related to their natural capacity for variable
binding to the address space which allows manipulation of memory content only based
upon schemas that utilize the address space for binding and retrieving items. We will
demonstrate the leverage that this characteristics provide for dynamic manipulation of
task relevant items in three dierent examples of address-based variable binding. We
will also discuss sever limitations of serially accessible working (SAWM) systems as the
alternative working memory management schema in variable binding. We will analyse a
serial access schemas for memory manipulation in same three intellectual/mental tasks
and we will show that at least in the case of one them a serial access schema is not
plausible.Thus we argue that at least with respect to the need for dynamic memory
manipulation through variable binding RAWM systems are the working memory systems
of choice. This argument is consistent with dierent evidences suggesting a correlation
between spatial cognition and the ability of performing intellectual task that require
dynamic working memory manipulation (McLean and Hitch, 1999; Reuhkala, 2001; Rudel
and Denckla, 1974) and explains involvement of posterior parietal cortex with known
spatial encoding features in those intellectual working memory tasks that feature working
memory manipulation (Koenigs et al., 2009; Osaka et al., 2007a).
6.5 Utilization of Random Access Working Memory for
Dual Counting
Imagine your are given the task of head-counting of adults and children in a party and you
take the challenge of counting both adults and children concurrently and mentally. As
112
your gaze shifts to a person in the living room, rst your visual system becomes engaged
in identifying whether the person at focus is an adult or a child. In the next step, one
of two running counts that matches the identied category should be increased by one.
The challenge is keeping track of two numbers and associating them to categories. A
spatial registry strategy is associating the existing count of adults n
a
to locationl
a
(e.g.,
left side in visual eld or under pinky nger of the left hand) and the existing count of
children n
c
to location l
c
(e.g., right side of visual eld or under index nger of the left
hand). Identifying the next child will trigger a shift of spatial attention to l
c
, to fetch the
current count of children. Once the increment operation is applied on the current count
the result will replace (by rst deletion and then insertion) the old count.
The symbolic schema can be conceptualized as a list of mappings of the current state
onto the next action. Here is a formal representation of an alternative SS for our head-
counting scenario.
AS
1
:fchild)shift to l
c
; adult)shift to l
a
g
AS
2
:fat l
c
)fetch n
c
; at l
a
)fetch n
a
g
AS
3
:fatl
c
&n
c
isretrieved)n
c
!n
c
+ 1 ; atl
a
&n
a
isretrieved)n
a
!n
a
+ 1g
AS
4
:fshift the gaze to next person & identify the categoryg
Each of these schemas may include other sub-schemas. For example n
c
! n
c
+ 1 in
AS
3
may include a sequence of operations over internal representation such as deletion
and insertion (binding to space).
Note how attention shifts might be used both for perception of the external world and
for selection of WM items, which, under the LRS hypothesis, might give rise to con
icts
in some situations, which in turn provides ways to test the hypothesis.
We will elaborate on a computational implementation of neural mechanisms of random
access schema for serial recall tasks in the next section of the paper. In our implementation
we use a population coding of space which features a continuous representation of address
113
space. In our implementation we will propose a mechanism which allows select registry
locations for forward and backward recall in this continuous representation of space.
6.6 Behavioral and Neural Correlates of the Randomly Accessible
Working Memory
Location Registry System describes functioning of a spatial shot-term memory supported
system which may help object-centered praxes in the physical environment. Due to the
adaptive relevance of interacting with objects in the environment one may nd many
instances of sensorimotor systems that satisfy our description of LRS, serving low-level
functions, even in lower animals. For example, in the context of eective navigation and
obstacle avoidance cats show the capacity of maintenance of visually collected information
about objects that may obstruct their hindlegs (McVea and Pearson, 2009). McVea et.
al showed that once step over an obstacle by forelegs, cats can maintain the information
about the obstacle and its location while it is out of their sight and use that information
up to several minutes for adjusting the height of hindlegs once they need to step over
the obstacle (McVea and Pearson, 2009; McVea et al., 2009). McVea and his colleagues
reported sustained neural activation in area number 5 in Posterior Parietal Cortex (PPC)
at a site homologue to the Superior Parietal Lobule (SPL) in humans (McVea et al., 2009).
Lajoie et. al showed that involvement of this region in maintenance of location of obstacles
is critical and lesioning this region of PPC will selectively impair organism's ability to
adjust the height of hindlegs over a short period delay (Lajoie et al., 2010). From an
evolutionary perspective it seems obvious that maintenance of information about objects
and their location would enable more complicated schemas such as ambush-like attack in
predators such as cats.
There are also situations in which sensory modes other than vision provide information
about behaviorally relevant objects and their locations in the environment. So one may
think of instances of LRS with auditory or haptic sensory elements.
114
However, armed with a spatial short-term memory system, a sophisticated system for
object recognition and a selective attention system, visual system in primates is the most
ubiquitous systems in supporting object-centered actions. Visual-spatial system has all
elements of a location registry system. A representation of space which helps register
visually perceived objects of interest with an internal representation of space accessible
to motor system. A working memory system with the capacity of maintaining limited
number of indexed objects (Luck and Vogel, 1997; Todd and Marois, 2004). A selective
attention mechanism which helps bring objects of interest into focus of attention both
for action and enhanced visual processing (Ballard et al., 1997). Yet, the visual system is
not a single faceted system and serves many dierent functions form object manipulation
to navigation and one may expect several visual-spatial instances of LRS functioning and
serving dierent rudimentary functions.
In the broad spectrum of functions of visual system the oculomotor system plays a
specic role. Its crucial function for vision in guiding the most relevant part of visual
scene into the foveal region of retina for a high-resolution processing has tied its function
to selective attention mechanisms so that the gaze location is considered as an indicator
of the overt attention and thus it is believed to work as a proxy for selective spatial
attention in the realm of visually guided tasks.
Studying the impact of intellectual tasks on eye movement behavior captured the
attention of researchers even before systematic study of eye movements in visual cognition
by Yarbus (Yarbus, 1967). Lorens and Darrow (Lorens and Darrow, 1962) reported
that performing double-digit multiplication `in head' and while sitting in the dark would
signicantly increase eye movement rates. Yet two more recent studies demonstrated that
those increased eye movements may indeed carry information about the the process of
mental intellectual tasks. In a study of eye movements during random number generation
(RNG) Loetscher et. al showed that eye positions can predict the magnitude of the
generated number for larger numbers correlated to right side and smaller numbers to the
left side (Loetscher et al., 2010).
115
Noori and Itti studied gaze shifts of their subjects during mental sorting of ve previ-
ously seen pseudo-random digits in front of a blank screen (Noori and Itti, 2011). They
noticed that gaze shifts are modulated along the orientation of initial presentation of
digits only during a mental sorting tasks and not during retaining those digits for a later
recall. They related this eect to possible use of short-term memory of previous location
of digits during the sorting process and not retaining process. Moreover they showed that
mental sorting of sequences with reveres relative order of digits results in gaze shift pat-
terns that are spatially symmetric. They concluded that their result shows that location
of digits in a visual working memory is operationally used for mentally sorting of digits
and symmetric gaze shifts during the mental sorting task indicates implication of visual
attention for shifting between those internally maintained locations.
Consistent with evidences of direct engagement of visual system in intellectual mental
tasks a number of studies conducted with dual-task paradigm have shown the impact
of engaging in a mental tasks on the function of visual system; from detecting expected
(Aky urek et al., 2007) or unexpected visual targets (Fougnie and Marois, 2007a) to visual
search (Han and Kim, 2004; He and McCarley, 2010).
A comparative review of the brain's functional neuroanatomy reveals neural evidence
in support of LRS hypothesis. In particular regions of the posterior parietal cortex (PPC)
with spatially enabled sensory and motor functions are consistently shown to be involved
along preforntal regions (PFC) in a range of high-level intellectual tasks that commonly
feature working memory manipulation. Notably, several neuroanatomical studies in both
in working memory and mathematical cognition have revealed the critical role that PPC
regions with strong spatial encoding features play in intellectual tasks (see (Olson and
Berryhill, 2009) for a review). Studying brain functions in other intellectual task such as
logical reasoning consistently have also shown engagement of PPC (Knau et al., 2000).
In particular two regions of interest in a many of these studies are the Superior Parietal
Lobule (SPL) and the Intraparietal Sulcus (IPS).
116
The SPL with a role in eye movement (Quintana and Fuster, 1993), spatial work-
ing memory (Jahanshahi et al., 2000; Osaka et al., 2007a), spatial attention (Colby and
Goldberg, 1999; Kanwisher and Wojciulik, 2000) and shift in spatial attention (Greenberg
et al., 2010; Nobre et al., 2004) is shown to be involved in intellectual tasks that feature
memory manipulation. A number of neuroimaging studies have revealed involvement of
SPL in executive working memory tasks which feature working memory manipulation. In
the context of studying neural correlates of intellectual tasks SPL is frequently referred to
as an active region. Involvement of SPL has been reported in deductive reasoning (Fang-
meier et al., 2006; Knau et al., 2003) , mental calculation with abacus (Hanakawa et al.,
2003), spatial imagery and deductive reasoning (Hanakawa et al., 2003), in numerical
comparison (Pesenti et al., 2000) mental arithmetic (Knops et al., 2009).
Furthermore, Koenig et. al reported that sustaining damage to SPL may result in
decit in intellectual working memory tasks that required manipulation of information
(Koenigs et al., 2009). Their nding establishes a crucial role of SPL in working memory
management during intellectual tasks.
In search for a shared underlying neural mechanism in high-level and low-level cogni-
tive functions some researchers have compared activation signals during both sensorimotor
and intellectual tasks in within-subject neuroimaging studies. SPL has been the region
of interest in some of these studies.
In one of these studies Knops et al. showed that BOLD signals induced by eye
movements in SPL are similar to that of mental arithmetic operations. They showed
this by using a classier trained for distinguishing the source of BOLD signals as the
rightward or the leftward saccades in an eye movement experiment to reliably classify the
mental operations as the addition or the subtraction operation based on their induced
BOLD signal in a dierent experiment. Therefore they argued that brain mechanisms
involved in eye movements are recruited for mental arithmetic. They propose this eect is
mediated by the role of shared spatial encoding mechanism with a role in representation
of numbers and in the meantime eye movements. Their account is evolutionary plausible
117
and yet restricted to representation of numbers only while the same region is shown to
be engaged in working memory tasks with non-numeric content.
From a dierent perspective and in search for neural basis of switching mechanisms,
in several within-subject fMRI studies Yantis and his colleagues found signicant cor-
relation between activation of medial SPL (mSPL) in switching spatial attention and
switching in dierent contexts including switching between objects of working memory
(Tamber-Rosenau et al., 2011), switching between non-spatial auditory attention (Shom-
stein and Yantis, 2006) and switching between categorization rules (Chiu and Yantis,
2009). Accordingly, they have proposed that SPL plays a role in a general purpose
switching mechanism. Whether mSPL is the seat of a general purpose switchboard lent
to dierent domains or a dedicated spatial attention shared across dierent domains,
ndings of Yantis and his colleagues suggest that mSPL plays a prominent role in shift
in spatial attention.
Evidences of engagement of PPC in intellectual tasks is also very well documented in
mathematical cognition literature. A host of neuoropsychological studies of acalculia and
dyscalculia frequently refer to co-existence of decit in spatial cognition and mathematical
cognition. Neuropsychological studies of mathematical decits suggest dierent roles for
dierent regions of PPC during mathematical cognition, with SPL in operational aspect
of the task and IPL in encoding numbers.
Moreover we argue the Intraparietal Sulcus (IPS) in IPL plays a role in binding mech-
anism. IPS is shown to be active in visual working memory and its activation level during
visual working memory tasks increases by the number of remembered items and reaches
to its maximum level around four items. This activation prole matches the capacity
of visual working memory in change detection paradigms. Interestingly in mathematical
cognition literature, damage to IPS is reported to decit in performing mental mathe-
matical operations. The evidence of object binding capacity of IPS in intellectual tasks
118
with number comes from neuroimaging studies that reported that counting visual ob-
jects engages IPS (Piazza et al., 2004; Sathian et al., 1999) whereas verbalizing numbers
without associating them to objects does not engage IPS (Zhou et al., 2006).
The LRS hypothesis gives an account for the correlation between strong relationship
between spatial cognition and ability in performing intellectual tasks tied at parietal sites
by assuming a utility role for space in random access to the working memory content
during intellectual tasks.
6.7 Serially Accessible Working Memory and Sensory-Articulatory
Circulative System
The capacity of serial access to the content of working memory in intellectual domain
is provided by assemblages of those sensorimotor systems which are capable of temporal
encoding of domain specic representations of tasks-relevant items. While the capacity
of simple temporal encoding of short-term information may enable serial access to the
working memory content, it might not be enough for fully explain critical operational
features such as variable binding, indenitely maintenance and
exible control over the
content of working memory for adding or removing items of interest. We suggest that it is
through coupling sensory and motor systems each of which with a short-term memory of
temporally augmented information that a system with enough
exibility for information
management in symbolic domain may emerge. The functional praxis of acquiring the
ability of communicating through an open-ended system of signals relies on a coupling of
sensory and motor systems which can exchange their short-term memory of temporally
encoded information. We use this functional praxis to explain a generic utility senso-
rimotor system capable of supporting serial access to working memory in intellectual
domain.
In describing our generic model we even relax the communication constraint and focus
on acquisition of the ability of articulation or production of arbitrary and yet syntactically
119
constrained sequences of specic type of signals of a shared lexicon codebase. Acquisition
of human language is probably the most ubiquitous example of such functional praxis
which serves communication between two parties based on the ability of reproduction or
articulation of similarly perceived signals in dierent modes (spoken or sign language).
Less prevalent examples which do not necessarily serve a communication purpose and yet
rely on a close relationship between sensory and motor systems for encoding open ended
set of codebase signals can be found in skilful musicians with strong coordination between
their hearing and instrument playing skills.
Acquiring the skill of generating arbitrary and yet syntactically constrained sequences
of lexicons relies on a dedicated system which allows learning sensory signal patterns by
example and forming representations at the perception end as well as training the artic-
ulation apparatus and developing motor representation at the articulation end. So, for
every lexicon of the associated codebase two corresponding representations in sensory and
motor system form. Formation of articulatory representation is the result of converging
articulation trials and crystallization of motor codes. This can be achieved in a loop
between sensory and articulatory systems.
Figure 6.4 depicts our proposal for a generic system comprising a pair of coupled
sensory and motor system in a closed loop. We refer to this generic system as Sensory-
Articulatory Circulative System or SACS for short. In addition to being a closed loop,
this system enjoys the support of some low-level and high-level signal processing systems
that allow processing
owing sensory or motor codes of the loop for integrating the loop
into high-level processes. So, main characteristics of a SACS system may be described
as follow: 1. the ability of temporal encoding at both sensory and motor ends 2. the
capacity of buering signals for later parsing/segmentation of signals into lexicons 3. a
mechanism for automatic conversion of sensory code to motor code and visa versa 4.
mechanisms for two-way interplay with a high-level signal processing system.
Both sensory and motor systems in SACS rely on their own dedicated temporary and
fast decaying buer which maintains a continues sequence of sensory or motor codes.
120
The content of these buers can be parsed into segments of sensory or motor codes.
Segment parsing of a sequence of sensory or motor code is the result of forming paired
representations in the paired sensory and articulatory systems. Each of sensory and
articulatory systems keeps a continuous trace of codes. Segmentation of this continues
trace is based on recognition of patterns associated to the individual's set of learned
lexicons and is subject to change as the individual acquires new lexicons.
The system bootstraps by perceptual learning of some basic components or lexicons.
Then training of articulatory motor system for articulation of those basic lexicons relies
on a feedback mechanism which allows comparing recently generated signals with already
stored sensory patterns and guides the training process of the motor articulatory system.
The capacity of buering sensory traces and parsing the content of buer to already
learned lexicons allows co-learning of more complex combination of lexicons in sensory
and articulatory systems. This in turn allows a more robust and fast and automatic con-
version of xed combination lexicons between sensory and articulatory systems. With the
support of buering and automatic parsing mechanisms, over the time, the complexity
of the codebase is folded which translates to a form of open-endedness in storing sensory
and articulatory codes of from combination of previous codes. This capacity of increas-
ing the complexity of codebase of communication signals is what distinguishes human
communication system with other organisms that remain relatively stable after a critical
period.
The act of parsing involves identifying a segment of either sensory or articulatory
code which maps onto a lexicon representation. Once segmentation is done the seg-
mented sensory or motor trace is automatically translated to the other form (sensory
segment translates to motor code and visa versa). Segmentation describes the process
through which the content of the buers are emptied. The process of segmentation results
in parsing a continuous buered code into separate lexicon representations and results
in
ushing out the content of each of buers. In contrast lexicon representations may
be spliced into a continuous motor or sensory code lling the buers. This buering
121
mechanism allows storing segments that are practically a sequence of already encoded
signals. This mechanism enables training the sensory and articulatory for independent
representation of longer pieces of codes. Imagine developing independent representation
of words as combination of phonemes as one level of folding and morphemes from words
as another level of folding. We argue that without a buering mechanism formation of
sensory and articulator representation of folded or combined signals is not possible. So,
the buering mechanism in addition to conversion mechanisms is crucial for functioning
of SACS as a system capable of learning open-ended system of codes.
Conversion of sensory code to motor code and motor code to sensory code is done
under two dierent functional and neural principles that are necessary to close the loop.
In one end sensory signals automatically translate to motor signals. This conversion
satises the denition of mirror systems (Fabbri-Destro and Rizzolatti, 2008). Mirror
systems enables imitations of action by observation of signals emitted by another a tutor
(the right end of the loop in Figure 6.4). Functioning of such system is crucial in imitation
is not limited to communication or language. Examples of such imitation can be seen in
learning manual skills in tool making. Sensory to motor conversion is done automatically
and the result is buered in the articulatory buer for motor execution.
In the meantime the organism needs to utilize a mechanism to distinguish self-emitted
signal/actions. This is crucial for stabilizing acquisition of the signalling by eliminating
self emitted signals as the material of perceptual learning. Moreover, being able to distin-
guish self-emitted signals is crucial for correction of motor code for learning articulation
of signals by repeating successful patterns and eliminating failed patterns. This requires a
system that can predict or project motor codes of articulation system onto the perceptual
system. This sensory projection of motor codes closes the loop on the other side (the
left end of the loop in Figure 6.4). This conversion of motor code to anticipatory sensory
signals in particular is crucial for covert articulation. As the result covert articulation can
rell the sensory buer with segments codes of sensory signals. A viable neural mech-
anism for supporting this mechanism is referred to as forward internal model. Forward
122
internal models can also be used for fast optimal articulation which relies on implemen-
tation of exact motor codes that can guarantee fast and accurate motor movement that
can produce the signal at the articulation end.
Phonological loop in humans which supports acquisition of human spoken language
with and auditory and a vocal component as sensory and motor components is the most
ubiquitous instance of such a system, yet sign language which relies on two dierent
sensory and motor ends also relies on its own instance of SACS for acquisition of the
language (Wilson and Emmorey, 1997). As it was previously mentioned, acquisition of
other non-lingual systems of signals such as what is associated to the skill of playing a
particular musical instrument may rely on a dedicated instance of SACS. However, what
gives an advantage to lingual SACS systems compared to non-lingual ones is the avail-
ability of a rich set of designated representations in the associated language. This rich set
of representation allows employing language-supported SACS systems to a wide range of
intellectual tasks. Yet, we argue that other novel instances of SACS systems (such as au-
ditory and hand movement system in professional piano players or professional stenotype
operators ) might be utilized if representations for intellectual tasks are developed.
A typical SACS with designated symbolic representations in its sensory and motor
systems might be utilized for serial access to working memory content which is described
in our proposed SAWM access schema. We describe specications of this type of access
schema with respect to functioning of a supporting SACS. In our model of SAWM we
use a single representation for working memory items. Thus we conceptualize each item
encoded in a SAWM system with only one symbolic code segment. The concept of symbolic
code segment which refers to both sensory and articulatory code segment is the basic
concept by which information encoding and access schema in serially accessible working
memory is modelled. In a similar way two dierent sensory and motor buers in SACS
are conceptualized with a single buer with a limited length for temporary storage of a
continuum of symbolic segment codes.
123
The length of a symbolic code segment is a fundamental attribution of it which de-
termines the maximum amount of capacity that a certain segment may occupy from the
buer. The decaying aspect of the buer in SAWM forces a segment out of the buer
and thus not only the length of a segment but also the age of the segment is important
in determining the actual capacity that a segment may occupy.
Based upon above schematic description of interaction between sensory and motor
systems we oer the following basic access schema for SACS supported serially accessible
working memory systems.
8
>
>
>
>
>
>
<
>
>
>
>
>
>
:
i: feth segment
ii:
0
process()?(process())
iii: push segment(
0
)
Where rst, i, ii and iii form a chain or sequence of operations which always ap-
pear in the order, second, and
0
are symbolic segments which may consist of other
spliced segments, third, is a function that denotes splicing two segments and, forth,
?(process()) is a void segments which is as long as process() may take to process.
Figure 6.6 shows three episodes of serial access schema in maintaining of ve symbolic
segment codes by employing this process.
In the simplest functioning mode of SAWM process() =I
0
whereI
0
is an instant
identity function with splicing a zero length delay to the end of . This mode describes
the function of SAWM in maintaining a sequence indenitely through active rehearsing.
Another trivial yet important operation is described by process() =?
1
where?
1
denotes a long pause. This operation results in erasing the whole content of SAWM. These
basic operations also allow removing one item from the buer by just rst identifying the
target segment and replacing it with a null segment. This removal or replacement of a
segment requires non-trivial involvement of the high-level signal processing system.
The functioning of SAWM can be described as continuous episodes of these basic
cyclic operations. The temporal aspect of this access mode which is captured by splicing
124
function and orderly management of segments is an inseparable part of operations in this
mode.
The reader may nd it useful to imagine segment codes in SAWM as a written seg-
ments on the surface of drum rotating constantly along its central axis (see Figure 6.5
as the reference). Two heads next to each other are responsible for reading and writing
symbolic codes. Segmentation of read codes is facilitated by pattern recognition unit
that supports the reading head. As a pattern is recognized the segment is wiped o the
surface and the code is transferred to a processing unit where depending on the rule of
processing will return a segment to writing head.
It is notable that the dierence between RAWM and SAWM systems is not limited
to only the access order but also the encoding nature of symbolic information is a major
factor in the dynamics of access and capacities and limitations of the access mode. As
an example, segmentation of sensory or articulatory codes in SACS allows a
exible
information encoding capacity in SAWM while RAWM systems has a xed capacity for
maintenance of information. As the result the capacity of SAWM in terms of number
of item might be increased by shortening the symbolic sensory or articulatory code. In
this sense one may achieve maintenance of items beyond the capacity of RAWM systems
through strategic shortening of symbolic segment codes.
One may even argue that the symbolic sensory or articulatory codes do not necessarily
have the same segment length every time they are articulated and their segment length
may vary in two dierent articulation. This segment length eect imposes an uncertainty
in estimation of the capacity of SAWM systems in maintenance of information in terms
of number of items.
Another aspect of SAWM systems compared to RAWM systems is related to relative
advantage of RAWM systems in variable binding. While the address space provides
a natural and explicit way for variable binding the same feature is missing in SAWM
systems. However the capacity of SAWM in encoding relative order of segments can be
utilized for an implicit variable binding schema. In this case a specic segment in the
125
buer { let's say
m
{ may proceed and mark a variable segment { let's say
v
{ in the
buer. So in a continues rehearsing
m
always proceeds
v
. We denote this sequences by
the following notation:
h
m

v
i
Identifying and processing
v
as the symbolic code of the variable is made possible by
rst identifying the marker segment (
m
) and then fetching and processing its following
segment (
v
) . We refer to this strategy of variable binding as sequential variable binding.
We will explain a practical use of the sequential variable binding in the next section and
in the context of performing a dual-counting task.
A disadvantage of using SWAM for variable binding is related to the insertion or
splicing void segments associated to processing of both marker and variable segments (or
?(process(
m
)) and ?(process(
v
)) ). These additional void segments may gradually
add up and occupy the capacity of SWAM if a housekeeping operation is not regularly
performed. This housekeeping operation is costly and may interfere with the main process
that requires the variable bonding. Yet, this variable binding capacity adds substantial

exibility to the SWAM system so that makes it a dynamic working memory system
rather than a short-term memory of serial orders.
It is very important to distinguish between a serially accessible working memory
system and short-term memory of serial order. The latter one is crucial for the former
one but it can not account for a complex and
exible schema for controlling information.
In fact, the proposed SACS schema explains how a combination of two systems with
endowed capacity of short-term encoding of serial order may give rise to a powerful
working memory system. In particular variable binding and the ability of maintenance
of information for an indenite period of time without relying on long-term memory
resources are two additional functionality that emerge from this combination.
indenite maintenance of information is a function that can be gained from utiliza-
tion of long-term memory resources too. Yet, there are important dierences between
utilization of SAWM (or SACS) and long-term memory. First, SAWM requires active
126
rehearsal without which the information will be lost quickly. Second, the capacity of
SAWM is limited while capacity of long-term memory is unlimited. Yet, as long as the
length of segment codes allows number of items in SAWM is independent of the semantic
information of segments while the eciency long-term memory resources depends on the
semantic relationship between items of the list. Moreover, long-term memory resources
are very slow compare to SAWM resources and highly limited in terms of variable binding.
The following section demonstrates how a SAWM system can be utilized for the double
counting scenario.
Finally it is important to note that the presence of SACS system does not guarantee
the presence of SAWM in a particular individual. While SACS re
ects the intrinsic
capacity for acquisition of language , presence of SAWM re
ects development of strategic
use of resources of SACS for working memory management in a context not directly
related to language acquisition. As we discussed structure of SACS can aord only for
serial access and it is the matter of strategic choice to reuse them in a symbolic context
and for manipulation of information in intellectual tasks. In fact, evidences from working
memory studies in normal children suggest that children of younger than 7 years of
age normally do not use phonological strategies (Henry, 1991; Vygotskij, 2012). This
is despite the fact that at those ages lingual resources of SACS are fully developed and
available. We entertain this hypothesis that formation of phonological SAWM strategies
is induced by schooling and school tasks. In our discussion we propose two accounts
which relate development of phonological SAWM strategies to schooling. In one account
we suggest that learning to read can induce or boos phonological SAWM strategies, the
other account relates it to class management policies and forcing student to prepare their
answers to classroom questions before fast and accurate response.
127
6.8 Utilization of Serially Accessible Working Memory for
Dual Counting
Once again we will consider the task of mental head-counting of adults and children in a
party. This time we devise an access schema based on availability of a serially accessible
memory. We devise our strategy based on a sequential variable binding schema for two
variables; one for number of children and the other one for number of adults. Employing
this strategy requires at least four symbolic segment codes: two consecutive segments for
marking and keeping children count and two consecutive segments for marking and keep-
ing adults count. For the sake of simplication of our notations we assume an phonological
SACS for implementation of our SAWM. We denote the sequence of relevant segments in
the maintenance mode as follow:
h ( ["y@N] ) ('(n
c
)) ( [" ol] ) ('(n
a
)) i
Where ["y@N] and [" ol] are respectively phoneme segment representations for `young'
and `old', is a function which maps phoneme to symbolic segment code and '(n
c
) and
'(n
a
) respectively represent phonetic representation of n
c
and n
a
.
We also dene two functions using basic operations to simplify our notation. One of
functions denes
agging a segment as the result of identifying a marked segment and the
other one for applying an increment operation on the
agged segment. Flagging can be
interpreted as switching to a transient mental state which cues preparation for increment
process on the parsed value of the next segment. The increment function is applied only
when the segment is
agged. During increment operation the numbern associated to
is retrieved and then
0
associated ton + 1 is pushed into the buer and hence replaces
in previous episode of the SAWM buer. Finally the increment operation should return
the mental state back to normal state (un
agged state). The following access schema
uses these functions to dene a strategy for SAWM-based dual counting.
AS
1
:fchild)flag( ["y@N] ); adult)flag( [" ol] )g
128
AS
2
:flagged )
8
>
>
>
>
>
>
<
>
>
>
>
>
>
:
i: fetch segment
ii:
0
('(1 +'
1
())) & unflag
iii: push in(
0
)
Where '
1
denotes the reverse of ' and identies the integer value associated to a
symbolic code segment.
Just for simplication of our notation we dropped the splicing operation from these
schema formulation, yet, it is important to note that mental operations such as encoding
a number from the code segment and then increasing are not instant operations and they
insert void spaces into the buer. Hence, this strategy requires housekeeping operations
for removing void segments from the buer which may interfere with the counting process.
In the case of utilization of a phonological system for deploying this strategy choos-
ing phonological symbolic representative of counts plays a role; phonologically confusing
phonological proxies may result mistakes in
agging segments by confusing segments.
On the other hand this strategy provides some room for optimization by 1. choosing
shorter word representation for variables (e.g. using [" o l] instead of [@-"d@lt] and ["y@N] or
even ["y@] instead of ["tSIldr@n]) 2. choosing less phonologically confusing and more easily
separable variable representations.
6.9 Neural and Behavioral Correlates of Serially Accessible
Working Memory
Humans do not have a monopoly over the ability of learning to signal for communication
through imitation. Learning and maintaining the ability of producing vocal signals for
communication in birds and human bear striking similarities. Similar to human children,
in some bird species, young birds learn to sing by imitating adult tutors within a critical
time window. However, unlike birdsong that seem to be limited both in terms of number
of syllabus and variation of combination of syllabuses, in human, the speech system
and the language system to which it serves are both diverse in terms of components
129
(syllabus, lexicons and vocabulary), and open-ended in terms combination of lexicons
and communication goals. Moreover, compare to birdsong human speech system retains
a part of its
exibility and in spite of critical changes in terms of limitation in adapting
and changing low-level speech features, a part of speech system may still be available for
acquisition of new lexicons and vocabulary or even new language during the adulthood
period. We discussed that this human specic feature relies on relatively large buering
capacity of sensory and articulatory systems in human speech system.
In both humans and birds, acquisition and maintaining the ability of emitting sound
signals relies on a feedback loop. During acquisition of singing, similar to children, young
birds bootstrap their learning by perceptual (auditory) learning of songs emitted by adult
birds followed by a stage of motor learning (Brainard and Doupe, 2000; Tchernichovski
et al., 2001). These learned patterns are used to train the vocalization motor system
(Kuhl, 1994).
It has been also shown that stability of adult vocal behavior in humans and birds
rely on the feedback of articulatory eort to sensory system. In the case of post-lingual
acquired deafness often patients experience a deterioration of some aspects of speech
production (Waldstein, 1990). In a similar way adult birds may lose their ability to
articulate their song when the feedback is perturbed for a long period (Leonardo and
Konishi, 1999) or blocked (Nordeen and Nordeen, 1992). It has been also established
that adult human (Jones and Munhall, 2000) and adult birds use auditory feedback to
compensate for perturbed the sensory feedback of articulated signal (Sober and Brainard,
2009).
Sensory feedback of motor action is a crucial part of acquisition of signalling in other
modes too. For example in sign language, acquisition of signing is equivalent to motor
leaning through observation. While a part of this process relies on a mirror system which
allows a common representation between tutor and learner, like every motor leaning task
articulation of signs relies on visual and proprioceptive feedback without which motor
imitation may fail (Scott, 2004; Shadmehr et al., 2010).
130
As the result of motor-learning for articulation in association with learned perceptual
patterns a mapping mechanism between sensory representation and motor representation
is established. Some researchers even posit that this sensory-motor mapping is an integral
part of representation of speech or in general language representation. Recent evidences
support that listening to spoken words selectively activates those muscles that are involved
in articulation of associated sounds. For example recording MEPs signals after TMS of
left hemisphere from the tongue muscles shows selective activation to listening of words
whose articulation involves tongue muscles.
This automatic mapping of sensory input to associated motor representation which
is activated during regenerating those sensory traces by the organism is ascribed to the
functioning of the mirror systems (Fabbri-Destro and Rizzolatti, 2008; Rizzolatti and
Craighero, 2004). Mirror systems were rst discovered when researchers observed acti-
vation of neurons in monkey's premotor cortex in response to observing grasping objects
independent of whether monkey was performing the action or the experimenter (Gallese
et al., 1996; Rizzolatti et al., 1996).
Some researchers have entertained this idea that language is evolved from a gestural
form, enabled by a visual-motor mirror system similar to what has been observed in
monkey's premotor cortex (Arbib, 2010,?, 2012). This hypothesis has received support
from evidences of presence of a mirror system in human brain situated in Broca's area
(Arbib, 2010; Rizzolatti and Arbib, 1998).
With regard to auditory and vocal articulatory loop in birds, recently evidences of
mirror neurons in forebrain of birds are reported. Those mirror neurons respond to
vocalization of a specic note patterns by the organism and when listening to other birds
generating the same sequence of notes (Prather et al., 2008).
Covert articulation in SACS relies on conversion of motor codes to sensory trajectories.
The importance of predicting sensory trajectories of motor actions has been established
in the motor control literature and is known as forward internal models (Kawato, 1999).
Forward internal models are speculated to play an important role in inverse internal
131
models which are discussed to be crucial for rapid and optimal control of motor movements
based on the desired trajectory. It has been argued that feedback control systems can
not account for initiating rapid motor actions and thus a reverse internal model guides
the action based on prediction of motor actions in forward internal models. Although
the cerebellum, because of its plasticity has been speculated to be a major player in
supporting neural substrates of internal models, evidences of involvement of cerebral
regions, especially the parietal operculum and the ventral supramarginal gyrus in subvocal
articulation have been reported (Price, 2010).
As we discussed previously cancelling sensory consequences of own articulation is
theoretically important in stabilizing sensory and motor representation. Forward models
might play a role in this regard. For example, it is known that self-tickling is sensed
dierently compared to tickles by an external source (Blakemore et al., 1998). The
source of this eect is attributed to the cancellation of eects of own action as the result
of forward internal models (Blakemore et al., 1998; Stout and Chaminade, 2012).
Evidence for separate pathways for overt and covert articulations was presented in a
study of dysarthic patients with impaired ability to speak as the result of acquired brain
damage and yet spared language ability (Baddeley and Wilson, 1985). These subjects
show normal memory span on written material and also demonstrate an eect of word
length suggesting that in spite of damaged overt articulatory system they have retained
an internal articulation ability for covert articulation.
Functional prole of an auditory-vocal system equipped with a limited temporary
buer as described above, is very similar to what is explored by cognitive psychologists
in investigating a system that enables maintenance of task material during WM tasks
especially in verbal mode. The role of a phonological system as a crucial component
of human working memory was underscored in Baddeley and Hitches original model
(Baddeley and Hitch, 1974) where it was referred to as the phonological loop.
Controlling for phonological features of WM material such as confusing eect of phono-
logical similarity of items (Baddeley, 1966; Conrad, 1964) or impact of irrelevant speech
132
(Baddeley and Salam e, 1986; Jones and Macken, 1993; Salam e and Baddeley, 1986) on
the capacity of maintenance of a list of verbal material revealed the role of a phonologi-
cal component in short-term serial recall (Baddeley et al., 1984). Also some researchers
presented evidences of engagement of a subvocal or covert articulatory mechanism by
showing that suppression of the articulatory system negatively aects the short-term
memory capacity (Murray, 1968). The debate over the phonological or the articulatory
nature of short-term memory was ended by Baddely and Hitch (Baddeley and Hitch,
1974) when they proposed the idea of a phonological loop that utilizes both phonological
and articulatory resources to store information. It was after presentation of the three-
component model that Baddeley and Buchanans presented evidence in support of limited
buer of the phonological loop by showing the eect of word length on the capacity of
working memory (Baddeley et al., 1975).
Studies of verbal working memory in sign language users have provided the oppor-
tunity to look at mechanisms of verbal (or language supported) working memory in a
mode dierent from the phonological mode. This in turn has been very informative in
isolating functional mechanisms of components of working memory in two modes. First
it was crucial to see whether there is any parallel between a phonological-based VWM
and a sign-based verbal WM. In several studies on American Sign Language (ASL) users
Wilson and Emmorey demonstrated a strong parallel between two modes. They estab-
lished the in
uence of modal features of the sign language on working memory span in
ASL users with same functional features as phonological case of verbal memory. They
could successfully replicate signature eects of the phonological loop in VWM of ASL
users. In particular, irrelevant visual input eect (Wilson and Emmorey, 2003) and sign
confusing eect similar to the eects of non-relevant speech and confusing words suggest
a role for visual-spatial input as sensory element of VWM in ASL users (Wilson and
Emmorey, 1997). Negative eect of articulatory suppression in ASL users by tapping
ngers suggests a role for an articulatory element similar to phonological loop (Wilson
and Emmorey, 1997) and nally the eect of sign length which similar to word length
133
eect in phonological loop suggests a role for a limited capacity buer (Wilson and Em-
morey, 1998). Emmorey and Wilson's ndings clearly suggests functioning of a sensory
system along an articulatory system supported by a limited capacity buer very similar
to phonological loop. Based on these evidences Wilson advocated for sensorimotor coding
in working memory (Wilson, 2001). Yet, Wilson's proposal is rst, focused on the case
of only verbal working memory which is equivalent to our proposed SACS.
Several neuropsychological and neuroimaging studies have been able to establish a
phonological and an articulatory component in verbal phonological memory. This studies
often employ baseline tasks that are assumed to implicate only one of two components
of the phonological loop. For example in the rst neuroimaging study that claimed to
distinguish these two components, a rhyming task on written words was used as a tasks
that relies only on articulatory component of the phonological loop. The dierence be-
tween activation during the rhyming task and retaining a list of words for later recall was
attributed to contribution of the phonological buer and common regions hypothesized
to be engaged in articulation (Paulesu et al., 1993). Accordingly in this study the left
supramarginal gyrus was reported as neural substrate of phonological buer and Broca's
area as the neural substrate of articulatory component.
Implication of Broca's area is also reported in the case of sign-base verbal WM task
(R onnberg et al., 2004). In a within-subjects PET study of Swedish Sign Language
interpretors similarities across language modalities was reported in Broca's area. Yet,
interestingly, in the same study, bilateral parietal activation patterns similar to that of
visouspatial tasks appeared.
cite poizner1990biological and mention the liberality of the brain in sign language.
Baddely and his colleagues also noted the relationship between phonological working
memory and acquisition of language. They rst reported the case of acquired decit in
verbal WM in subject P.V. in their investigation of possible relationships between long-
term memory and short-term memory (Baddeley et al., 1988). They noticed that subject
P.V. with a signicant decit in verbal WM is able to establish long term memory by
134
learning pairs of previously known words yet she was unable to learn new words. Papango
and Vallar showed that in normal subjects, phonological similarity and word length aects
which are signature of phonological working memory aect learning association between
only non-words (Papagno and Vallar, 1992).
paring previously learned word are The relationship between phonological working
memory and acquisition of language was also discovered in studies with the goal of inves-
tigating possible roles of working memory in language processing (Baddeley, 2003a).
135
Neural Schemas
Sensorimotor
Schemas
Operational
Schemas
Access Schemas
Execution
Schema
Level of Analysis
Symbolic Level
Generic Sensorimotor Level
generic sensorimotor mechanims
access modes
sensorimotor mechanims
Concrete Sensorimotor Level
Neural Level
neural mechanims
Constraint Type
component type
Neural
Observations
Biological
Observations
Evolutionary
History
Behavioral
Observations
The Symbolic Interface
Levels of analysis
Figure 6.1: Pseudo-tabular depiction of the Symbolic Interface, our proposed four-level frame-
work for describing human behavior in the realm of intellectual tasks and supporting brain and
neural mechanisms. In this pseudo-table neither rows nor columns are completely independent.
Components and mechanisms in this framework are concepts that are related to each other. For
example, a solid arrow between two components denotes conceptual dependency of the component
at the start of the arrow on the component at the end of arrow. Each row is indeed a place holder
for more detailed and probably schema-theoretic description of either a behavior or a mechanism.
At the lowest row (level), the behavior is described in terms of symbolic processing primitives
conforming with psychologists' descriptions of human behavior in the domain of intellectual sym-
bolic tasks. At the top, neural mechanisms with direct reference to brain regions or networks are
presented. Between these two end levels (rows) are sensorimotor systems: one level for generic
and another one for actual or concrete sensorimotor system instances. A generic sensorimotor
system describes general characteristics of those sensorimotor systems that support components
at the symbolic level (access modes in our model). A generic sensorimotor system can also be
viewed as a general description of those actual sensorimotor systems that may potentially ll
each other's role in supporting access modes as components of the symbolic level. In addition
to between-level constraints, formalisms of each level have their own constraints (shown in left
column). The direct constraint on the symbolic level is determined by behavioral observations. At
the second level the constraint is set by the biological context conforming with evolutionary history
of the organism. At the third level biological observations determine limitations on functioning
of every instance of sensorimotor systems. At the highest level constrains are set by neurobiolo-
gists' observations of neuro-anatomical functioning of the brain. This four-level analysis results
in a hierarchical schema-theoretic description of the functioning of sensorimotor utility systems in
supporting memory management for intellectual tasks. One may ll this structure with dierent
types of components towards a formal description of a neural basis of information management
in intellectual domain. Our SWMS model gives an example of such eort (See Figure 6.2).
136
RAWM
SAWM
Concrete
Instances
Generic
Systems
Location Registry
System
(LRS)
Sensory-Articulatory
Circulative System
(SACS)
Neural Level
Symbolic Working Memory System
Mixed
Strategies
Multi-system
Assemblage
LRS2
LRS1
SACS 1
SACS 2
Symbolic Level
Sensorimotor
Levels
Figure 6.2: A depiction of the Symbolic Working Memory System (SWMS) as it roughly lls
four levels of description in the Symbolic Interface (SI) (see Figure 6.1). Solid arrows in this
gure denote dependency of the component at the start of arrow to the component at the end
of arrow. Dashed arrows denote possible contribution of the component at the start of arrow
to the component at the end of arrow. At the symbolic level, two pure access modes (random
access mode and serial access mode) plus a mixed mode are shown separately. At the generic
level, Location Registry Systems (LRS) and Sensory-Articulatory Circulative Systems (SACS)
respectively support random access modes (represented by RAWM) and serial access modes (rep-
resented by SAWM). LRS is specialized in utilizing space in supporting object-centered actions
by maintaining location and identity of a few task-relevant objects. In comparison, SACS are
specialized in utilizing time in supporting acquisition of communication abilities through emitting
signals. At the instance or concrete level, one or more instances of LRS and SACS may exist.
The most prevalent instances in normal subjects are the oculomotor system and the phonological
system as instances of LRS and SACS. Yet, somato-motor or audio-motor in blind subjects, and
visual-motor system in deaf subjects can serve the same function for LRS and SACS. Variations
of instances in dierent individuals is partially responsible for individual dierences in performing
mental tasks. Another source of individual dierences is related to the strategy of using utility
systems. Some strategies may optimize usage of SWMS resources.
137
Attentional
Selection
Sensory
Input
m S P L
P F C
d lP F C
Object
Representation
Location
Representation
Binding
P T C
I P S
S P L
Sustainer
Motor
Output
Motor Control
Sequencer
Location Registry System (LRS)
T-D
T-D
T-D
T-D: Top-Down Influences
Figure 6.3: A depiction of the Location Registry System (LRS) augmented with information
about candidate neural regions associated with a visual-spatial instance of such a system. LRS is
a generic utility system that supports random access to the working memory content. The original
functional praxis of LRS is maintenance of locational information of a few objects which are most
relevant to the behavior and thus are most likely to be the target of an immediate action { e.g.
physical manipulation or eye movement. LRS receive sensory input for encoding location and
object identities. The source of information for location and identity might be dierent sensory
systems yet a maintained item needs to register with a location. This registry is supported by
a binding mechanism which comes with a limited capacity. An active mechanism is required
to sustain activation of the item representation, the location representation and the registry. A
mechanism for internal shift of attention to locations allows switching between registry locations
and accessing to objects of memory for required operations. This shift may result to triggering an
action in the outside world (shifting gaze to the associate location) and thus interact with motor
system. A sequencer is indeed a place holder for a program which determines contingencies for
shift of attention to required locations for retrieving, binding, or deleting items from the registry
system. This program might be generated on-line as the state of the mind changes. (PTC:
Posterior Temporal Cortex, IPS: Interaparietal Sulcus, SPL: Superior Parietal Lobule mSPL:
medial SPL, PFC: Prefrontal Cortex, dlPFC: dorsal-lateral PFC)
138
Top-down
influences
Sensory Buffer
Articulatory Buffer
Sensory
Segmentation
Mirror-Neuron
Echo System
Sensory
Input
High-Level
Language
processing
Articulatory
Output
Internal Sensory
Projection
Articulatory
Segmentation
Covert
Articulation
Overt
Articulation
Figure 6.4: A depiction of the Sensory-Articulatory Circulative System (SACS). SACS is a generic
sensorimotor utility system which supports serial access to working memory (SAWM) content.
The functional praxis associated to SACS is acquisition of the skill of communicating through
emitting signals of a codebase. This skill requires training a dedicated articulatory/motor system
in coordination with a dedicated perception/sensory system. Acquisition of a language in dierent
forms (sign or spoken language) or learning to generate any type of open ended set of signals from
a codebase of lexicons are instances of this praxis which rely on dierent combinations of sensory
and motor systems. This system relies on conversion of sensory codes of a sensory buer to motor
codes, and on conversion of motor codes of a motor buer to sensory codes. Conversion of sensory
code relies on segmentation of a piece of sensory code from the sensory buer and converting
it to motor code via a Echo-Mirror system. Segmentation takes in
uence from high-level signal
processing units. This conversion is supported by a specialized mirror system which automatically
converts sensory codes (Fabbri-Destro and Rizzolatti, 2008; Fadiga et al., 2002) to motor codes.
At the other end (left side of the loop), a segmented piece of motor code is converted to sensory
code through an internal sensory projection of motor action. Two dierent modes are possible for
conversion of motor code to sensory code: overt articulation and covert articulation.
139
Spliceing
Head
Segmenting
Head
σ
σ'
process
Pattern
Recognition
Unit
Figure 6.5: Conceptualizing the sequence of operations in a serially accessible working memory
(SAWM). A rotating drum with two heads for segmentation and splicing read and write on the
surface of the drum. Segmentation is supported by a pattern recognition unit. Once a segment is
identied to match a known pattern a copy of code segment is sent to a processing unit. Depending
on the the type of loaded process another segment is sent to splicing head to be written on the
drum. The drum is rotating with a constant speed and the longer the process takes the less
surface is available for writing process codes.
140
i. Segmentation
iii. Splicing
σ' = I
0
(σ)
ii. Process
i. Segmentation
iii. Splicing
σ' = I
0
(σ)
ii. Process
Figure 6.6: Two episodes of segmentation, processing and splicing of ve symbolic segment
codes in a serially accessible working memory system, during the task of maintaining a list of
four characters: v,r,c,k. The segmentation process is shown at the far right section of the buer
and splicing is shown at the far left. Each episode includes i. segmentation ii. processing and
iii. splicing a symbolic segment code to the content of the buer. In this example the operation
at each stage is a noisy identity process which takes in the parsed segment and splices it to the
buer with no delay. Note that there is only one piece of symbolic code ahead of the segmentation
process. Although this piece of code in this example is made up of splicing ve pieces of symbolic
code, it is treated as a continuous piece in the buer. During the segmentation process a segment
from the end of this continuous buer is read and removed from the buer. After processing, the
processed code segment is spliced to the content of the buer. Segmentation may take in
uence
from high-level processes. Depending on such high-level in
uences, the noisy identity process may
turn a dierent confused code other than original segmented code (e.g., because of priming or
intrusion from another episode). Yet, in this scenario it is likely that the mistaken code may have
very similar modal signatures (e.g., in the verbal-phonological case b can be mistakenly changed
to v during the I
0
process because of phonetic similarity of two items).
141
Chapter 7
The Case of Immediate Serial Recall from SWMS
Perspective
7.1 Bidirectional recall a challenge for computational models
Our goal in this section is demonstrating the power of our symbolic interface method in
explanation and prediction of neural basis of behavior with respect to memory manage-
ment requirements of a mental task. This will bring the dierence between our proposal
and other models of working memory into a sharper focus by showing practical benets of
proposed access-based analysis and executive schemas in nally addressing the source of
drastic dierences between recalling a list of items in forward or backward order. Despite
of a wide range of evidences that show that forward and backward recall rely on dierent
strategies, mechanisms and resource standards model of working memory have not been
able to give an account for these dierences from an information processing perspective.
Some researchers have referred to these dierences from executive attention perspec-
tive and have related the dierence to a higher demand of backward recall for executive
attention. Yet it is not explained what is special about the backward recall which makes
it more attentional demanding compared to forward recall. One might say that backward
recall requires manipulation of information and that is the source of higher demand for
executive attention.
142
However, evidences are more specic in terms of the nature of the dierence between
strategies and mechanisms in forward and backward recall. Evidences from behavioral
studies, clinical studies and neuroimaging studies consistently show that backward recall
relies on spatial resources and spatial strategies while forward recall (at least in hearing-
abled subjects) relies on phonological resources.
Behavioral evidences show that manipulations of dierent parameters might aect
performance of normal subjects during forward and backward recall dierently. While
manipulation of phonological aspects of the task aects performance of forward recall
they do not aect the performance in backward recall (Bireta et al., 2010). Bireta et
al. tested four benchmark eects that demonstrate the role of phonological resources
in immediate forward recall tasks { the word length eect, the irrelevant speech eect,
the acoustic confusion eect and the concurrent articulation eect{ for both directions of
recall. They reported that the benchmark eect `was either absent or greatly attenuated
with backward recall despite being present with forward recall'. In contrast performance of
backward recall seems to be sensitive to manipulation of visual-spatial parameters while
forward recall practically is insensitive to those manipulations (Li and Lewandowsky,
1995a).
Neuropsychological evidence also supports that neurological damage to phonological
resources of the brain impairs forward digit span while damage to spatial resources of
the brain impairs backward digit span (Koenigs et al., 2009; Rudel and Denckla, 1974).
Consistent with these observations, neuroimaging studies also have revealed dierences
in cortical regions which are active during the two dierent recall orders (Hoshi et al.,
2000; Sun et al., 2005). In particular, these studies have revealed signicant activation
of cortical areas with spatial processing characteristics in backward recall compared to
forward recall.
Now we can rephrase our question in a more specic form: what is special about
the backward recall which makes it rely on spatial strategies and spatial resources and
what is special about forward recall which makes it rely on phonological strategies and
143
phonological resources? What are the costs and the benets of performing backward
recall with only phonological resources? How about the costs and benets of performing
forward recall with only spatial resources?
We give an answer to these questions by analysing ramications of employing serial
access and random access strategies for both forward and backward recall tasks. Our
analysis shows pure serial access schemas have a clear disadvantage in backward recall so
that a performing backward recall is implausible by employing a pure serial access schema.
In contrast serial recall schemas not only are feasible but also have a relative advantage
compared to random access schemas for forward recall. We also show that random access
schemas not only are feasible for backward recall but also they can account for humans'
error patterns in recalling a list of items. We show this by a computational model of
LRS system and simulation of forward and backward recalls using LRS model. We will
use these results to give an account for involvement of spatial strategies and mechanism
in backward recall and involvement of phonological strategies and resources in forward
recall.
7.2 Assessment of pure strategies for serial recall tasks
Here are proposed some pure strategies for using either serial access or random access for
the task of retaining and then recalling a list of items. The goal is assessing feasibility of
recalling in each of orders with each of access strategies and also identifying the source
of limitations and optimization opportunities.
7.2.1 Forward recall with serial access schema
The ability of SAWM and their supporting SACS systems in encoding serial order makes
them an ideal choice for retaining a sequence of items indenitely and through utilization
of their sensory-articulatory loop repeatedly. In addition to a symbolic segment for each
list item, marker segment is necessary for marking the beginning or the end of list. The
144
recall process in this case is indeed an overt articulation of list items followed by a long
{innite{ pause. When recalling a list is cued, the end or starter marker can be utilize
for starting the overt articulation at the correct place. The end marker segment may also
be used to identify when stop the overt articulation and erasing the content of SAWM.
Here is a compact representation of the SAWM schema for retaining a list of items :
While recall is not cued :h
1

2

n

?
i
Where
i
denotes the symbolic segment associated to the i-th item ,
?
denotes pause
segment working as an end marker and denotes an identity process with covert artic-
ulation.
When recalling is cued then after the rst appearance of the end marker an identity
process followed by overt articulation is executed and nally a long pause is added to the
end of the list. Here is a compact representation of SAWM schema for recalling the list
of items:
When recall is cued :h
?

1

2

n

?1
i
Where denotes an over articulation after an identity process and
?1
denotes the
innite pause which results to erasing the buer.
There are some ways for optimizing an instance of SACS for forward recall. One
optimization opportunity using only SACS resources is related to maximizing number
of items in the list by shortening associated symbols or a faster articulation of segments
(Baddeley et al., 1984). Utilization of other resources such as long-term memory may even
help optimize the capacity of SACS in the recall task by choosing proxy segments which
are very short and highly distinguishable. Choosing distinguishable proxy segments may
result in faster segmentation and reduce confusion errors that may occur as the result of
similarity of segments.
7.2.2 Forward recall with random access schema
To employ a random access strategy for forward recall the address space needs a means for
encoding the order. There are a number of ways for applying an order on address space
145
provided by LRS. One way is relying on the mechanism for deploying sequential programs:
a programmed shift of selective attention between locations (e.g. left, up, right and down
as registry locations ). Such a program needs to be represented in the form of schema
for utilizing a number of addresses in a specic order (e.g. access to left, up, right and
down either in clockwise order or counter-clockwise order). A stateless implementation is
also possible wherein items are selected based on a unique feature of activation patterns
in the address space. We give an example of such stateless mechanism in our simulation
of an LRS supported strategy for immediate recall. In this scenario activation of space
representation is subject to a slow decay which allows distinguishing activation patterns
based on their order of registry. Either of these mechanisms can provide the capacity of
assigning items in order and retrieving them based on the required order.
Implementation of an access schema for forward recall in a RAWM system then is as
simple as binding symbolic items to the address space in the order of presentation and
retrieving them in the same order.
In terms of feasibility, this strategy faces a hard constraint in terms of the number of
items that can be maintained in order. This limitation is a manifestation of limitation
in number of items that can be registered to the address space. So a pure schema that
employs only random access strategy can not be deployed to storing more than maximum
number of registries and thus unlike the case SAWM and serial access strategies the
number of items held can not be increased by relying only on the supporting LRS system.
However, one may utilize other resources such as the long-term memory for chunking
strategy in which one representation replaces two or more representations in order.
7.2.3 Backward recall with serial access schema
The unidirectional encoding capacity of SAWM which allows a natural way for encoding
the order of items for forward recall makes it dicult to utilize a pure serial access
strategy for backward recall. This diculty originates from this feature of SACS systems
that the last item of a list can not be identied unless an end marker is decoded. In fact,
146
the strategy of
agging a segments which was previously explained can be applied only
in one direction which is determined by encoding order. This means that once the end
marker is parsed the symbol for the last item of the list which is processed before the end
marker is already inserted in the articulatory buer.
One way to circumvent this issue is utilizing another source of memory which can
maintain the last parsed item of the list constantly. So, along decoding a segment from
sensory buer and inserting the converted articulatory code into the articulatory buer
a representation associated to the parsed items needs to be maintained in an auxiliary
memory as the latest processed item. This one item should be replaced with the next
item unless the next symbol is the end marker which this may cue for overt articulation
of the item by recalling articulatory code of the item of the auxiliary memory. A source
of diculty in performing this strategy is related to the overt articulation of the last item
which may interfere with the content of articulatory buer. This requires
ushing the
queue in the articulatory buer covertly and before the overt articulation of the last item
which in the meantime will ll the sensory buer. In this scenario articulation of last item
will add its sensory code to the end of sensory buer again which needs to be eliminated
by rst tagging with symbol for dropping from the articulation in the next run.
Altogether, our minimal model of serial access schema and its language-supported
sensory-articulatory system which can be easily utilized for a forward recall task can not
be utilized for backward recall unless is augmented with an auxiliary working memory.
We will discuss that this issue also can be circumvented by employing a random access
schema and utilizing LRS systems instead.
7.2.4 Backward recall with random access schema
As it was explained for the case of forward recall, an ordered address space for random
access can be instrumental in accessing items in a specic order. The only dierence
in applying a random access schema in backward recall compared to forward recall is
related to the schema for shifting the selection attention between ordered space during
147
binding phase and recall phase. Implementation of an access schema for backward recall
in a RAWM system then is as simple as binding symbolic items to the address space in
the order that matches presentation of items and retrieving them in the reverse order
(e.g. bind items to left, up, right and down in a clockwise order and retrieve them in
counter-clockwise order).
In sum, our analysis of pure access schemas for recall tasks reveals the disadvantage of
a pure serial access schema in backward recall of items which forces the use other resources
for performing mental backward such as LRS resources either as an auxiliary working
memory or the main utility for employing a random access schema. Also our analysis
revealed the relative advantage of utilizing SACS systems in forward recall compared to
LRS systems at least in terms
exibility in the number of items that can be encoded.
However, the number of items can not be the main concern if the quality of recall {
measured by error rates{ is not comparable with what can be achieve by utilizing LRS
resources. To assess the capacity of LRS-based strategies in immediate recall tasks for
both forward and backward tasks we implemented a computational model of a simple
implementation of LRS systems. The result of simulations show that LRS strategies
can account for human data for both forward and backward recall. The signicance of
this simulation is in showing that 1. a LRS-based strategy can not achieve a better
performance in forward recall compared to SAL strategies and thus it justies the use of
SAL resources for forward recall 2. a LRS-based strategy very well accounts for backward
recall which in turn justies the use of LRS resources for backward recall.
In the next section we present the result of our computational modeling of both
forward and backward recall using a simple stateless LRS model.
148
7.3 Computational Modeling of LRS for Immediate Serial
Recall
Forward recall, disproportionately, has received more attention in modeling attempts.
This is mainly related to the importance of temporal serial order in everyday tasks
(Glasspool, 2005). As the result, there are many neural models, behavioural models,
and mathematical models dedicated to describing forward recall. In contrast, for back-
ward recall, theoretical eorts mostly have focused on augmenting or reusing models of
forward recall. In the face of abundant behavioural and neural evidence that serial recall
in forward and backward directions draw on dierent brain mechanisms, it is not sur-
prising that models of backward recall have gained remarkably less success in describing
human behaviour compared to forward recall (Bireta et al., 2010). Only
exible math-
ematical models with enough degrees of freedom, such as the Temporal Ratio Model
(Brown et al., 2007), have been able to successfully model both recall tasks in one shot
(Bireta et al., 2010).
In terms of modling eorts, Bietra et al. have brie
y reviewed existing models. Their
review indicated that those models that take the phonological aspect of forward serial
recall are not successful in modeling backward recall, and only models that are agnostic
to the dierence in underlying mechanisms of serial recalls in two dierent directions are
relatively successful in modeling both tasks. Unsuccessful attempts in utilizing phonolog-
ical models for backward recalls is consistent with the result of our analysis in limitation
of serially accessible working memory systems in supporting backward recall.
7.3.1 Elements of the computational model
Here we devise a computational model of serial recall using a random access strategy by
utilizing a Location Registry System. In our model we apply a representation of space
using a population coding of a one dimensional space in the array of neurons as the
addressing or registry space. Population coding of neurons has been extensively explored
149
(Pouget et al., 2000) in the literature and is popular for neural modeling of visuospatial
working memory (Constantinidis and Wang, 2004).
Moreover, we implement a stateless strategy for selecting items in the required order.
This means that we rely on a simple condition for selecting the next registry item: select
the most active neuron in the space neural representation and choose the object with
closest registry location (see Figure 7.1). Yet to dier strategies between forward recall
and backward recall we employ a bias in our selection: in forward recall task items are
registered in a random distance from the last registered item in opposite direction of the
spatial bias, so that items that are registered earlier receive a boost from the bias factor.
In backward recall items are registered in a random distance from the latest registry in
the direction of the spatial bias. In this case items which are registered later receive more
boost from the spatial bias factor.
We also have relaxed the maximum binding constraint to explore the capacity of
employing a known neural representation of space on error patterns.
This array of neurons encodes a parametric space spanning the range of -1 to 1. The
tuning curve for neurons in this array was characterized by
0
+x
n
where
0
is the
tuning band parameter of the neuron at the center of space, x
n
is neuron's peak response
location, and a constant which controls the variability of tuning band in the array of
neurons.
Registering with a specic location would trigger noisy activation in the population
around the target memory eld. The share of a registry at x
r
in activation amplitude of
a neuron at x
n
is determined by A
0
e

(xrxn )
2
2n
2
. In case of registering several items in the
activation of a neuron is dened as the sum of evoked signals of all registries as long as
the sum of signals is less than a saturation valueS. So the base response amplitude of
neuron n is dened as follow:
max(S;
N
X
i=1
A
i
(t)e

(xr
i
xn )
2
2n
2
) (7.1)
150
where i is the index for registered items, x
r
i
is the registry location of the item i and
A
i
(t) denotes the eective amplitude of the ith registry at time t which is dened by:
A
0
e

tt
i

d
(7.2)
wheret is the current time, t
i
the registry time of item i and
d
, the damping factor
which controls the decay rate of registry eects.
The schema for the immediate recall task includes two phases: binding and recall.
During the binding phase, independent of the recall order, items of the list orderly register
with locations from left to right so that each item in the list registers on the right side
of previously registered item (except the rst item). The exact times and locations of
registries are perturbed by dierent random distributions. The distances between registry
locations are determined by a Weibull distribution with two parameters (shape factor
and scale factor). Duration of registry and recall processes are dened by two separate
Gaussian distributions, which adds four more parameters to our model.
In the recall phase, a part of the schema is independent of recall direction, which
is the condition for identifying the most active neuron, and for selecting the next item
(until all items are removed from the registry space). Neurons in the array compete for
gaining control of a registry recalling unit. The item at the closest registry location to the
selected neuron will be recalled. Recalling memory items from registry involves inhibiting
neurons in the array associated with registration of the recalled item.
Another part of the recall schema which is sensitive to the direction of recall is charac-
terized by a bias. The bias is applied by a multiplicative exponential factor of the position
which acts as a biased modulation of neural activities. For forward recall, this bias will
enhance the activity of neurons on the left side of the space, and during the backward
recall this bias enhances activities of neurons on the right side of the space. So, a part
of the schema for recall is selecting the bias direction; however, once the bias direction is
selected items will be selected only based on the order of most active neurons.
151
This implementation only accommodates positional or movement errors in which items
are recalled in the wrong order. This type of error is the most prevalent error among adults
(McCormack et al., 2000) in recall tasks. However there are other types of errors such as
omissions, intrusions and repetitions with less signicant eect. Table 7.1 summarizes all
parameters of this implementation.
7.3.2 Simulation
To explore tuning parameters we used serial position error for a list of ve items from Li
and Lewandowsky's study (Li and Lewandowsky, 1995a). A genetic algorithm was used
to optimize the parameters based on the sum of absolute distance of predicted positional
error over the ground truth data for both directions. So optimization of parameters
was performed with regard to ground truth data for both directions simultaneously and
forward and backward error data played equal roles in the tness function. However
a closer inspection of the result revealed that the nal parameters shifted in favour of
the backward data. The best tting parameters among 2857 independently generated
solutions yielded a prediction for backward recall with 5.6% absolute distance to the
human data (out of 500% maximum possible distance) while the same set of parameters
yielded a prediction for the forward recall with 14.8% absolute distance to the human
data (see Figure 7.3). Further analysis of best rst 100 independent solutions of the
optimization process showed that the quality of predicted solution for backward recall was
signicantly better than forward recall (t(198) = 47:93;p< 0:0001), where the dierence
between mean of tness qualities was 6.6% in favour of the backward recall.
Moreover, a closer inspection of all generated parameter sets during optimization
process revealed two highly distinguishable modes for
0
, the rst order tuning curve
parameter. A population of solutions with narrow tuning curve at the center peaked
around
0
= 0:04 which included 847 solutions all with
0
< 0:1. Another population
of wide tuning curve at the center peaked around
0
= 0:57 all with
0
> 0:38 included
2010 solutions. Later analysis of the tness values of these solutions showed that the
152
Par Description Par Description

0
Spatial tuning at the center Tuning band var factor
Bias factor Noise factor

d
Damping factor K Binding shape factor
Binding scale factor S Saturation factor

b
Mean for binding duration
b
STD for binding duration

r
Mean for fetching duration
r
STD for fetching duration
Table 7.1: Parameters of the LRS model for serial recall.
population of wide tuning curve (WTC) on average scored better tness value than the
population of narrow tuning curve (NTC). The wide tuning curve population (WTC)
generally scored better in each of recall types compared with the narrow tuning curve
population. Moreover, WTC and NTC populations were also highly separable with regard
to other parameters including the bias factor, and temporal characteristics of binding and
recall of item. In particular the for WTC the average duration of the task was correlated
with the damping factor of neural activity while the duration of the task was independent
of the damping factor of neural activities. In sum, WTC population both outnumbers
and outperforms NTC population (see Figure 7.4) and produces solutions that are more
plausible. So, for the rest of our discussion and analysis we focus on WTC population.
To test the predicting power of the model we used the parameter of the best solution
discovered in the optimization of the previous phase to simulate the movement errors
(the distance between order of an incorrectly recalled item, and its true order; e.g., if
item 3 is recalled as item 2, the movement error is 1) in forward recall data for six
items, from another study (McCormack et al., 2000). Note that number of items for
training was dierent than for testing. Moreover, positional error data, which is used
for optimization of parameters, is independent of movement errors (which we conrmed
through simulation, not shown here).
Figure 7.6 shows the result of our simulation in the same graph with the data of two
adult human subject groups, tested in two dierent experiments with dierent settings for
a forward recall task (McCormack et al., 2000). Our simulation result sits in between the
153
n Permutation p
forward
p
backward
1 (0,1,2,3,4) 0.6046403 0.00015968562
2 (0,1,2,4,3) 0.0121734 0.00068997736
3 (0,1,3,2,4) 0.10311276908 0.00016032288
4 (0,1,3,4,2) 0.03157189188 0.00032157369
5 (0,1,4,2,3) 0.00154016671 0.00033422539
6 (0,1,4,3,2) 0.07464263158 0.00046375339
7 (0,2,1,3,4) 0.00010147144 0.00139273776
8 (0,2,1,4,3) 0.00250100693 0.00047199018
9 (0,2,3,1,4) 0.00059827815 0.00323071585
10 (0,2,3,4,1) 0.0008942140 0.0003289762
11 (0,2,4,1,3) 0.0000011791 0.00357683509
12 (0,2,4,3,1) 0.0031144277 0.00034608175
13 (0,3,1,2,4) 0.00197864610 0.00003511804
14 (0,3,1,4,2) 0.01089355518 0.00020281040
15 (0,3,2,1,4) 0.03095597716 0.01711802701
16 (0,3,2,4,1) 0.00926732975 0.00004554765
17 (0,3,4,1,2) 0.00079234745 0.00154017014
18 (0,3,4,2,1) 0.00908012591 0.00072838894
19 (0,4,1,2,3) 0.00030371760 0.00147133730
20 (0,4,1,3,2) 0.00271571818 0.00011698265
21 (0,4,2,1,3) 0.00063935151 0.02324862186
22 (0,4,2,3,1) 0.00457526782 0.00008697225
23 (0,4,3,1,2) 0.00343418376 0.00184444443
24 (0,4,3,2,1) 0.01047781214 0.00034668616
25 (1,0,2,3,4) 0.00065745704 0.00034548544
26 (1,0,2,4,3) 0.00035652888 0.00013601739
27 (1,0,3,2,4) 0.00017789269 0.00002837850
28 (1,0,3,4,2) 0.00012194637 0.00037954442
29 (1,0,4,2,3) 0.00040352362 0.00035766989
30 (1,0,4,3,2) 0.00042284632 0.00013689855
31 (1,2,0,3,4) 0.00085006412 0.00043182322
32 (1,2,0,4,3) 0.00075792094 0.00168630127
33 (1,2,3,0,4) 0.00017893688 0.00014459110
34 (1,2,3,4,0) 0.00030602363 0.00580797787
35 (1,2,4,0,3) 0.00068317284 0.00013761211
36 (1,2,4,3,0) 0.00027449512 0.00242081374
37 (1,3,0,2,4) 0.00005774096 0.00011119342
38 (1,3,0,4,2) 0.00099057822 0.00026847969
39 (1,3,2,0,4) 0.00021950386 0.00016061928
40 (1,3,2,4,0) 0.00214711129 0.00040082264
41 (1,3,4,0,2) 0.00032589975 0.00016099225
42 (1,3,4,2,0) 0.00129564915 0.00058408523
43 (1,4,0,2,3) 0.00104248490 0.00136860811
44 (1,4,0,3,2) 0.00009279852 0.00010073202
45 (1,4,2,0,3) 0.00294780818 0.00004179213
46 (1,4,2,3,0) 0.00135661034 0.01293769026
47 (1,4,3,0,2) 0.00385672251 0.00037448245
48 (1,4,3,2,0) 0.00069253997 0.00424715405
49 (2,0,1,3,4) 0.00007693830 0.00017166904
50 (2,0,1,4,3) 0.00047608667 0.00025314897
51 (2,0,3,1,4) 0.00055558306 0.00346949140
52 (2,0,3,4,1) 0.00018092401 0.00044993139
53 (2,0,4,1,3) 0.00003916164 0.00543177381
54 (2,0,4,3,1) 0.00061179619 0.00003121876
55 (2,1,0,3,4) 0.00124361454 0.00043275151
56 (2,1,0,4,3) 0.00019844832 0.00002446409
57 (2,1,3,0,4) 0.00080017654 0.00014895411
58 (2,1,3,4,0) 0.00325932587 0.00787156924
59 (2,1,4,0,3) 0.00144758735 0.00022116438
60 (2,1,4,3,0) 0.00070685546 0.00367433716
n Permutation p
forward
p
backward
61 (2,3,0,1,4) 0.00006647121 0.00106365810
62 (2,3,0,4,1) 0.00089635448 0.00013288573
63 (2,3,1,0,4) 0.00238929229 0.00003127253
64 (2,3,1,4,0) 0.00050094308 0.00120891691
65 (2,3,4,0,1) 0.00097361563 0.00016895119
66 (2,3,4,1,0) 0.00066256967 0.13667276981
67 (2,4,0,1,3) 0.00025753014 0.00396579565
68 (2,4,0,3,1) 0.00174435329 0.00000189341
69 (2,4,1,0,3) 0.00004760983 0.00041155193
70 (2,4,1,3,0) 0.00141826140 0.00253759648
71 (2,4,3,0,1) 0.00006470538 0.00005698367
72 (2,4,3,1,0) 0.00132945330 0.03146024695
73 (3,0,1,2,4) 0.00026912736 0.00035569076
74 (3,0,1,4,2) 0.00101008193 0.00009342554
75 (3,0,2,1,4) 0.00017914005 0.00024356113
76 (3,0,2,4,1) 0.00063877241 0.00002678006
77 (3,0,4,1,2) 0.00056134183 0.00053831830
78 (3,0,4,2,1) 0.00034557478 0.00049406793
79 (3,1,0,2,4) 0.00020701326 0.00038877037
80 (3,1,0,4,2) 0.00299647364 0.00046024253
81 (3,1,2,0,4) 0.00003529741 0.00032930542
82 (3,1,2,4,0) 0.00336826810 0.00003402121
83 (3,1,4,0,2) 0.00241469881 0.00001432137
84 (3,1,4,2,0) 0.00040894446 0.00068977017
85 (3,2,0,1,4) 0.00008546612 0.00014900577
86 (3,2,0,4,1) 0.00017073633 0.00007911114
87 (3,2,1,0,4) 0.00056245112 0.00007389609
88 (3,2,1,4,0) 0.00061675097 0.00011918280
89 (3,2,4,0,1) 0.00061838027 0.00039966248
90 (3,2,4,1,0) 0.00061981234 0.00830154931
91 (3,4,0,1,2) 0.00065608355 0.00006557678
92 (3,4,0,2,1) 0.00094771991 0.00031940136
93 (3,4,1,0,2) 0.00093440812 0.00024736164
94 (3,4,1,2,0) 0.00137330518 0.00086861290
95 (3,4,2,0,1) 0.00175017749 0.00011611779
96 (3,4,2,1,0) 0.00061752188 0.00446243173
97 (4,0,1,2,3) 0.00099632874 0.00038738959
98 (4,0,1,3,2) 0.00059710903 0.00002950148
99 (4,0,2,1,3) 0.00120551142 0.00014459770
100 (4,0,2,3,1) 0.00035339206 0.0001953142
101 (4,0,3,1,2) 0.00068249394 0.0006052290
102 (4,0,3,2,1) 0.00036570356 0.0003323089
103 (4,1,0,2,3) 0.00049486110 0.0000684398
104 (4,1,0,3,2) 0.00045602949 0.0001556186
105 (4,1,2,0,3) 0.00068379331 0.0001671947
106 (4,1,2,3,0) 0.00011968751 0.1498838799
107 (4,1,3,0,2) 0.00148923503 0.0003163002
108 (4,1,3,2,0) 0.00163075390 0.0021679217
109 (4,2,0,1,3) 0.00000616132 0.0002164008
110 (4,2,0,3,1) 0.00039196403 0.0004408783
111 (4,2,1,0,3) 0.00017327856 0.0000625864
112 (4,2,1,3,0) 0.00030830035 0.0017918149
113 (4,2,3,0,1) 0.00065045396 0.0003567193
114 (4,2,3,1,0) 0.00183534215 0.1234094848
115 (4,3,0,1,2) 0.00140391723 0.00004259033
116 (4,3,0,2,1) 0.00098699330 0.00021708397
117 (4,3,1,0,2) 0.00219732498 0.00000278926
118 (4,3,1,2,0) 0.00029542573 0.00005648891
119 (4,3,2,0,1) 0.00051954759 0.0000604640
120 (4,3,2,1,0) 0.00059909853 0.4090940741
Table 7.2: An over-t probabilistic model (OPM) for both forward (OPM-F) and backward
(OPM-B) recall with 240 degrees of freedom which ts the positional error graphs for human
performance. The chance of recalling a sequence in a particular permuted order during a certain
recall is given wby the corresponding probability value.
154
data points for two dierent results for adult human subject groups, which demonstrates
that our prediction is in the range of the variability of the performance of human subjects,
and clearly demonstrates the predictive power of the model.
In sum, the result of our simulation shows that LRS for immediate serial recall can
account for human behaviour. However, as it was explained, the quality of our solution
for backward recall is signicantly better than the quality of our result for forward recall
when both recall orders played the same role in optimization of parameters.
This does not mean that visual-spatial resources cannot be used for forward recall. In
fact previous studies have shown that articulatory suppression during working memory
task with written verbal material can eliminate the eect of other signature eects such
as word length eect or acoustic confusion eect without diminishing subjects' capacity
for remembering the serial order (Baddeley et al., 1984; Wilson, 2001). These evidences
suggest that once the speech recognition-vocalization system as the primary source of
encoding serial recall is no longer accessible (by articulatory suppression) and working
material are presented in visual format, another mechanism is utilized for encoding serial
order and recall task which does not rely on phonological resources.
Moreover it has been established that children after the age of 7 start utilizing phono-
logical strategies for short-term retention of information. The evidence comes from this
observation that in recalling items by pointing to pictures, it is at the age of 7 that word
length eects and phonological confusion start appearing (Henry, 1991).
Our results suggest that LRS resources can be used for serial order and forward recall
and thus they are may replace phonological resources for forward recall. In this case,
based on the prediction of our LRS model the overall performance in forward recall
using LRS resources would not be signicantly better than what can be achieved by
phonological resources even when no limitation on the capacity of LRS is imposed (see
Figure 7.3(a)). Using phonological resources has two advantages in forward recall: rst
the opportunity for achieving a larger span for forward recall by adjusting the length and
speed of articulation and second, using phonological resources will leave LRS resources
155
(often visual-spatial resources) free and available for crucial perception-action routines.
So, choosing phonological resources over visual-spatial resources for forward recall seem
to be the result of the relative advantage of phonological resources. However, as our
analysis in the case of backward recall replacing visual-spatial resources with phonological
resources is not feasible.
7.3.2.1 Testing independence of positional errors and displacement errors
As it was discussed, we trained our model with positional error data for forward and
backward data and tested its prediction for displacement error. To establish the power of
this prediction here we show that the displacement error data and positional error data
are independent and the success of our model is not indeed a result of piggybacking on
over-tting the positional error data.
To do so we built a probabilistic model with high dimension of parameter space to
t our target positional error data. This model is built by training probability values
for recalling a list in possible permutations of the list and thus includes 240 parameters
(each for a particular permutation in a specic recall). We call such a model as Overt
Probabilistic Model or OPM for short.
We used similar evolutionary principles for training dierent OPM s. Table 7.2 gives
one instance of OPM trained model that ts the the positional error curves very well
(see Figure 7.7). Testing OPM model on displacement error shows that this model
fails to produce the signature displacement error data (see Figure 7.8) which exponen-
tially decreases with increasing the displacement (see Figure 7.6). This established the
independence of positional error and displacement error data.
156
l
A
A
l
A
A
l
B
B
l
A
A
l
B
l
C
B
C
l
A
A
l
B
l
C
B
C
l
A
A
l
B
l
C
B
C
Objecet
Representaion
Location Representaion
Neural Activation Level
a. b. c.
d. e.
Figure 7.1: A depiction of our simulated locational binding to a one-dimensional space (top row),
and then applying spatial biases for forward and backward recalls (bottom row). In our simulation,
symbolic items are assigned to locations from one end (e.g., left) and are registered with space at a
random distances. A,B andC are symbolic representation of task-relevant items registered with
dierent spatial locations (respectively l
A
; l
B
and l
C
). Each panel shows activation level (spike
rate) of neurons in a population which is responsible for the one-dimensional encoding of space.
Registering with a specic location would trigger noisy activation in the population around the
target memory eld. The big oval contains symbolic object representations of task-relevant items.
For example in panels a3, b1 and b2, three objects are maintained while in panels a1 and a2
respectively one and two items are maintained. Relative size of circles of symbolic objects shows
the relative chance of objects to be retrieved in the rst recall in that stage. This means that in
panel b1 items are likely to be recalled in the order of registry (forward recall), and in panel b2 items
are likely to be recalled in the reverse order of registry (backward recall). When a neuron wins
the competition for gaining control of the recaller unit, the nearest object in the one-dimentional
space will be selected. The most active neuron at the time of recall wins the competition, and
thus the chance of selecting an object during recall is determined by the maximum activation level
of its associated neurons. These panels do not show decay and saturation eects and are here to
explain our modeling of binding, bias and recall.
157
N
0 0.1 0.2 0.3 0.4 0.5 0.6
0
100
200
300
σ
0
Figure 7.2: The histogram of
0
s for best solutions of dierent runs of the evolutionary algorithm.
Two separate populations are distinguishable; narrow spatial tuning (
0
< 0:1) with 847 solutions
and wide spatial tuning (
0
> 0:38) with 2010 solutions.
158
0.3
0.1
1 2 3 4 5
Er r o r
0.5
P o s i t i o n
(a) Forward Recall
0.3
0.1
1 2 3 4 5
Error
0.5
Position
Human performance
Two-way best fit
(b) Backward Recall
Figure 7.3: Positional error for human performance superimposed on the best two-way t among
2857 independently generated solutions. Each solution is created with a genetic algorithm with
minimizing the sum of deviations from human data as the target of the genetic modication of
the parameters of the simulation.
159
15 20 25 30 35 40 45 50
0
200
400
Overal fitting % error (sum) for both forward and backward recall
n
Wide Spatial Tuning Curve (N=2010)
Narrow Spatial Tuning Curve (N=847)
Figure 7.4: Distribution of tness values for solutions with wide tuning curve (WTC) and narrow
tuning curve (NTC). Wide tuning curve solutions outnumber and outperform narrow tuning curve
solutions.
160
15 25
5
15
25
Forward Recall Overall Error (in percents)
Backward Recall Overall Error (in percents)
N = 2010
Forward Error = Backward Error
Sum of errors = 26%
Figure 7.5: Comparing the quality of prediction in backward recall and forward recall for all 2010
solutions with wide tuning curve. For the best 356 rst solutions the quality of backward solutions
were better than the quality of forward solutions. This suggests that overall, the quality of our
simulation for backward recall was signicantly better than the quality of solution for forward
recall.
161
1 2 3 4 5
Distance
Proportion of Movements
0.1
0.2
0.3
0.4
0.5
0.6
Simulation
Human Exp 2
0.7
Human Exp 1
Figure 7.6: Prediction of LRS model with large binding capacity along experimental data for
movement errors during serial recall of six items. Without a capacity limitation forward recall
with a random access strategy and using LRS matches performance of normal subjects very well.
162
0.5
0.3
0.1
1 2 3 4 5
b.
Data
Model
0.5
0.3
0.1
1 2 3 4 5
P o s i t i o n
a.
Er r o r
Er r o r
Figure 7.7: Performance of the over-t probabilistic model for a. forward recall (OPM-F) and
b. backward recall (OPM-B). Each graph shows the probability of making error in recalling the
item in the given position.
163
0
Displacement
Proportion of displacement
Backward
Forward
1 2 3 4
0.4
0.8
Figure 7.8: Recall displacement error for the over-t probabilistic model forforward recall (OPM-
F) and backward recall (OPM-B).
164
Chapter 8
The Case of Concurrent Counting from SWMS perspective
8.1 Introduction
Ecient utilization of the brain's limited capacity in short-term maintenance of informa-
tion (Cowan, 2001; Miller, 1956) is key to ecient performance of complex intellectual
tasks. A challenge in this regard is selection of particular representations in working
memory which are relevant to a particular stage of the process or the running process.
Take for example the case of mental addition of 34 and 59. Following a commonly taught
procedure for multi-digit addition which starts with adding digits in lower value posi-
tions(Butterworth, 2005; Hamann and Ashcraft, 1986; McCloskey et al., 1991), at rst,
digits 9 and 4 are more relevant to the process and needed to be acted upon rst while two
other digits should be maintained for next stages of the process. These two digits should
trigger recalling the addition fact 9+4=13 (Hamann and Ashcraft, 1986). Recalling this
addition fact from memory will insert two more digits (1 and another 3) to the content
of working memory while 9 and 4 lose their relevance and thus are safe to be abandoned
and leave the limited space of working memory for the two newly generated digits. While
the newly inserted digit 3 should be retained up to the end of the task to be uttered as a
part of the answer only 1, 3 and 5 become relevant to the dynamic of the process in the
next stage.
165
The challenge here and in many non-trivial intellectual tasks stems from the fact that
over the course of the task execution, the role of residents of working memory and their
relevance to the current process dynamically change; at some stages, one representation
may reside in the working memory without aecting the result of the running process
while it may aect the outcome of the process in another moment or stage of the process.
Flexible, reliable and fast mechanisms that allow random access to residents of working
memory may help ecient performance of complex intellectual tasks that are more com-
plex than short-term retaining of a phone number. An executive model for functioning a
working memory model needs to be specic about accessing memory content.
The standard model relies on the CE for its information management operations
which is postulated to do its job through regulating limited executive attentional re-
sources (Baddeley, 1996). This paradigm for explaining functioning of the CE does not
give detailed account for how selection of items takes place. As the result explaining dy-
namics of memory management in particular those aspects that are related to selection
of items are not very well covered by the standard model. Thus CE-based models need
additional assumptions for explaining functioning of information processing through se-
lection processes. The extensive debate over the nature of representation in the working
memory community has encumbered reaching to a detailed picture of the central execu-
tive in all alternative models so that until recently no attempt serious attempt was made
to integrate the concept of item selection to theoretical frameworks of working memory.
In cases of particular intellectual tasks of practical or theoretical importance some
researchers have circumvented a deadlock by making specic data driven assumptions
about the nature of representation in those particular domains. For example in the light
of neuropsychological evidences of disabilities in performing arithmetic tasks McCloskey
has proposed a cognitive model which incorporates elements of Baddeley and Hitch's
multi-component model of working memory which applies to number processing in math-
ematical tasks(McCloskey, 1992; McCloskey et al., 1985). Or, Knau and his colleagues
who have studied spatial reasoning have borrowed some elements from Kosslyn's mental
166
imagery model (Kosslyn, 1980) to devise an execution model for reasoning in dierent
domains from spatial reasoning (Knau et al., 1998) to deductive reasoning (Knau et al.,
2002). Although these executive models are devised for specic tasks they often include
modeling components that can be spared for a more general execution model.
Recently some attempts have been made towards building a general executive ma-
chinery that aims a broader WM framework for intellectual tasks by using the concept
of object selection or selective attention in the internal domain. These accounts use the
concept of selective attention in dierent contexts as the means to access to the con-
tent of working memory for cognitive processes. The rst account has been proposed
by Oberauer (Oberauer, 2002, 2003) and builds on Cowan's embedded working working
model which assumes representations in working memory as long-term representations in
an activated state and under the focus of attention (Cowan, 1999).
Oberauer's model for functioning of working memory incorporates two forms of focal
attention in the internal domain in an embedded three-layer working memory model. The
rst layer includes representations of long-term memory in an activated state which are
not directly accessible to cognitive processes unless they are at the focus of relatively
broad attention and become a part of the second layer. The second or middle layer in
Oberauer's model employs a form of attention which may span up to three or four chunks
of information (Oberauer, 2002, 2003). This layer is closely related to what Cowan has
previously proposed in his embedded model of working memory (Cowan, 1999). The third
layer in Oberauer's model employs a narrower focus of attention which holds a single item
of the second layer with the highest relevance to the current stage of the cognitive process.
Experimental evidences for a narrow focus of attention comes from studies of time
course of cognitive processes (McElree, 1998). Patterns of speed-accuracy trade-o in
Sternberg's task (McElree and Dosher, 1989) and n-back task (Mcelree, 2006; McElree,
2001) or execution times in concurrent counting (Garavan, 1998; Voigt and Hagendorf,
2002) or mental arithmetic (Oberauer, 2003) suggest that the most recently accessed item
in the working memory is privileged in terms of the speed of processing. Some researchers
167
suggest that the speedy process of the most recently processed item pertains to the cost
of selecting an object of working memory for processing, a cost which is not necessary
for brining the most recently processed objects of working memory into focus (Jonides
et al., 2008a; McElree and Dosher, 1989; McElree, 1998, 2001). The broader form of
attention is what brings a limited number of long-term representations into the working
memory and in an accessible mode and the narrower focus of attention selects the most
relevant representation to the running process. In terms of mechanisms for controlling
the shift in focus of attention Oberauer's model follows Cowan's model in assuming a
modality-neutral role for the CE.
Oberauer's concept of layered focus of attention operating on the long-term repre-
sentation has some attractive features, for one, it gives a selection-based account for the
interplay of storage and process in the working memory which can be used towards more
detailed account for the dynamics of working memory. However Oberauer has built on
Cowan's model uses the concept of attention to give an account for limitation in the
capacity of working memory and thus is not supplied with enough details to explain
mechanisms for shift of this focus of attention.
Our proposed Symbolic Working Memory System (SWMS) can also be described as
an access based theory for management of information in the working memory which
operates on symbolic representations of task items in sensorimotor systems. The way our
model addresses the issue of selection of working memory content is by means of built of
selection mechanisms in sensorimotor systems which are explained in detail in Part III.
We suggested two access modes supported by two dierent categories of sensorimotor
systems with advantages and limitations which are associated to underlying mechanisms
for management of information. In particular, we suggested that limitations of serial
access systems in processing items in a unidirectional way makes them highly inecient
in those mental tasks that require unexpected and random access to the content of working
memory. It was discussed the preferred systems for these scenarios are those systems that
utilize space for random access to symbolic content of working memory. We also discussed
168
that the most ubiquitous system for such purpose is ocular system. Our experiments in
Part II demonstrated the degree to visual-spatial system is engaged in mental sorting and
concurrent counting. In particular Chapter 3 demonstrated involvement of ocular system
in mental tasks.
This discussion suggests that manipulation of access to visual-spatial resources may
eect the performance of intellectual tasks specially when the task heavily relies on fast
random access to the the content of working memory. Concurrent counting tasks in
which some sensory events should be mapped onto internal representations which hold
state of events are suitable for inducing such criteria. Thus for testing our hypothesis we
used a self-paced version of concurrent counting task previously introduced by Garavan
(Garavan, 1998). This paradigm allows study of execution times (or reaction time) in
response to manipulation of item presentation process and has been previously used to
study the eect of switching between memory items in several studies (Garavan, 1998;
Garavan et al., 2000; Gehring et al., 2003; Oberauer, 2003; Voigt and Hagendorf, 2002).
We devised two versions of the same paradigm for concurrent triple-counting of visual
or spatial targets. Having two version of the same paradigm allowed the use of error rates
in two parallel counting tasks in addition to execution times.
In Garavan's study and other replications of its paradigm the main variable is sim-
ilarity of two working memory items in two consecutive operational steps; in no-switch
condition two consecutive operations are carried on a single object of working memory
(e.g. the same internal counters is updated consecutively) and in switch condition two
consecutive operations are carried out on two dierent working memory objects. A faster
execution time in no-switch condition compared to switch condition has been consistently
reported (Garavan, 1998; Oberauer, 2003; Voigt and Hagendorf, 2002). Several studies
have tried to pinpoint the source of the dierent cost of no-switch and switch conditions
and have ruled out perceptual priming (Garavan, 1998) or rescheduling the rehearsal of
working memory items or retrieval of items (Oberauer, 2003). The cost of shifting the
focus of attention from one item in working memory to another item which presumably is
169
not necessary for no-switch condition has been considered as the likely source of observed
time course dierence between switch and no-switch conditions.
In our study we adopted the concept of self-paced concurrent-counting from Garavan's
study and modied it into two dierent versions: an identity-based triple-counting (IBTC)
and a location-based triple-counting (LBTC) paradigm. During location-based counting,
subjects ignored identity of items and attended only to locations of target presentation for
identifying counting events so that appearance of two consecutive targets (independent
of visual identity of targets) at same locations can be considered as no-switch condition
and presenting two targets in dierent locations can be considered as switch condition.
This created a congruency between shift of attention between spatial locations in the
external domain and internal working memory representations. Contrary to location-
based counting in identity-based counting subjects needed to attend to the type or identity
of visual targets and had to ignore the location of presentation. This condition results
to an incongruity between switching condition in external space and internal working
memory domain.
The congruous switching condition allows binding internal counters to the internal
representation of locations in presentation space and this in turn facilitates ecient ran-
dom access to items of working memory through a direct binding to locations of target
presentation. Our account predicts that the incongruous switching condition in identity-
based counting will preclude the use of the same visual-spatial representation for internal
and external representation which in turn results to higher cost of switching and less
accuracy by forcing subjects to either use their serially accessible resources or less ef-
cient random access resources. In contrast a modality-neutral account would predict
a dierence which is determined by diculty of target discrimination or target-counter
association.
We will show that even in terms of a very liberal measure for counting error which
discounts for any mistake in distinguishing counting events and confusing the order of
counters during counting or reporting, the identity-based counting is remarkably less
170
accurate than location-based counting. Moreover, subjects were remarkably slower in
switch condition for identity based compared to no-switch condition. In our second
experiment we use visual targets of dierent categories to test whether observed eects
are a result of possible target-counter binding and specic to visual items of our rst
experiment. In our last experiment we asked subjects to attend to the identity of targets
during a version of location-based counting. We show that attending to the identity of
targets has no eect on the cost of location-based counting and thus the eect of attending
to and processing the identity of visual targets can be ruled out as the source of the cost
of identity-based counting.
8.2 Experiments
8.2.1 Experiment 1
8.2.1.1 Apparatus
Stimuli were displayed on a 46-inch LCD monitor (Sony Bravia XBR-III,89cm 50cm),
97.8 cm in front of participants (corresponding eld of view is 54:7

32:65

). To control
the viewing distance, subjects used a chin rest to maintain their head position during the
experiment. A gray background (0.62 cd/m2) was displayed during the experiment.
8.2.1.2 Procedure
Figure 8.1 depicts a schematic view of the triple counting paradigm in which subjects
were supposed to concurrently maintain the count of three kinds of events. An event was
dened as the brief presentation of a visual target in one of three visible boxes on the
screen. In each paradigm three specic visual items used as visual targets. Upon subject's
key press one of images would appear for 500 milliseconds in one of three boxes. The
order of displaying visual items and their display locations were selected independently
and randomly. With 50% of the chance the same visual item presented in two consecutive
171
0
1
1
%
2
#:1
?:0
%:1
#:3
?:4
%:3
5
4
Reporting counters:
(self paced)
Presenting initial counters
(self paced)
Location-Based
Triple Counting
Identity-Based
Triple Counting
9 stimulus presentation events:
(self paced)
Presenting initial counters
(self paced)
Reporting counters:
(self paced)
Figure 8.1: Schematic view of the triple-counting paradigm for the identity-based and location-
based counting.
events, also, with 50% of the chance visual items of two consecutive events were presented
in the same box.
Each event would add 1 to an associated internally maintained counter. Each trial
consisted of nine self-initiated events. In each trial, each of counters started either from
0 or 1. The assignment of initial counters to events was random. Each trial started with
presentation of initial counts and ended with reporting the value of the counters using a
computer keyboard.
In each trial boxes centered at vertices of a virtual square with edges subtended an
angle of 3:5

from subject's view point. In each trial three out of four possible boxes were
172
selected randomly and remained on screen throughout the counting process. The edge of
presentation boxes subtended approximately 1:0

from subject's view point.
Our counting paradigms were either identity-based or location-based. In an identity-
based trial dierent visual items dened dierent counting events and the location of
presentation was irrelevant. In contrast, in a location based trial, dierent locations
dened dierent counting events and the identity of the visual item was irrelevant. So, in
identity-based triple-counting (IBTC), counters were associated to the identity of visual
items and represented the number of times each of items appeared on the screen plus
the initial value of the associated counter. In contrast, in location-based triple counting
(LBTC) each counter was associated to one of boxes and represented the number of times
that visual targets appeared in that box in addition to its initial counter.
Trials of identity-based counting started with presentation of initial counters for each
of visual items, one item at at time, in random order, at the center of screen and in
a self-paced way. Trials of location-based counting started with presentation of initial
counters at center of each box, in random order and in a self-paced way. At the end of
each trial subjects reported the counters using the keyboard, in the same order of initial
counter presentation and in a self-paced way.
The self-paced design of the paradigm which was adopted from Garvan's experi-
mental design (Garavan, 1998) allowed measurement of execution times. Given that
we controlled for location and identity of stimuli, except rst events ( which were ex-
cluded from analyses of execution times), other events classied into four categories
based on similarity of the identity and the location of their associated target presen-
tation with that of their preceding event. For each subject counting events (except
the rst one in each trial) were pooled together in these four categories to calculate
the mean execution times: location:same-identity:same, location:same-identity:changed,
location:changed-identity:same and location:changed-identity:changed.
In our rst experiment three keyboard characters ( $, # or ? ) were used as visual
targets for both IBTC and LBTC. So presentation of counting events would look exactly
173
similar in both IBTC and LBTC although subjects would attend to dierent aspects of
the stimulus presentation relevant to the the task. The only dierence in trials of LBTC
and IBTC was related to presentation of initial counters and reporting nal counts.
In this experiment each session consisted of blocks of ve trials of each of two alter-
native paradigms. Subjects were informed about the type counting task in a block by
display of a written message before the rst trial of the block. The message would stay
on the screen until a key-press. This resulted to a self-paced transition between blocks
which could be as short as 500 milliseconds. Each subject performed between 20 to 30
blocks which left us between 50 to 75 trials of each counting paradigm. At most 10 blocks
were performed in each session. A ve minute break administered between each two con-
secutive sessions. Six subjects started with a block of LBTC. Subjects performed a short
training session including two blocks of two dierent paradigms and two trails per block.
8.2.1.3 Subjects
Seven female and four male USC undergraduate students with normal or corrected to
normal vision, participated for course credit. Participants' ages ranged from 19 to 21
years (M = 19:7;SD = 0:78).
8.2.1.4 Results
We quantied counting errors using four dierent measures: a. the proportion of trials
that had at least one mistake in the reported counters (Trial Error), b. the absolute
dierence between the reported values and the actual values (Counter Mean Error), c.
the absolute dierence between reported values and actual values, after sorting both
the counters and the reported values (Sorted Counter Mean Error) and d. the absolute
dierence between the sum of reported values and the sum of actual counters (Count Sum
Error). Among these four measures, a. and b. are the most sensitive measures, while
c. discounts any error in incorrectly reporting the order of counters, and d. is the least
174
Error Metric
Task Trial Err % Counts Mean Err Sorted Mean Err Count Sum Err
3* Exp 1.
LBTC 20:5% 5:4% 0:11 0:03 0:10 0:03 0:24 0:08
IBTC (characters) 39:3% 7:1% 0:31 0:07 0:22 0:04 0:40 0:09
signicance ** ** ** *
4* Exp 2.
IBTC (characters) 43:2% 5:8% 0:36 0:07 0:26 0:05 0:51 0:1
IBTC (shapes) 39:6% 6:4% 0:29 0:05 0:21 0:04 0:48 0:09
IBTC (objects) 42:4% 7:4% 0:29 0:06 0:23 0:05 0:47 0:09
signicance n.s. n.s n.s. n.s.
3* Exp 3.
LBTC 19:1% 3:9% 0:11 0:02 0:08 0:02 0:21 0:05
IC-LBTC 18:1% 4:1% 0:10 0:03 0:07 0:02 0:17 0:04
signicance n.s. n.s n.s. n.s.
LBTC: Location-Based Triple Counting; IBTC: Identity-Based Triple Counting; IC-LBTC: Identity-Controlled LBTC
:p< 0:05, :p< 0:01 ,n.s. : non-signicant
Table 8.1: Mean SE of error measures for all three experiments.
sensitive measure as it does not account for any error in adding to the right counter or
in reporting the counters in the correct order.
The top section of Table 8.1 shows mean standard error values for each of these
error measures for two types of concurrent triple counting. To assess the signicance of
the eect of the type of the counting task on each of these four measures, the data of
each error measure was submitted to a separate within-subject one-way ANOVA.
A signicant main eect of the counting type on the error rates was revealed for all
four measures of error. The signicance of this impact on the rst measure is quantied
by (F (1; 10) = 20;p = 0:001)); for the absolute dierence, this signicance is quantied
by F (1; 10) = 15:4;p = 0:0028; for the absolute dierence in sorted sequence of counters
and reported values, the signicance is quantied by (F (1; 10) = 17:2;p = 0:00197); and
nally for the absolute dierence between the sum of counters and the sum of reported
values, the signicance of the main eect is quantied by (F (1; 10) = 9:11;p = 0:0129).
175
Moreover, further analysis revealed that with all measures for counting error, counting
location-based events was more accurate than counting identity-based events.
Figure 8.3 shows the average execution time for each of paradigms and in four dif-
ferent categories of events. For the sake of simplicity, in our analyses the relevance of a
feature to the counting task was taken into account for grouping factors in the analysis of
variance. Hence, in LBTC location was treated as counting-relevant feature and identity
as counting-irrelevant feature while in IBTC location was treated as counting-irrelevant
feature and identity as the counting-relevant feature.
A three-way ANOVA was used to investigate possible eects of type of counting
task, similarity of counting-relevant feature and similarity of counting-irrelevant feature.
Execution times were submitted to a 222 three-way within subjects ANOVA with two
levels for the the type of the task (LBTC or IBTC), two levels for similarity of counting-
relevant features in two consecutive counting events (same or changed) and two levels
for similarity of counting-irrelevant features in two consecutive counting events (same or
changed).
The analysis revealed a signicant main eect of type of counting task ( F (1; 10) =
16:4;p = 0:0024 ) with an average longer execution time for identity-based triple count-
ing (1641m:s: 165m:s: (MeanSEM)) compared to location-based triple counting
(1210m:s:104m:s (MeanSEM)). On average subjects were 432m:s:107m:s: (Mean
SEM) faster in counting location-based counting events compared to identity-based
counting events.
Also, a signicant main eect of dierence in counting-relevant feature was revealed
( F (1; 10) = 171:4;p = 0:00011 ). Further analysis showed that execution time of an
event was longer when the counting-relevant feature was dierent from that of its pre-
ceding event (1840m:s: 152m:s: (MeanSEM)), compare to when two consecutive
events shared same counting relevant features (1012m:s: 106m:s: (MeanSEM)),
so that changing the relevant feature between two consecutive counting events incurred
176
827m:s: 210m:s: (MeanSEM) extra execution time. Meanwhile a signicant in-
teraction between dierence in counting-relevant feature and type of the counting task
observed (F (1; 10) = 24:14;p = 0:00061). Further analysis showed that execution times
during IBTC were more aected by switching the counting-relevant feature. The average
dierence in execution times for IBTC was 956m:s:80m:s: (MeanSEM) while during
LBTC this dierence was 694m:s: 53m:s: (MeanSEM).
No signicant main eect of dierence in counting-irrelevant feature was observed
( F (1; 10) = 1:9;p = 0:195 ). Also no signicant interaction between dierence in
counting-irrelevant feature with type of the tasks was observed (F (1; 10) = 0:001;p =
0:795) which means changing the identity in LBTC and changing the location in IBTC
had no signicant eect on the execution time. Also no signicant interaction be-
tween dierence in counting-relevant feature and counting-irrelevant feature was observed
(F (1; 10) = 1:008;p = 0:25). The interaction between three factors was also insignicant
(F (1; 10) = 0:14;p = 0:74).
8.2.1.5 Discussion
Previous studies that employed Garavan's self-paced paradigm were focused only on time
course of the task execution in particular the time dierence between switch and no-
switch conditions. In both versions of studied triple-counting tasks in this experiment,
similarity of counting-relevant features in two consecutive target presentations determined
whether the same or a dierent counter should be updated. Thus the observed signicant
main eect of counting-relevant feature on execution times in both paradigms of this
experiment is consistent with previous reports (Garavan, 1998; Oberauer, 2003) of the
eect of switching between objects of working memory on the execution time.
Moreover we observed that counting with location cues is faster than counting with
identity cues. This is also consistent with Oberauer's previous report that locational
cuing for arithmetic operation on several items results to faster execution compared to
color cuing. In Oberauer's study several items were maintained for sequential arithmetic
177
(addition or subtraction). In one paradigm operands were presented at the center of screen
and were cued by the color in which operand appeared. In locational cuing operands
appeared in separate boxes. Oberauer reported that \ instead of adding costs for shifts
of visual attention, using spatial positions to identify items and operations made the task
easier ". He attributed this unexpected result to possible relative advantage of location
cuing to color cuing and did no give further analysis on the signicance of dierence in
cuing method on time dierence between switch and no-switch condition. In our study
we see a similar eect in comparing location cuing and character-identity cuing. In
particular our paradigm removed any dierence in target presentation and thus shift in
spatial attention during target perception in both paradigms were similar and any eect
can be attributed to the dierence between cuing method.
Yet the striking nding was that, this eect was more pronounced in the case of
switch condition so that no-switch execution time of IBTC was about 300 m.s. longer
than no-switch execution time of LBTC while execution time in switch condition for
IBTC was 550 m.s. longer than LBTC's. This 250 m.s. dierence rstly shows that the
time dierence between no-switch and switch condition can not be exclusively explained
by rescheduling a similar rehearsal process for maintaining the internal counters in two
paradigms: the sequence of target presentations in both paradigms were identical and
even the interference of a possible shift in spatial attention for target identication is
similar and thus rescheduling a verbal rehearsal of internal counters can not account for
this 250 m.s. time dierence. Moreover, this interaction between counting paradigm
shows that spatial cuing remarkably facilitates the shift between counters which needs to
be explained.
Yet another striking result of this experiment is related to the result of analysis of
counting errors for both counting paradigms.Even with a measure as liberal as error in
sum of counters which discounts for adding to the right counter or reporting counters
in correct order LBTC revealed to be signicantly more accurate. Given that execution
times during LBTC were also shorter compared to IBTC our ndings about error rates
178
shows that slower execution during IBTC compared to LBTC does not buy more accuracy
and thus speed-accuracy trade-o can not explain the discrepancy in performance of
subjects in two dierent concurrent counting tasks. This in turn may suggest that both
execution times and counting errors are manifestation of a unitary source of cost dierence
in two paradigms.
8.2.2 Experiment 2
Results of our rst experiments showed relative advantage of location-based counting in
terms of both speed and accuracy. One may suggest that the higher cost of identity-
based counting in our rst experiment might be related to the category of targets and
thus identity-specic. The category of items may aect the eciency of the working
memory task in dierent ways. For example diculty in discriminating visual targets
or rarity of items may cause less ecient target-counter binding. Another possibility is
related to automatic processes which is related to readability of visual targets in the rst
experiment which might cause a Stroop-like interference with counting process. To test
possible eects of type of stimuli in our rst experiment we used visual targets of dierent
visual categories and compared them counting characters of the rst experiment.
In this experiment we asked our subjects to perform a shape-based and an object-based
along with a character-based IBTC in alternative blocks. Including the character-based
IBTC similar to experiment 1 allowed us to examine the eect of switching between
IBTC and LBTC in consecutive blocks by performing between-subject analyses. Three
commonly used physical objects (a cup, a fork and a key) with easily distinguishable
shapes were used for object-based counting.
8.2.2.1 Subjects
9 female and 3 male USC undergraduate students with normal or corrected to normal
vision participated for course credit. Participants' ages ranged from 18 to 34 years (M =
21:2;SD = 4:2).
179
8.2.2.2 Procedure
The same set of characters as experiment 1 were used for our character-based IBTC trials
(CB-IBTC). A triangle, a circle and a square all lled-in with black were used for the
shape-based IBTC (SB-IBTC). Line drawings of a mug, a fork and a key were used for
the object-based IBTC (OB-IBTC). Visual presentation of items and task specications
were similar to the IBTC of experiment 1.
Five trials of each type of IBTC formed a block. A xed sequence of three types of
IBTC was administered for each subject. The sequence of block types to each subject was
assigned randomly and we balanced for the number of subjects for each possible sequence
of blocks. Each subject performed 7 to 9 blocks of each type of IBTC which left us with
35 to 45 trials of each type.
Similar to the previous experiment the transition between blocks was self-paced and
by display of a written message about the number of trials and type of items for counting
in the upcoming block. Subjects stated the experiment by a short training session of one
block for each counting type including two trials.
8.2.2.3 Results
Figure 8.4 shows the average execution time for all possible combinations of change in
location and identity of visual targets in two consecutive presentations for all three ver-
sions of IBCT. The eects of type of items, similarity of identities and locations of two
consecutive visual targets on execution times were investigated by submitting the data
to a 3 2 2 within-subjects three-way ANOVA with corresponding factors.
The analysis returned only a signicant main eect of change in the identity of two
consecutive visual presentations. This change is associated to switching between two
counters which again proved to have signicant eect on the execution time (F (1; 11) =
57:58;p < 0:00001) with average faster execution time when items in two consecutive
presentation shared same identity (1108m:s: 101m:s:(MeanSEM)) compared to
when identities were dierent (1898m:s: 159m:s:(MeanSEM)).
180
In average subjects were slightly faster in counting objects compared to shapes (with
average dierence of 79m:s: 42m:s:(MeanSEM)) or characters (with average dier-
ence of 105m:s: 51m:s:(MeanSEM)), however these dierences were not signicant
(F (2; 22) = 2:839;p = 0:133). Moreover no signicant interaction between type of count-
ing task and change in identity of items in two consecutive presentations was observed
(F (2; 22) = 2:17;p = 0:175). This suggests that the switching cost between dierent
counters was not signicantly dierent in counting dierent kinds of items.
The middle section of Table 8.1 shows average standard error of mean for dierent
measures of error and for three dierent categories of counting items. The data of four
measures of error were submitted to separate one-way within-subject ANOVAs to test the
signicance of the eect of category of items on error rates. None of analyses returned
a signicant main eect of the category of items on error values. The least notable
dierence was related to the error in sum of counters with F (2; 22) = 0:328;p = 0:753
and the most notable dierence was related to the mean error in counter values with
F (2; 22) = 2:82;p = 0:081.
To explore possible impacts of the cost of switching between dierent counting paradigms
on the cost of performing IBTC, execution time and error data of counting characters
in experiment 2 and IBTC of experiment 1 were submitted to between-subjects analy-
ses of variance. A 2 2 2 between-subjects three-way ANOVA revealed no signicant
main eect of the identity of experiment (F (1; 84) = 0:69;p = 0:41). Also no signicant
interaction between the identity of experiment and the similarity of characters in two con-
secutive stimulus presentations observed (F (1; 84) = 0:345;p = 0:56) suggesting the cost
of switching between two counters was not signicantly dierent in counting characters
in experiments 1 and 2.
Also the data for each of error measures was separately submitted to four dierent
one-way between-subjects ANOVA with the identity of the experiment as the main factor
(experiment 1 vs. experiment 2 as its two levels). None of these analyses showed a
signicant main eect of the experiment identity. The most signicant result belonged
181
to to error in sum of counters with F (1; 21) = 0:706;p = 0:40. The least signicant
result among four dierent measures of errors belonged to the the proportion of trials
with errors with F (1; 21) = 0:187;p = 0:67.
Our last between-subjects analyses suggest that performing LBTC or other types of
IBTC along with triple counting of our set of characters did not aect the performance
of IBTC with characters and thus the high cost of IBTC relative to LBTC in experiment
1 is not likely to be eect of switching between two paradigms.
As the analysis on execution time revealed, switching between dierent versions of
IBTC did not show two signicant eects that were observed in our rst experiment: the
eect of the type of the task and interaction between type of the tasks and its interaction
with changing the counting-relevant feature in two consecutive task. This suggests that
the slower execution time of IBTC relative to LBTC is not likely to be the result of type
of items that were used for counting task.
8.2.3 Experiment 3
Our previous experiment showed that the relative disadvantage in IBTC compared to
LBTC is not likely to be related to the the identity of items used in IBTC or the cost
of switching between two dierent paradigms. Yet one may argue that the dierence in
cost of IBTC and LBTC might be an epiphenomenon of engaging dierent perceptual
mechanisms associated to the the target identication stage of the task in two dierent
paradigms. This in particular may sound relevant given that processing visual forms and
visual locations are suggested to engage dierent neural mechanisms. One may argue
that these two dierent systems for processing visual forms and visual locations may
draw on general domain cognitive resources needed for intellectual part of the counting
task dierently. One may even refer to signicant impact of processing identities on
performing counting task as an evidence for dependency of identication of visual items
on executive resources.
182
To test whether engaging in identication of visual forms is the source of disadvantage
in IBTC we modied our LBTC paradigm and designed an Identity-Controlled Location-
Based Triple-Counting (IC-LBTC) paradigm in which subjects were supposed to count
location-based events while paying attention to the identity of targets. We asked our
subjects to perform blocks of IC-LBTC along with blocks of LBTC to be able to measure
the cost of engaging in identity discrimination on location-based triple counting on the
basis of a within subject analysis. Moreover, similar to our previous experiment we
performed between subjects analysis for LBTC in this experiment and LBTC in the
rst experiment to explore those costs associated to switching between an identity-based
paradigm and a location-based paradigm.
8.2.3.1 Subjects
Seven female and four male USC undergraduate students with normal or corrected to
normal vision participated for course credit. Participants' ages ranged from 19 to 22
years (M = 20:39;SD = 1:4).
8.2.3.2 Procedure
Two of three keyboard characters of previous experiments (# and % ) were used as targets
of location-based counting in LBTC and IC-LBTC. The third character was used only
in IC-LBTC as a dummy character which would appear occasionally and at most once
in a trial. Upon its appearance in a box the counter for that box had to be reported as
zero. With 50% of the chance a trial of IC-LBTC would contain the dummy character
and subjects did not know about the appearance of the dummy character in advance.
Appearance of the dummy character would change the rule of counting and would change
the task to a simpler dual-counting task. Thus during IC-LBTC trials subjects required
to attend to the identity of targets in addition to performing a location-based triple-
counting task. If attending to the identity of items in IBTC is the source of extra cost
compared to LBTC then one would expect slower execution time and higher error rates
183
for IC-LBTC compared to LBTC. Approximately in half of trials of IC-LBTC subjects
counted all nine events which were exclusively included in our analysis. The reported
counter value of boxes of dummy character in trials of IC-LBTC with display of dummy
character were used to test whether subjects attended to the identity of visual targets in
IC-LBTC trials.
Since trials of LBTC and IC-LBTC looked exactly similar, to assure that subjects
are fully aware of the change in their task, at the and of each block, the written message
about the type counting task would stay on the screen for at least 10 seconds after which
could disappear by a key press. This imposed at least 10 seconds delay between blocks.
The experiment was arranged in sessions of alternative blocks of two types of tasks.
Each block consisted of ve trials. Each subject performed between 20 to 30 trials which
left us between 50 to 75 trials of each counting paradigm. Six subjects started their
experiment with a block of LBTC. Subjects also performed one short block of each of
counting tasks including two trials in each block for the training which were excluded
from all analyses.
8.2.3.3 Results
Figure 8.5 shows the average execution time for all possible combinations of change in
location and identity of visual targets in both LBTC and IC-LBTC trials with full triple-
counting trials. The execution time data was submitted to a 2 2 2 three-way, within-
subjects ANOVA. Three main factors included type of the task (two levels: LBTC and
IC-LBTC), similarity of locations in two consecutive stimulus presentation (two levers:
same or changed) and similarity of identity of two consecutive stimulus presentations
(two levels: same or changed). The analysis returned only a signicant main eect of
similarity of locations in two consecutive stimulus presentations which in both paradigms
is associated to switching between counters with F (1; 10) = 53:9;p = 0:00002.
In particular the main eect of type of the task was highly insignicant [F (1; 10) =
0:011;p = 0:92]. However, a marginally-signicant eect of the similarity of items in
184
two consecutive stimulus presentations was observed [F (1; 10) = 4:13;p = 0:07]. This
eect was in particular more pronounced when items presented in dierent locations. In
this case subjects, in average were 75m:s: 31m:s: slower when two consecutive items
were dierent.This eect might be related to the fact that subjects were also attending
to the identity of items. In particular since the interaction between similarity of identity
of targets and type of counting tasks was highly insignicant [F (1; 10) < 0:001;p
0:994] this shows that subjects were probably attending to the identity of items in both
LBTC and IC-LBTC. A separate 2 2 within subject ANOVA on execution time when
target location changed revealed that a signicant eect of similarity of identities in two
consecutive target presentation [F (1; 10) = 6:04;p = 0:034] while there was no signicant
eect of counting task [F (1; 10) = 0:015;p = 0:90] with no signicant interaction between
two factors [F (1; 10) =;p = 0:89].
Except for one subject, all subjects correctly reported the incidence of appearance of
the dummy character with 100% accuracy. The average error in reporting a non-zero for
the dummy contained box was 1:1% 1:1% while the mean value for the total trial error
for dummy presented trials was 11:7% 4% which shows subjects were attending to the
identity of targets during IC-LBTC trials.
The bottom section of Table 8.1 shows the average value for dierent measures of
error for both counting tasks in experiment 3. The data for all measures of error were
separately submitted to within-subject one-way ANOVAs with type of counting task as
the main factor. None of the analyses returned a signicant main eect of the counting
paradigm on the error rates. The least notable dierence between error measures was
related to the Counter Error which was determined by [F (1; 10)0:097;p = 0:76] and the
most notable dierence between error measures was related to the Sum Error which was
quantied by [F (1; 10) = 1:49;p = 0:25].
To assess the eect of rule switching context on the performance of LBTC we applied
some between-subjects analyses on the execution time and error data of LBTC trials
of experiment 1 and 3. The execution time data for LBTC from experiment 1 and
185
experiment 3 were submitted to a 222 between-subjects analysis of variance with the
identity of experiment, similarity of location and similarity of identity of two consecutive
visual targets as three factors. The analysis revealed a signicant main eect of the
experiment [F (1; 80) = 10:27;p = 0:002]. Further analysis revealed that subjects were
faster in performing LBTC in experiment 3 with average execution time of 984 m:s:
81m:s:(MeanSEM) compared to LBTC in experiment 1 with average execution time of
1210m:s:103m:s:. The main eect of similarity of locations in two consecutive stimulus
presentation (which associated to switching between two counters in two consecutive
update) was also signicant [F (1; 80) = 75:36;p< 0:0001]. No signicant main eect of
similarity of items in two consecutive stimulus presentations was observed [F (1; 80) =
0:23;p = 0:63] . None of interactions were also signicant.
The data from all measures of error were submitted to separate between-subjects one-
way ANOVA with the identity of experiment as the main factor for all analyses. None
of the analyses showed a signicant main eect with least notable eect on proportion of
trials with error [F (1; 20) = 0:05;p = 0:82] and the most notable eect for count errors
of sorted counters [F (1; 20) = 0:26;p = 0:62]. So, although the switching context had a
signicant eect on execution time for counting LBTC yet it had no signicant impact
on the accuracy of counting.
8.2.3.4 Discussion
As our results suggest the type of the counting task in this experiment had virtually no
eect on the performance of our subjects. One may argue that subjects in this experiment
might have employed the same counting strategy for both LBTC and IC-LBTC and could
have considered the identity of items in LBTC task to minimize the cost of switching
between two counting rules. In fact our analysis suggest that when location of target
changed, the change of identity added a signicant 70 m.s. 30 30 m.s. delay to
execution time which was almost similar for both LBTC and IC-LBTC.
186
Nevertheless, comparing LBTC in experiments 1 and 3 showed that in fact LBTC in
experiment 3 was signicantly faster than experiment 1. If similarity of performance in
LBTC and IC-LBTC in experiment 3 was the result of attending to the identity of items
in both counting paradigms then attending to the identity of items would not explain the
extra cost of IBTC compared to LBTC in experiment 1.
The shorter execution time of LBTC in experiment 3 compared to experiment 1
might also be related to the fact that subjects used the same strategy for performing
LBTC and IC-LBTC in the second experiment which might have saved on the cost of
switching between counting rules. Also, applying an extra 10 second delay between blocks
of experiment 3. might have helped lower the cost of switching between counting rules
in consecutive blocks.
With respect to all error measures, none of our analyses revealed a signicant dier-
ence between performing LBTC or IC-LBTC in experiment 3 and LBTC in experiment 3
and 1. Taken together, the results of our analysis on execution times and error rates sug-
gest that attending to the identity of items during a location-based triple counting task
did not result to any cost which is comparable to the cost of identity-based triple-counting
tasks.
8.3 General Discussion
The chief goal of the present study was investigating mechanisms that may play a role in
random selection of items in working memory. In particular we were interested in test-
ing proposed hypothesis about the important role of visual-spatial systems in supporting
random accessibility to the content of working memory for fast and reliable mental oper-
ations.
We devised two paradigms for a concurrent triple-counting task with similar sequence
of target presentations in both versions. In one of paradigms which counting events were
187
dened based on location of target presentation accessing to a visual-spatial represen-
tation of external presentation screen was facilitated through congruency of switching
attention between counting target presentation and switching between internal counters:
shifting spatial attention in presentation screen would require a shift between counters.
According to SWMS model this may help utilization of visuospatial short-term memory
which feeds into sensory-motor system for visually guided actions for directly binding
the counters to the location of presentation. This in turn may help utilization of this
addressable space for binding and retrieving residents of working memory when needed.
In contrast in the other paradigms in which counting events were dened based on
identity of targets incongruous condition of shifting focus of attention between counting
objects and shifting spatial attention in visual-spatial domain would preclude the use of
visual space for binding which in turn would force subjects to utilize other mechanisms
for access to memory items triggered by random presentation of cues.
In Garavan's process model for a dual-counting task, each counting event consists of
a sequence of ve steps: 1. stimulus identication, 2. orientation of attention, 3. updating
the associated count, 4. rehearsing the other count, 5. key-press. He suggests that the
source of a 300 to 400 msec dierence in updating the same counter subsequently versus
updating two dierent counters is related to the cost of the second step: a recently
attended resident of the working memory is privileged in terms of processing speed or
accuracy (Garavan, 1998), and thus updating a counter which was just updated saves on
the cost of bringing the item of working memory into the focus.
This model could be adopted for the triple-counting task by considering a third counter
which needs to be included in the switching and the rehearsing steps. This model does not
assume that the second step of this process, which accounts for the extra switching cost
between two dierent counts, is dependent on perceptual aspects of the counting tasks.
Likewise, no other model of working memory, to our knowledge, in which the focus of
attention plays a critical functional role in regulating the process, assumes that the second
and the third steps of this process are relevant to the perceptual aspects of the counting
188
task. Hence, according to this process model, steps 2, 3 and 5 should be independent of the
type of counting events. Our second experiment controlled for the in
uence of potential
eects of perceptual dierences, and showed that the source of speed dierence in two
paradigms cannot be attributed to the perception of events. Consequently, according to
this model, the only source of dierence in counting speed might be in rehearsing other
counts (step 4). However, this eect would aect the speed of counting in a similar
way for both updating the same counter or updating counters alternatively. Moreover,
the analysis of errors adds another dimension to our argument: even when misplaced
counters and signals are discounted in the error calculation, the location-based counting
is still signicantly more accurate. In sum, we argue that a model that confers a special
role to space (unlike Garavan's process model), may be necessary to fully explain our
ndings.
The proposed spatial registry framework for manipulation of information in working
memory is an example of a model where space plays a special role. According to our
model, which assumes that working memory items are bound to spatial locations, access-
ing items in the internal domain draws on shifting spatial attention to dierent internal
representation of locations. It is explicit in our model that the internal representation
that supports manipulation of information for the intellectual task utilizes a type of short-
term memory with an interface to external physical environment. We argued that the
main function of these systems is supporting object-centered actions in the physical envi-
ronment and thus those internal representation need to be mapped onto and register with
the representation of the external world. It was based upon a similar assumption that we
searched for traces of mental sorting in eye movements in our eye-tracking experiments
(see Chapter 3).
This means that incongruous shift in external attention and shift between internal
registry locations during identity-based triple counting would add to the cost of working
management. This extra cost might be related to establishing additional registry location
for maintenance of counters in the case that subject follows a visual-spatial strategy. So,
189
Execution Time in milliseconds ( meanSEM )
Locations: Same Locations: Same Locations: Changed Locations: Changed
Task Identity: Same Identity: Changed Identity: Same Identity: Changed
2* Exp 1.
LBTC 834 77 889 102 1551 136 1564 115
IBTC (characters) 1126 135 2122 191 1198 143 2120 206
3* Exp 2.
IBTC (characters) 1084 107 1985 185 1185 132 1935 165
IBTC (shapes) 1090 79 1927 207 1218 123 1856 149
IBTC (objects) 1053 106 1845 172 1030 86 1852 120
2* Exp 3.
LBTC 719 68 716 71 1214 105 1285 105
IC-LBTC 724 71 710 60 1204 113 1285 105
LBTC: Location-Based Triple Counting; IBTC: Identity-Based Triple Counting; IC-LBTC: Identity-Controlled LBTC
Table 8.2: Mean SE of execution times for all three experiments.
in this case once a target is located at the location of one of boxes the a shift of attention
to a dierent location is required. This shift should be planed based upon current location
of spatial attention and the location of associated registry. However, even in the case of
wrong retrieval the error should not appear in the sum of counter values.
One may argue that this extra cost may encourage the use of a serial access strategy for
keeping track of three variables. Maintaining three variables by utilizing the phonological
system requires a span of 6 items including word forms for three digits and three character
or object name. With regard to high cost of target identication in the phonological loop
and the increment cost this strategy is highly volatile. In fact subjects with very low
accuracy and very slow execution pace often reported the use of a phonological strategy.
Another option is utilizing other sensorimotor systems for random access as a re-
placement for visual-spatial system. In fact during our experiments we observed that
a substantial number of subjects would utilize their body parts such as their ngers or
limbs for maintaining numbers. These subjects were among high performance subjects
and would try dierent strategies before settling with a nal strategy. We documented
three of these instances which varied in the way that body parts were used for tracking
and accessing numbers.
190
%
%
#
?
Location-Based Counting Identity-Based Counting
% : 1
# : 0
? : 1
% : 2
# : 0
? : 1
1 0
1
% : 3
# : 0
? : 1
% : 3
# : 1
? : 1
?
% : 3
# : 1
? : 2
% : 3
# : 1
? : 3
2 0
1
2 1
1
2 2
1
3 2
1
4 2
2
Initial
Counts
Initial
Counts
no-switch
no-switch
switch
switch
switch
no-switch
switch
no-switch
start
Δ T
1
Δ T
3
Δ T
2
Δ T
4
Figure 8.2: Same sequence of stimulus presentation results to dierent sequences of counting
events with two dierent counting paradigms. Those events that require updating a recently
updated counter are considered as `no-switch' and those that result to updating the same counter
are considered as `switch' events.
191
=
Location
=
Character
500
1,000
1,500
2,000
2,500
Execution Time (m.s.)
Location-Based Triple Counting
= Location
≠ Character
≠ Location
= Character
≠ Location
≠ Character
= Location
= Character
= Location
≠ Character
≠ Location
= Character
≠ Location
≠ Character
Identity-Based Triple Counting
Figure 8.3: Average execution time for location-based triple counting (LBTC) and identity-
based triple counting (IBTC) (experiment 1). A = next to a feature denotes same feature during
two consecutive stimulus presentations while a6= denotes that two consecutive presentation were
dierent with respect to that feature. Counting events preceded by a dierent counting-relevant
feature are signicantly slower compared to when two consecutive events share the same counting-
relevant features. This eect is signicantly more pronounce in the case of IBTC compared to
LBTC.
192
500
1,000
1,500
2,000
2,500
Execution Time (m.s.)
= Location
= Object
= Location
≠ Object
≠ Location
= Object
≠ Location
≠ Object
= Location
= Shape
= Location
≠ Shape
≠ Location
= Shape
≠ Location
≠ Shape
= Location
= Character
= Location
≠ Character
≠ Location
= Character
≠ Location
≠ Character
Identity-Based Triple Counting
(Characters)
Identity-Based Triple Counting
(Shapes)
Identity-Based Triple Counting
(Objects)
Figure 8.4: Average execution time for three versions of identity-based triple counting (with
characters, shapes and objects) (experiment 2). Counting objects was slightly and yet insigni-
cantly faster than both other counting types. Yet unlike the location-based triple counting even
this slight advantage is independent of whether two consecutive items were the same or dierent.
193
500
1,000
1,500
2,000
2,500
Execution Time (m.s.)
Location-Based Triple Counting Identity-Based Triple Counting
=
Location
=
Character
= Location
≠ Character
≠ Location
= Character
≠ Location
≠ Character
=
Location
=
Character
= Location
≠ Character
≠ Location
= Character
≠ Location
≠ Character
Figure 8.5: Average execution time for two versions of location-based triple counting (experiment
3). During identity-controlled location-based triple counting (IC-LBTC) subjects attented to the
identity of items in addition to counting location-based events, however, this did not in
uence the
execution time.
194
Bibliography
Phillip L Ackerman. Individual dierences in information processing: An investigation of
intellectual abilities and task performance during practice. Intelligence, 10(2):101{139,
1986.
Phillip L Ackerman and Eric D Heggestad. Intelligence, personality, and interests: evi-
dence for overlapping traits. Psychological bulletin, 121(2):219, 1997.
Phillip L Ackerman, Margaret E Beier, and Mary O Boyle. Working memory and intel-
ligence: The same or dierent constructs? Psychological bulletin, 131(1):30, 2005.
Elkan G Aky urek and Bernhard Hommel. Memory operations in rapid serial visual
presentation. European Journal of Cognitive Psychology, 18(4):520{536, 2006.
Elkan G. Aky urek, Bernhard Hommel, and Pierre Jolicur. Direct evidence for a role of
working memory in the attentional blink. Memory & Cognition, 35(4):621{627, 2007.
T.P. Alloway. How does working memory work in the classroom? Educational Research
and Reviews, 1(4):134{139, 2009.
John R Anderson. Act: A simple theory of complex cognition. American Psychologist,
51(4):355, 1996.
M.A. Arbib. Interacting schemas for motor control. Tutorials in motor behavior, pages
71{81, 1980.
M.A. Arbib. Levels of modeling of mechanisms of visually guided behavior. Behavioral
and Brain Sciences, 10(03):407{436, 1987.
M.A. Arbib. Schema theory. Encyclopedia of articial intelligence, 2:1427{1443, 1992.
M.A. Arbib. From monkey-like action recognition to human language: An evolutionary
framework for neurolinguistics. Behavioral and brain sciences, 28(02):105{124, 2005.
M.A. Arbib. Mirror system activity for action and language is embedded in the integration
of dorsal and ventral pathways. Brain and language, 112(1):12{24, 2010.
M.A. Arbib and J.Y. Lee. Describing visual scenes: Towards a neurolinguistics based on
construction grammar. Brain research, 1225:146{162, 2008.
M.A. Arbib and J.S. Liaw. Sensorimotor transformations in the worlds of frogs and
robots. Articial Intelligence, 72(1):53{79, 1995.
195
M.A. Arbib, J. Hill, and J. Conklin. From schema theory to language. Oxford University
Press, 1987.
Michael A Arbib. The handbook of brain theory and neural networks. Bradford Book,
2003.
Michael A Arbib. How the brain got language: the mirror system hypothesis, volume 16.
Oxford University Press, 2012.
Richard C Atkinson and Richard M Shirin. Human memory: A proposed system and
its control processes. The psychology of learning and motivation, 2:89{195, 1968.
E Awh, J Jonides, and P A Reuter-Lorenz. Rehearsal in spatial working memory. Journal
of Experimental Psychology. Human Perception and Performance, 24(3):780{790, June
1998. ISSN 0096-1523. PMID: 9627416.
Edward Awh, John Jonides, Edward E. Smith, Richard B. Buxton, Larry R. Frank,
Tracy Love, Eric C. Wong, and Leon Gmeindl. Rehearsal in Spatial Working Memory:
Evidence From Neuroimaging. Psychological Science, 10(5):433{437, 1999.
A. Baddeley. Working memory. Science, 255(5044):556, 1992. ISSN 0036-8075.
A. Baddeley. Working memory and language: an overview. Journal of communication
disorders, 36(3):189{208, 2003a.
A. Baddeley. Working Memory: Theories, Models, and Controversies. 2012.
A Baddeley, N Thomson, and M Buchanan. Word length and the structure of short-term
memory. Journal of Verbal Learning and Verbal Behavior, 14(6):575{589, December
1975. ISSN 00225371. doi: 10.1016/S0022-5371(75)80045-4.
A. Baddeley, V. Lewis, and G. Vallar. Exploring the articulatory loop. The Quarterly
Journal of Experimental Psychology, 36(2):233{252, 1984.
AD Baddeley. Working memory. Philosophical Transactions of the Royal Society of
London. B, Biological Sciences, 302(1110):311{324, 1983.
A.D. Baddeley and G. Hitch. Working memory. The psychology of learning and motiva-
tion, 8:47{89, 1974.
Alan Baddeley. Exploring the Central Executive. The Quarterly Journal of Experimental
Psychology Section A: Human Experimental Psychology, 49(1):5, 1996. ISSN 0272-4987.
doi: 10.1080/713755608.
Alan Baddeley. Is Working Memory Still Working?, 2002.
Alan Baddeley. Working memory: looking back and looking forward. Nat Rev Neurosci,
4(10):829{839, October 2003b. ISSN 1471-003X. doi: 10.1038/nrn1201.
Alan Baddeley and Pierre Salam e. The unattended speech eect: Perception or memory?
Journal of Experimental Psychology: Learning, Memory, and Cognition, 12(4):525,
1986.
196
Alan Baddeley and Barbara Wilson. Phonological coding and short-term memory in
patients without speech. Journal of Memory and Language, 24(4):490{502, 1985.
Alan D Baddeley. Short-term memory for word sequences as a function of acoustic,
semantic and formal similarity. The Quarterly Journal of Experimental Psychology, 18
(4):362{365, 1966.
Alan D Baddeley, C. Papagno, and G. Vallar. When long-term learning depends on
short-term storage. Journal of Memory and Language, 27(5):586{595, 1988.
D H Ballard, M M Hayhoe, P K Pook, and R P Rao. Deictic codes for the embodiment
of cognition. The Behavioral and Brain Sciences, 20(4):723{742; discussion 743{767,
December 1997. ISSN 0140-525X. PMID: 10097009.
Dana H. Ballard, Mary M. Hayhoe, and Je B. Pelz. Memory Representations in Natural
Tasks. Journal of Cognitive Neuroscience, 7(1):66{80, April 2011. ISSN 0898-929X.
Aron K Barbey, Roberto Colom, Erick J Paul, and Jordan Grafman. Architecture of

uid intelligence and working memory revealed by lesion mapping. Brain Structure
and Function, pages 1{10, 2013.
F C Bartlett. Remembering: A study in experimental and social psychology, volume 14.
Cambridge University Press, 1932.
Daniel B Berch. Working memory and mathematical cognitive development: limitations
of limited-capacity resource models. Developmental neuropsychology, 33(3):427{446,
2008.
T.J. Bireta, S.E. Fry, A. Jalbert, I. Neath, A.M. Surprenant, G. Tehan, and G.A. Tolan.
Backward recall and benchmark eects of working memory. Memory & cognition, 38
(3):279{291, 2010.
Sarah-J Blakemore, Daniel M Wolpert, and Chris D Frith. Central cancellation of self-
produced tickle sensation. Nature neuroscience, 1(7):635{640, 1998.
F.L. Bookstein. Principal warps: thin-plate splines and the decomposition of deforma-
tions. IEEE Transactions on Pattern Analysis and Machine Intelligence, 11(6):567{585,
1989. ISSN 01628828.
Michael S Brainard and Allison J Doupe. Auditory feedback in learning and maintenance
of vocal behaviour. Nature Reviews Neuroscience, 1(1):31{40, 2000.
G.D.A. Brown, I. Neath, and N. Chater. A temporal ratio model of memory. Psychological
Review, 114(3):539, 2007.
Rebecca Bull, Kimberly Andrews Espy, and Sandra A Wiebe. Short-term memory,
working memory, and executive functioning in preschoolers: Longitudinal predictors
of mathematical achievement at age 7 years. Developmental neuropsychology, 33(3):
205{228, 2008.
197
Brian Butterworth. The development of arithmetical abilities. Journal of Child Psychol-
ogy and Psychiatry, 46(1):3{18, 2005.
John B Carroll. Human cognitive abilities. Cambridge University Press Cambridge, 1993.
John B Carroll. A three-stratum theory of intelligence: Spearmans contribution. Human
abilities: Their nature and measurement, pages 1{17, 1996.
Christopher H. Chatham, Seth A. Herd, Angela M. Brant, Thomas E. Hazy, Akira
Miyake, Randy O'Reilly, and Naomi P. Friedman. From an Executive Network to
Executive Control: A Computational Model of the n-back Task. Journal of Cognitive
Neuroscience, 23(11):3598{3619, October 2011. ISSN 0898-929X.
Yu-Chin Chiu and Steven Yantis. A Domain-Independent Source of Cognitive Control
for Task Sets: Shifting Spatial Attention and Switching Categorization Rules. The
Journal of Neuroscience, 29(12):3930 {3938, March 2009.
Carol L Colby and Michael E Goldberg. Space and attention in parietal cortex, 1999.
Reuben Conrad. Acoustic confusions in immediate memory. British journal of Psychology,
55(1):75{84, 1964.
C. Constantinidis and X.J. Wang. A neural circuit basis for spatial working memory. The
Neuroscientist, 10(6):553{565, 2004.
N Cowan. Activation, attention, and short-term memory. Memory & Cognition, 21(2):
162{167, March 1993. ISSN 0090-502X. PMID: 8469124.
N. Cowan. An embedded-processes model of working memory. Models of working memory:
Mechanisms of active maintenance and executive control, pages 62{101, 1999.
Nelson Cowan. The Magical Number 4 in Short-Term Memory: A Reconsideration of
Mental Storage Capacity. Behavioral and Brain Sciences, 24(01):87{114, 2001.
Fergus IM Craik and Robert S Lockhart. Levels of processing: A framework for memory
research. Journal of verbal learning and verbal behavior, 11(6):671{684, 1972.
Clayton E Curtis and Mark D'Esposito. Persistent activity in the prefrontal cortex during
working memory. Trends in cognitive sciences, 7(9):415{423, 2003.
M. Daneman and P.A. Carpenter. Individual dierences in working memory and reading.
Journal of verbal learning and verbal behavior, 19(4):450{466, 1980.
Meredyth Daneman and Philip M Merikle. Working memory and language comprehen-
sion: A meta-analysis. Psychonomic Bulletin & Review, 3(4):422{433, 1996.
Stanislas Dehaene and Laurent Cohen. Cultural Recycling of Cortical Maps. Neuron, 56
(2):384{398, 2007. ISSN 0896-6273.
Stanislas Dehaene, Serge Bossini, and Pascal Giraux. The mental representation of parity
and number magnitude. Journal of Experimental Psychology General, 122:371{371,
1993.
198
N. Derbinsky and J.E. Laird. Extending Soar with dissociated symbolic memories. In
Proc. of the Symposium on Human Memory for Articial Agents. AISB, pages 31{37,
2010.
M. D'Esposito, G. K. Aguirre, E. Zarahn, D. Ballard, R. K. Shin, and J. Lease. Functional
MRI studies of spatial and nonspatial working memory. Cognitive Brain Research, 7
(1):1{13, 1998. ISSN 0926-6410.
M. D'Esposito, B. R. Postle, D. Ballard, and J. Lease. Maintenance versus Manipulation
of Information Held in Working Memory: An Event-Related fMRI Study. Brain and
Cognition, 41(1):66{86, October 1999. ISSN 0278-2626.
M D'Esposito, B R Postle, and B Rypma. Prefrontal cortical contributions to working
memory: evidence from event-related fMRI studies. Experimental Brain Research. Ex-
perimentelle Hirnforschung. Exp erimentation C er ebrale, 133(1):3{11, July 2000. ISSN
0014-4819. PMID: 10933205.
Mark D'Esposito. From cognitive to neural models of working memory. Philosophical
Transactions of the Royal Society B: Biological Sciences, 362(1481):761{772, May 2007.
Mark D'Esposito, John A. Detre, David C. Alsop, Robert K. Shin, Scott Atlas, and
Murray Grossman. The neural basis of the central executive system of working memory.
1995.
Howard Ehrlichman and Dragana Micic. Why do people move their eyes when they think?
Current Directions in Psychological Science, 21(2):96{100, 2012.
Howard Ehrlichman and Arthur Weinberger. Lateral eye movements and hemispheric
asymmetry: A critical review. Psychological Bulletin, 85(5):1080, 1978.
Howard Ehrlichman, Dragana Micic, Amber Sousa, and John Zhu. Looking for answers:
Eye movements in non-visual cognitive tasks. Brain and cognition, 64(1):7{20, 2007.
Randall W. Engle. Working Memory Capacity as Executive Attention. Current Directions
in Psychological Science, 11(1):19{23, February 2002. ISSN 09637214. ArticleType:
primary article / Full publication date: Feb., 2002 / Copyright 2002 Association for
Psychological Science.
Randall W Engle, Stephen W Tuholski, James E Laughlin, and Andrew RA Conway.
Working memory, short-term memory, and general
uid intelligence: a latent-variable
approach. Journal of Experimental Psychology: General, 128(3):309, 1999.
R.W. Engle, J.J. Carullo, and K.W. Collins. Individual dierences in working memory for
comprehension and following directions. The Journal of Educational Research, pages
253{262, 1991.
Maddalena Fabbri-Destro and Giacomo Rizzolatti. Mirror neurons and mirror systems
in monkeys and humans. Physiology, 23(3):171{179, 2008.
199
Luciano Fadiga, Laila Craighero, Giovanni Buccino, and Giacomo Rizzolatti. Speech lis-
tening specically modulates the excitability of tongue muscles: a tms study. European
Journal of Neuroscience, 15(2):399{402, 2002.
Thomas Fangmeier, Markus Knau, Christian C Ru, and Vladimir Sloutsky. fmri ev-
idence for a three-stage model of deductive reasoning. Journal of Cognitive Neuro-
science, 18(3):320{334, 2006.
Stefano Ferraina, Alexandra Battaglia-Mayer, Aldo Genovesio, Philippe Archambault,
and Roberto Caminiti. Parietal encoding of action in depth. Neuropsychologia, 47(6):
1409{1420, 2009. ISSN 0028-3932.
S.E. Fienberg. A brief history of statistics in three and one-half chapters: a review essay,
1992.
Martin H Fischer, Nele Warlop, Robin L Hill, and Wim Fias. Oculomotor bias induced
by number perception. Experimental Psychology, 51(2):91{97, 2004. ISSN 1618-3169.
John E Fisk and Charles A Sharp. The role of the executive system in visuo-spatial
memory functioning. Brain and Cognition, 52(3):364{381, 2003.
Daryl Fougnie and Ren e Marois. Executive working memory load induces inattentional
blindness. Psychonomic Bulletin & Review, 14(1):142{147, 2007a. ISSN 1069-9384.
Daryl Fougnie and Ren e Marois. Executive working memory load induces inattentional
blindness. Psychonomic Bulletin & Review, 14(1):142{147, February 2007b. ISSN
1069-9384. PMID: 17546744.
M. J. Frank, B. Loughry, and R. C. O'Reilly. Interactions between frontal cortex and
basal ganglia in working memory: A computational model. Cognitive, Aective, &
Behavioral Neuroscience, 1(2):137{160, June 2001. ISSN 1530-7026.
JAM Frederiks. Disorders of the body schema. Handbook of clinical neurology, 4:207,
1969.
S. Funahashi, C. J. Bruce, and P. S. Goldman-Rakic. Mnemonic coding of visual space in
the monkey's dorsolateral prefrontal cortex. J Neurophysiol, 61(2):331{349, Feb 1989.
Vittorio Gallese, Luciano Fadiga, Leonardo Fogassi, and Giacomo Rizzolatti. Action
recognition in the premotor cortex. Brain, 119(2):593{609, 1996.
Francis Galton. Inquiries into human faculty and its development. Macmillan and Com-
pany, 1883.
Hugh Garavan. Serial attention within working memory. Memory & Cognition, 26(2):
263{276, March 1998. ISSN 0090-502X.
Hugh Garavan, TJ Ross, S-J Li, and EA Stein. A parametric manipulation of central
executive functioning. Cerebral Cortex, 10(6):585{592, 2000.
200
Susan E Gathercole and Graham J Hitch. Developmental changes in short-term memory:
A revised working memory perspective. Theories of memory, 1:189{209, 1993.
William J Gehring, Richard L Bryck, John Jonides, Roger L Albin, and David Badre. The
mind's eye, looking inward? In search of executive control in internal attention shifting.
Psychophysiology, 40(4):572{585, July 2003. ISSN 0048-5772. PMID: 14570165.
B.K. Gerton, T.T. Brown, A. Meyer-Lindenberg, P. Kohn, J.L. Holt, R.K. Olsen, and
K.F. Berman. Shared and distinct neurophysiological components of the digits forward
and backward tasks as revealed by functional neuroimaging. Neuropsychologia, 2004.
J Gl ascher, D Rudrauf, R Colom, LK Paul, D Tranel, H Damasio, and R Adolphs. Dis-
tributed neural system for general intelligence revealed by lesion mapping. Proceedings
of the National Academy of Sciences, 107(10):4705{4709, 2010.
Jan Gl ascher, Daniel Tranel, Lynn K Paul, David Rudrauf, Chris Rorden, Amanda Hor-
naday, Thomas Grabowski, Hanna Damasio, and Ralph Adolphs. Lesion mapping of
cognitive abilities linked to intelligence. Neuron, 61(5):681{691, 2009.
D.W. Glasspool. Serial order in behaviour: Evidence from performance slips. Connec-
tionist models in cognitive psychology, page 241, 2005.
gocognitive. Alan baddeley on the cognitive revolution. http://www.youtube.com/
watch?v=wyfEETtWgCY, October 2010. Retrieved 2013, October, 02.
P.S. Goldman-Rakic et al. Working memory and the mind. 1992.
Patricia S GoldmanRakic. Circuitry of Primate Prefrontal Cortex and Regulation of
Behavior by Representational Memory. Comprehensive Physiology, 1987.
Adam S. Greenberg, Michael Esterman, Daryl Wilson, John T. Serences, and Steven
Yantis. Control of Spatial and Feature-Based Attention in Frontoparietal Cortex. The
Journal of Neuroscience, 30(43):14330 {14339, October 2010.
I. C Grin and A. C Nobre. Orienting attention to locations in internal representations.
Journal of Cognitive Neuroscience, 15(8):1176{1194, 2003. ISSN 0898-929X.
Raquel E Gur. Conjugate lateral eye movements as an index of hemispheric activation.
Journal of Personality and Social Psychology, 31(4):751, 1975.
Mary Sue Hamann and Mark H Ashcraft. Textbook presentations of the basic addition
facts. COGNITION AND INSTRUCTION, 3(3):173{192, 1986.
D.Z. Hambrick and R.W. Engle. The role of working memory in problem solving. The
psychology of problem solving, pages 176{206, 2003.
Sang-Hoon Han and Min-Shik Kim. Visual Search Does Not Remain Ecient When
Executive Working Memory Is Working, 2004.
201
Takashi Hanakawa, Manabu Honda, Tomohisa Okada, Hidenao Fukuyama, and Hiroshi
Shibasaki. Neural correlates underlying mental calculation in abacus experts: a func-
tional magnetic resonance imaging study. Neuroimage, 19(2):296{307, 2003.
Thomas E Hazy, Michael J Frank, and Randall C O' Reilly. Banishing the homunculus:
making working memory work. Neuroscience, 139(1):105{118, 2006.
Jibo He and Jason S. McCarley. Executive working memory load does not compromise
perceptual processing during visual search: Evidence from additive factors analysis.
Attention, Perception, & Psychophysics, 72(2):308{316, 2010.
Lucy A Henry. The eects of word length and phonemic similarity in young children's
short-term memory. The Quarterly Journal of Experimental Psychology, 43(1):35{52,
1991.
R. N. A. Henson, N. Burgess, and C. D. Frith. Recoding, storage, rehearsal and grouping
in verbal short-term memory: an fMRI study. Neuropsychologia, 38(4):426{440, April
2000. ISSN 0028-3932. doi: 10.1016/S0028-3932(99)00098-6.
Richard J Herrnstein and Charles Murray. Bell curve: Intelligence and class structure in
American life. SimonandSchuster. com, 2010.
Yukikazu Hidaka, Cheol E Han, Steven L Wolf, Carolee J Winstein, and Nicolas
Schweighofer. Use it and improve it or lose it: Interactions between arm function
and use in humans post-stroke. PLoS computational biology, 8(2):e1002343, 2012.
Merrill Hiscock and Kathleen J Bergstrom. Ocular motility as an indicator of verbal and
visuospatial processing. Memory & Cognition, 9(3):332{338, 1981.
Graham J Hitch, M Sebastian Halliday, Alison Dodd, and Janet E Littler. Development
of rehearsal in short-term memory: Dierences between pictorial and spoken stimuli.
British Journal of Developmental Psychology, 7(4):347{362, 1989.
Y. Hoshi, I. Oda, Y. Wada, Y. Ito, Y. Yamashita, M. Oda, K. Ohta, Y. Yamada, and
M. Tamura. Visuospatial imagery is a fruitful strategy for the digit span backward
task: a study with near-infrared optical tomography. Cognitive Brain Research, 9(3):
339{342, 2000.
Edward M Hubbard, Manuela Piazza, Philippe Pinel, and Stanislas Dehaene. Interactions
between number and space in parietal cortex. Nature Reviews. Neuroscience, 6(6):435{
448, 2005. ISSN 1471-003X.
L. Itti, C. Koch, and E. Niebur. A model of saliency-based visual attention for rapid
scene analysis. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 20
(11):1254{1259, 1998. ISSN 0162-8828.
M Jahanshahi, G Dirnberger, R Fuller, and CD Frith. The role of the dorsolateral
prefrontal cortex in random number generation: a study with positron emission to-
mography. Neuroimage, 12(6):713{725, 2000.
202
William James. The principles of psychology. Digireads. com, 1890-2011.
Marc Jeannerod. The cognitive neuroscience of action, volume 1997. Blackwell Oxford,
1997.
Dylan M Jones and William J Macken. Irrelevant tones produce an irrelevant speech ef-
fect: Implications for phonological coding in working memory. Journal of Experimental
Psychology: Learning, Memory, and Cognition, 19(2):369, 1993.
Jeery A Jones and Kevin G Munhall. Perceptual calibration of f0 production: evidence
from feedback perturbation. The Journal of the Acoustical Society of America, 108:
1246, 2000.
John Jonides, Richard L. Lewis, Derek Evan Nee, Cindy A. Lustig, Marc G. Berman,
and Katherine Sledge Moore. The Mind and Brain of Short-Term Memory. Annual
Review of Psychology, 59(1):193{224, January 2008a. ISSN 0066-4308.
John Jonides, Richard L Lewis, Derek Evan Nee, Cindy A Lustig, Marc G Berman, and
Katherine Sledge Moore. The mind and brain of short-term memory. Annu. Rev.
Psychol., 59:193{224, 2008b.
Stanley A. Lorens Jr. and Chester W. Darrow. Eye movements, EEG, GSR and EKG
during mental multiplication. Electroencephalography and Clinical Neurophysiology, 14
(5):739{746, 1962. ISSN 0013-4694.
Michael J. Kane, M. Kathryn Bleckley, Andrew R. A. Conway, and Randall W. Engle. A
controlled-attention view of working-memory capacity. Journal of Experimental Psy-
chology: General, 130(2):169{183, 2001. ISSN 1939-2222. doi: 10.1037/0096-3445.130.
2.169.
Ruth Kanfer and Phillip L Ackerman. Motivation and cognitive abilities: An
integrative/aptitude-treatment interaction approach to skill acquisition. Journal of
applied psychology, 74(4):657, 1989.
N Kanwisher and E Wojciulik. Visual attention: insights from brain imaging. Nature
Reviews. Neuroscience, 1(2):91{100, 2000. ISSN 1471-003X. PMID: 11252779.
Liane Kaufmann. More evidence for the role of the central executive in retrieving arith-
metic facts{a case study of severe developmental dyscalculia. Journal of Clinical and
Experimental Neuropsychology, 24(3):302{310, 2002.
Mitsuo Kawato. Internal models for motor control and trajectory planning. Current
opinion in neurobiology, 9(6):718{727, 1999.
WL Kilmer, WS McCulloch, and J Blum. A model of the vertebrate central command
system. International Journal of Man-Machine Studies, 1(3):279{309, 1969.
M. Knau, J. Kassubek, T. Mulack, and M.W. Greenlee. Cortical activation evoked by
visual mental imagery as measured by fMRI. NeuroReport, 11(18):3957, 2000.
203
Markus Knau, Reinhold Rauh, Christoph Schlieder, and Gerhard Strube. Mental models
in spatial reasoning. In Spatial cognition, pages 267{291. Springer, 1998.
Markus Knau, Thomas Mulack, Jan Kassubek, Helmut R Salih, and Mark W Green-
lee. Spatial imagery in deductive reasoning: a functional mri study. Cognitive Brain
Research, 13(2):203{212, 2002.
Markus Knau, Thomas Fangmeier, Christian C Ru, and PN Johnson-Laird. Reasoning,
models, and images: Behavioral measures and cortical activity. Journal of cognitive
neuroscience, 15(4):559{573, 2003.
Andre Knops, Bertrand Thirion, Edward M. Hubbard, Vincent Michel, and Stanislas De-
haene. Recruitment of an Area Involved in Eye Movements During Mental Arithmetic.
Science, 324(5934):1583{1585, 2009.
Michael Koenigs, Aron K. Barbey, Bradley R. Postle, and Jordan Grafman. Superior
Parietal Cortex Is Critical for the Manipulation of Information in Working Memory.
J. Neurosci., 29(47):14980{14986, 2009.
Stephen Michael Kosslyn. Image and mind. Harvard University Press, 1980. ISBN
9780674443662.
Patricia K Kuhl. Learning and representation in speech and language. Current opinion
in neurobiology, 4(6):812{822, 1994.
J.E. Laird, A. Newell, and P.S. Rosenbloom. Soar: An architecture for general intelligence.
Articial intelligence, 33(1):1{64, 1987.
K. Lajoie, J. E. Andujar, K. Pearson, and T. Drew. Neurons in area 5 of the posterior
parietal cortex in the cat contribute to interlimb coordination during visually guided
locomotion: a role in working memory. Journal of neurophysiology, 103(4):2234{2254,
2010.
G.J. Larrabee and R.L. Kane. Reversed digit repetition involves visual and verbal pro-
cesses. International journal of neuroscience, 30(1-2):11{15, 1986.
Denny C LeCompte. Irrelevant speech, serial rehearsal, and temporal distinctiveness:
A new approach to the irrelevant speech eect. Journal of Experimental Psychology:
Learning, Memory, and Cognition, 22(5):1154, 1996.
Anthony Leonardo and Masakazu Konishi. Decrystallization of adult birdsong by per-
turbation of auditory feedback. Nature, 399(6735):466{470, 1999.
S.C. Li and S. Lewandowsky. Forward and backward recall: Dierent retrieval processes.
Journal of Experimental Psychology: Learning, Memory, and Cognition, 21(4):837,
1995a.
Shu-Chen Li and Stephan Lewandowsky. Forward and Backward Recall: Dierent Re-
trieval Processes,. Journal of Experimental Psychology: Learning, Memory, and Cog-
nition, 21(4):837{847, July 1995b. ISSN 0278-7393.
204
Tobias Loetscher, Christopher J Bockisch, Michael ER Nicholls, and Peter Brugger. Eye
position predicts what number you have in mind. Current Biology, 20(6):R264{R265,
2010.
R.H. Logie, K.J. Gilhooly, and V. Wynn. Counting on working memory in arithmetic
problem solving. Memory & cognition, 22(4):395{410, 1994.
Robert H. Logie, Gesualdo M. Zucco, and Alan Baddeley. Interference with visual short-
term memory. Acta Psychologica, 75(1):55{74, October 1990. ISSN 0001-6918.
S Jr Lorens and C Darrow. Eye movements, EEG, GSR and EKG during mental mul-
tiplication. Electroencephalography and Clinical Neurophysiology, 14(5):739{746, 1962.
ISSN 00134694. doi: 10.1016/0013-4694(62)90088-3.
Steven J. Luck and Edward K. Vogel. The capacity of visual working memory for features
and conjunctions. Nature, 390:279{281, November 1997.
Donald J Mabbott and Jerey Bisanz. Computational skills, working memory, and con-
ceptual knowledge in older children with mathematics learning disabilities. Journal of
Learning Disabilities, 41(1):15{28, 2008.
Michael McCloskey. Cognitive mechanisms in numerical processing: Evidence from ac-
quired dyscalculia. Cognition, 44(1):107{157, 1992.
Michael McCloskey, Alfonso Caramazza, and Annamaria Basili. Cognitive mechanisms
in number processing and calculation: Evidence from dyscalculia. Brain and cognition,
4(2):171{196, 1985.
Michael McCloskey, Donna Aliminosa, and Scott M Sokol. Facts, rules and procedures
in normal calculation: Evidence from multiple single-patient studies of impaired arith-
metic fact retrieval. Brain and Cognition, 17(2):154{203, 1991.
Teresa McCormack, Gordon D. A. Brown, Janet I. Vousden, and Richard N. A. Henson.
Children's Serial Recall Errors: Implications for Theories of Short-Term Memory De-
velopment. Journal of Experimental Child Psychology, 76(3):222{252, July 2000. ISSN
0022-0965. doi: 10.1006/jecp.1999.2550.
B Mcelree. Accessing Recent Events. In Psychology of Learning and Motivation, vol-
ume 46, pages 155{200. Elsevier, 2006. ISBN 0125433468.
B. McElree and B.A. Dosher. Serial position and set size in short-term memory: The
time course of recognition. Journal of Experimental Psychology: General, 118(4):346,
1989.
Brian McElree. Attended and Non-Attended States in Working Memory: Accessing
Categorized Structures. 1998.
Brian McElree. Working memory and focal attention. Journal of Experimental Psychol-
ogy: Learning, Memory, and Cognition, 27(3):817{835, 2001. ISSN 1939-1285.
205
J.F. McLean and G.J. Hitch. Working memory impairments in children with specic
arithmetic learning diculties. Journal of Experimental Child Psychology, 74(3):240{
260, 1999.
D.A. McVea and K.G. Pearson. Object avoidance during locomotion. Progress in Motor
Control, pages 293{315, 2009.
D.A. McVea, A.J. Taylor, and K.G. Pearson. Long-lasting working memories of obstacles
established by foreleg stepping in walking cats require area 5 of the posterior parietal
cortex. The Journal of Neuroscience, 29(29):9396{9404, 2009.
D.E. Meyer and D.E. Kieras. A computational theory of executive cognitive processes
and multiple-task performance: Part I. Basic mechanisms. Psychological Review; Psy-
chological Review, 104(1):3, 1997.
G.A. Miller. The magical number seven, plus or minus two: some limits on our capacity
for processing information. Psychological review, 63(2):81, 1956.
G.A. Miller. The cognitive revolution: a historical perspective. Trends in cognitive
sciences, 7(3):141{144, 2003.
Akira Miyake and Priti Shah. Models of working memory: mechanisms of active mainte-
nance and executive control. Cambridge University Press, 1999. ISBN 9780521587211.
Akira Miyake, Naomi P Friedman, David A Rettinger, Priti Shah, and Mary Hegarty. How
are visuospatial working memory, executive functioning, and spatial abilities related?
a latent-variable analysis. Journal of Experimental Psychology: General, 130(4):621,
2001.
Stephen Monsell and Jon Driver. Control of cognitive processes: Attention and perfor-
mance, volume 18. The MIT Press, 2000.
DJ Murray. Articulation and acoustic confusability in short-term memory. Journal of
Experimental Psychology, 78(4p1):679, 1968.
Allen Newell and Herbert A Simon. Computer science as empirical inquiry: Symbols and
search. Communications of the ACM, 19(3):113{126, 1976.
Allen Newell, Herbert Alexander Simon, et al. Human problem solving, volume 14.
Prentice-Hall Englewood Clis, NJ, 1972.
M Nielsen. Gerstmann syndrome: Finger agnosia, agraphia, confusion of right and left
and acalculia: Comparison of this syndrome with disturbance of body scheme resulting
from lesions of the right side of the brain. Archives of Neurology and Psychiatry, 39
(3):536, 1938.
A. C Nobre, J. T Coull, P. Maquet, C. D Frith, R. Vandenberghe, and M. M Mesulam.
Orienting Attention to Locations in Perceptual Versus Mental Representations. Journal
of Cognitive Neuroscience, 16:363{373, April 2004. ISSN 0898-929X. doi: 10.1162/
089892904322926700. ACM ID: 1161440.
206
Nader Noori and Laurent Itti. Spatial Registry Model: Towards a Grounded Account
for Executive Attention. Proceedings of the 33rd Annual Conference of the Cognitive
Science Society, pages 3187{3192, 2011.
Kathy W Nordeen and Ernest J Nordeen. Auditory feedback is necessary for the main-
tenance of stereotyped song in adult zebra nches. Behavioral and neural biology, 57
(1):58{66, 1992.
Donald A Norman and Tim Shallice. Attention to Action: Willed and Automatic Control
of Behavior. In Consciousness and self-regulation, volume 4, pages 1{18. New York:
Plenum, 1986.
Klaus Oberauer. Access to information in working memory: exploring the focus of atten-
tion. Journal of Experimental Psychology. Learning, Memory, and Cognition, 28(3):
411{421, May 2002. ISSN 0278-7393. PMID: 12018494.
Klaus Oberauer. Selective attention to elements in working memory. Experimental Psy-
chology (formerly Zeitschrift f ur Experimentelle Psychologie), 50(4):257{269, 2003.
P.J. Olesen, H. Westerberg, and T. Klingberg. Increased prefrontal and parietal activity
after training of working memory. Nature Neuroscience, 7(1):75{79, 2004.
Ingrid R. Olson and Marian Berryhill. Some surprising ndings on the involvement of the
parietal lobe in human memory. Neurobiology of Learning and Memory, 91(2):155{165,
2009. ISSN 1074-7427. Special Issue: Parietal Cortex.
Randall C. O'Reilly. Biologically Based Computational Models of High-Level Cognition.
Science, 314(5796):91{94, October 2006. doi: 10.1126/science.1127242.
Randall C. O'Reilly and Michael J. Frank. Making Working Memory Work: A Com-
putational Model of Learning in the Prefrontal Cortex and Basal Ganglia. Neural
Computation, 18(2):283{328, February 2006. doi: 10.1162/089976606775093909.
Randall C O'Reilly, Todd S Braver, and Jonathan D Cohen. A biologically based com-
putational model of working memory. In Akira Miyake and Priti Shah, editors, Models
of Working Memory, chapter 11.
M. Osaka, M. Komori, M. Morishita, and N. Osaka. Neural bases of focusing attention
in working memory: An fMRI study based on group dierences. Cognitive, Aective,
& Behavioral Neuroscience, 7(2):130, 2007a. ISSN 1531-135X.
Mariko N. Osaka, Mie Komori, Masanao Morishita, and Naoyuki Osaka. Neural bases
of focusing attention in working memory: An fMRI study based on group dierences.
Cognitive, Aective, & Behavioral Neuroscience, 7(2):130{139, 2007b.
Jacques Paillard. Motor and representational framing of space. Brain and space, pages
163{182, 1991.
C. Papagno and G. Vallar. Phonological short-term memory and the learning of novel
words: The eect of phonological similarity and item length. The Quarterly Journal
of Experimental Psychology, 44(1):47{67, 1992.
207
E. Paulesu, C. D. Frith, and R. S. J. Frackowiak. The neural correlates of the verbal
component of working memory. Nature, 362(6418):342{345, March 1993. doi: 10.
1038/362342a0.
Mauro Pesenti, Marc Thioux, Xavier Seron, and Anne De Volder. Neuroanatomical
substrates of arabic number processing, numerical comparison, and simple addition: A
pet study. Journal of Cognitive Neuroscience, 12(3):461{479, 2000.
W. A. Phillips. Short-Term Visual Memory. Royal Society of London Philosophical
Transactions Series B, 302:295{308, August 1983.
Jean Piaget and Margaret Trans Cook. The origins of intelligence in children. 1952.
Manuela Piazza, V eronique Izard, Philippe Pinel, Denis Le Bihan, and Stanislas Dehaene.
Tuning curves for approximate numerosity in the human intraparietal sulcus. Neuron,
44(3):547{555, 2004.
Bradley R. Postle. Working Memory as an Emergent Property of the Mind and Brain.
Neuroscience, 139(1):23{38, April 2006. ISSN 0306-4522. PMID: 16324795 PMCID:
1428794.
Alexandre Pouget, Peter Dayan, and Richard Zemel. Information processing with popu-
lation codes. Nat Rev Neurosci, 1(2):125{132, November 2000. ISSN 1471-003X. doi:
10.1038/35039062.
Jonathan F Prather, S Peters, S Nowicki, and R Mooney. Precise auditory{vocal mirror-
ing in neurons for learned vocal communication. Nature, 451(7176):305{310, 2008.
Cathy J Price. The anatomy of language: a review of 100 fmri studies published in 2009.
Annals of the New York Academy of Sciences, 1191(1):62{88, 2010.
Zenon W Pylyshyn. Computation and cognition. Cambridge Univ Press, 1984.
Zenon W. Pylyshyn. Computing in cognitive science. Foundations of cognitive science,
pages 51{91, 1989.
JG Quinn. Irrelevant pictures in visual working memory. The Quarterly Journal of
Experimental Psychology: Section A, 49(1):200{215, 1996.
Javier Quintana and Joaquin M. Fuster. Spatial and Temporal Factors in the Role of
Prefrontal and Parietal Cortex in Visuomotor Integration. Cereb. Cortex, 3(2):122{132,
1993.
G. Repovs and A. Baddeley. The multi-component model of working memory: Explo-
rations in experimental cognitive psychology. Neuroscience, 139(1):5{21, April 2006.
ISSN 0306-4522.
M. Reuhkala. Mathematical skills in ninth-graders: Relationship with visuo-spatial abil-
ities and working memory. Educational Psychology, 21(4):387{399, 2001.
208
Daniel Richardson and Michael Spivey. Representation, space and Hollywood Squares:
looking at things that aren't there anymore. Cognition, 76(3):269{295, 2000.
Giacomo Rizzolatti and Michael A Arbib. Language within our grasp. Trends in neuro-
sciences, 21(5):188{194, 1998.
Giacomo Rizzolatti and Laila Craighero. The mirror-neuron system. Annu. Rev. Neu-
rosci., 27:169{192, 2004.
Giacomo Rizzolatti, Luciano Fadiga, Vittorio Gallese, and Leonardo Fogassi. Premotor
cortex and the recognition of motor actions. Cognitive brain research, 3(2):131{141,
1996.
T W Robbins, E J Anderson, D R Barker, A C Bradley, C Fearnyhough, R Henson, and
S R Hudson. Working memory in chess. Memory & Cognition, 24(1):83{93, January
1996. ISSN 0090-502X. PMID: 8822160.
Jerker R onnberg, Mary Rudner, and Martin Ingvar. Neural correlates of working memory
for sign language. Cognitive Brain Research, 20(2):165{182, 2004.
R.G. Rudel and M.B. Denckla. Relation of forward and backward digit repetition to
neurological impairment in children with learning disabilities. Neuropsychologia, 12(1):
109{118, 1974.
Pierre Salam e and Alan D Baddeley. Phonological factors in stm: Similarity and the
unattended speech eect. Bulletin of the Psychonomic Society, 1986.
Krish Sathian, Tony J Simon, Scott Peterson, Gargi A Patel, John M Homan, and
Scott T Grafton. Neural evidence linking visual object enumeration and attention.
Journal of Cognitive Neuroscience, 11(1):36{51, 1999.
Stefan Schaal and Nicolas Schweighofer. Computational motor control in humans and
robots. Current opinion in neurobiology, 15(6):675{682, 2005.
Stephen H Scott. Optimal feedback control and the neural basis of volitional motor
control. Nature Reviews Neuroscience, 5(7):532{546, 2004.
Reza Shadmehr and Sandro Mussa-Ivaldi. Biological learning and control: how the brain
builds representations, predicts events, and makes decisions. MIT Press, 2012.
Reza Shadmehr, Maurice A Smith, and John W Krakauer. Error correction, sensory
prediction, and adaptation in motor control. Annual review of neuroscience, 33:89{
108, 2010.
Priti Shah and Akira Miyake. The separability of working memory resources for spa-
tial thinking and language processing: an individual dierences approach. Journal of
Experimental Psychology: General, 125(1):4, 1996.
T Shallice and E K Warrington. Independent functioning of verbal memory stores: a
neuropsychological study. The Quarterly Journal of Experimental Psychology, 22(2):
261{273, May 1970. ISSN 0033-555X. PMID: 5431401.
209
Claude Elwood Shannon and Warren Weaver. A mathematical theory of communication,
1948.
Sarah Shomstein and Steven Yantis. Parietal Cortex Mediates Voluntary Control of
Spatial and Nonspatial Auditory Attention. The Journal of Neuroscience, 26(2):435
{439, January 2006.
E.E. Smith and J. Jonides. Working memory: A view from neuroimaging. Cognitive
psychology, 33:5{42, 1997.
Samuel J Sober and Michael S Brainard. Adult birdsong is actively maintained by error
correction. Nature neuroscience, 12(7):927{931, 2009.
C. Spearman. " general intelligence," objectively determined and measured. The Amer-
ican Journal of Psychology, pages 201{292, 1904.
C. Spearman. The abilities of man: Their nature and measurement. Kessinger Publishing,
1927-2006.
Dietrich Stout and Thierry Chaminade. Stone tools, language and the brain in human
evolution. Philosophical Transactions of the Royal Society B: Biological Sciences, 367
(1585):75 {87, January 2012.
X. Sun, X. Zhang, X. Chen, P. Zhang, M. Bao, D. Zhang, J. Chen, S. He, and X. Hu.
Age-dependent brain activation during forward and backward digit recall revealed by
fMRI. Neuroimage, 26(1):36{47, 2005.
Benjamin J. Tamber-Rosenau, Michael Esterman, Yu-Chin Chiu, and Steven Yantis.
Cortical Mechanisms of Cognitive Control for Shifting Attention in Vision and Working
Memory. Journal of Cognitive Neuroscience, 23(10):2905{2919, October 2011. ISSN
0898-929X.
Ofer Tchernichovski, Partha P Mitra, Thierry Lints, and Fernando Nottebohm. Dynamics
of the vocal imitation process: how a zebra nch learns its song. Science, 291(5513):
2564{2569, 2001.
J.J. Todd and R. Marois. Capacity limit of visual short-term memory in human posterior
parietal cortex. Nature, 428(6984):751{754, 2004.
John Tooby and Leda Cosmides. The psychological foundations of culture. The adapted
mind: Evolutionary psychology and the generation of culture, pages 19{136, 1992.
John Tooby and Leda Cosmides. Toward mapping the evolved functional organization of
mind and brain. The new cognitive neurosciences, pages 1167{1178, 2000.
Alan M Turing. On computable numbers, with an application to the entscheidungsprob-
lem. Proceedings of the London mathematical society, 42(2):230{265, 1936.
Alan M Turing. Computing machinery and intelligence. Mind, 59(236):433{460, 1950.
210
Marilyn L Turner and Randall W Engle. Is working memory capacity task dependent?
Journal of Memory and Language, 28(2):127{154, 1989.
GIUSEPPE Vallar et al. Spatial neglect, balint-homes' and gerstmann's syndrome, and
other spatial disorders. CNS spectrums, 12(7):527, 2007.
Sabine Voigt and Herbert Hagendorf. The role of task context for component processes
in focus switching. Psychologische Beitrage, 2002.
John Von Neumann. First Draft of a Report on the EDVAC. In Randell B., editor, The
Origins of Digital Computers, pages 383{92. Springer-Verlag, Berlin, 1982.
Lev S Vygotskij. Thought and language. MIT press, 2012.
Robin S Waldstein. Eects of postlingual deafness on speech production: implications
for the role of auditory feedback. The Journal of the Acoustical Society of America,
88:2099, 1990.
SCOVILLE W.B. and B MILNER. Loss of recent memory after bilateral hippocampal
lesions. Journal of Neurology, Neurosurgery, and Psychiatry, 20(1):11{21, February
1957. ISSN 0022-3050. PMID: 13406589.
N. Wiener. Cybernetics; or control and communication in the animal and the machine.
1948.
Margaret Wilson. The case for sensorimotor coding in working memory. Psychonomic
Bulletin & Review, 8(1):44{57, 2001.
Margaret Wilson and Karen Emmorey. A visuospatial phonological loop in working
memory: Evidence from american sign language. Memory & Cognition, 25(3):313{320,
1997.
Margaret Wilson and Karen Emmorey. A word length eect for sign language: Further
evidence for the role of language in structuring working memory. Memory & Cognition,
26(3):584{590, 1998.
Margaret Wilson and Karen Emmorey. The eect of irrelevant visual input on working
memory for sign language. Journal of Deaf Studies and Deaf Education, 8(2):97{103,
2003.
Guilherme Wood, Klaus Willmes, Hans-Christoph Nuerk, and Martin H. Fischer. On the
cognitive link between space and number: a meta-analysis of the SNARC eect, 2008.
G. F. Woodman and S J. LUCK. Visual search is slowed when visuospatial working
memory is occupied. Psychonomic Bulletin & Review, 11(2):269{274, April 2004.
G F Woodman, E K Vogel, and S J Luck. Visual search remains ecient when visual
working memory is full. Psychological Science: A Journal of the American Psychological
Society / APS, 12(3):219{224, May 2001. ISSN 0956-7976. PMID: 11437304.
211
Sarah S Wu, Meghan L Meyer, Uta Maeda, Valorie Salimpoor, Sylvia Tomiyama, David C
Geary, and Vinod Menon. Standardized assessment of strategy use and working mem-
ory in early mental arithmetic performance. Developmental neuropsychology, 33(3):
365{393, 2008.
A.L. Yarbus. Eye movements and vision. Plenum press, 1967.
Laure Zago and Nathalie Tzourio-Mazoyer. Distinguishing visuospatial working memory
and complex mental calculation areas within the parietal lobes. Neuroscience letters,
331(1):45{49, 2002.
Xinlin Zhou, Chuansheng Chen, Hongchuan Zhang, Gui Xue, Qi Dong, Zhen Jin, Lei
Zhang, Chunyan Peng, Hui Zhao, Yi Guo, et al. Neural substrates for forward and
backward recitation of numbers and the alphabet: A close examination of the role of
intraparietal sulcus and perisylvian areas. Brain research, 1099(1):109{120, 2006.
212

Abstract (if available)

Abstract The capacity of the brain in maintenance of task-relevant information over a short period of time is known to be crucial for performing a wide range of activities from low-level perception-action routines in lower animals to high-level intellectual tasks in human. Working memory is the common term that has been used among different communities to refer to the manifestation of this feature across different domains. However, despite this common agreement, dominant theoretical paradigms for describing working memory management systems in the domains of low-level/perception-action and and high-level/intellectual functions follow drastically different principles: embedded and distributed in the low-level domain, disembodied and centralized in the high-level domain. Given that the human cognitive system functions at both levels in different contexts simultaneously this question arises whether indeed there are two types of working memory systems running in parallel under two different operational principles in human brain or, a more parsimonious account can explain all different manifestations of working memory in all domains. ❧ In attempt to achieve a more parsimonious account, a theory is developed for functioning of working memory in the context of high-level and intellectual domain which accounts for information management during those tasks that feature symbolic information processing. This effort was partially motivated by theoretical inconsistencies and biological/evolutionary implausibility of standard models of working memory in cognitive psychology with centralized executive paradigm. ❧ The proposed framework demonstrates how novel assemblage of embedded schemas in existing sensorimotor systems may supply a system for management of symbolically represented sensory and motor information serving intellectual tasks. In the proposed framework, strategic and evolutionarily-constrained reuse of sensorimotor resources for management of respectively spatially-organized and temporally-sensitive information support random access and serial access schemas for management of symbolic information. Through grounding access schemas for management of symbolic information in sensorimotor systems we are able to predict ramifications of working memory management during the performance of mental tasks at behavioral and neural levels. A detailed example in applying this methodology in well-studied cases of forward and backward recall tasks will be presented with additional computational modeling and the results of simulations. ❧ Our systematic approach in mapping spatial/temporal characteristics of sensorimotor systems onto access modes provides a symbolic interface to other frameworks and architectures for describing the symbolically-intelligent mind. Proposed framework provides for the first time a neurally-grounded and sensorimotor-based account for management of symbolic information with embodied cognition prospects with opportunities for experimental validations and applications. ❧ Proposed theory offers ample opportunities for experimental validations and predicts novel sources of working memory for symbolic tasks. Moreover, through providing a mechanistic account for management of working memory, the proposed theory, defines a practical framework for optimization of working memory resources for performing mental tasks which has potential educational consequences in measuring IQ and optimal designing of mental operations. ❧ Mechanistic specifications of the model about details of interactions between functioning of visuospatial systems as the main supplier of random access to symbolic working memory in normal subjects has allowed a number of predictions that are validated in experimental studies. These studies have been able to generate significant result in support of the proposed theory.

Linked assets

University of Southern California Dissertations and Theses

Conceptually similar

PDF

Schema architecture for language-vision interactions: a computational cognitive neuroscience model of language use

PDF

Linking eyes to mouth: a schema-based computational model for describing visual scenes

PDF

Transfer learning for intelligent systems in the wild

PDF

Depth inference and visual saliency detection from 2D images

PDF

Theoretical and computational foundations for cyber‐physical systems design

PDF

Defending industrial control systems: an end-to-end approach for managing cyber-physical risk

PDF

Noradrenergic mechanisms of arousal-enhanced memory selectivity

PDF

Gap analysis: transition age youth educational resources for high school graduation and college and career readiness

PDF

Customized data mining objective functions

PDF

Behavior understanding from speech under constrained conditions: exploring sparse networks, transfer and unsupervised learning

PDF

Toward understanding speech planning by observing its execution—representations, modeling and analysis

PDF

Towards health-conscious spaces: building for human well-being and performance

PDF

Heart, brain, and breath: studies on the neuromodulation of interoceptive systems

PDF

Acculturation team-based clinical program: pilot program to address acculturative stress and mental health in the Latino community

PDF

Detecting joint interactions between sets of variables in the context of studies with a dichotomous phenotype, with applications to asthma susceptibility involving epigenetics and epistasis

Asset Metadata

Creator Noori, Nader (author)

Core Title The symbolic working memory system

School Viterbi School of Engineering

Degree Doctor of Philosophy

Degree Program Computer Science

Publication Date 12/04/2013

Defense Date 10/15/2013

Publisher University of Southern California (original), University of Southern California. Libraries (digital)

Tag brain,cognition,cognitive neuroscience,embodied working memory system,grounded working memory system,human brain,intellectual tasks,memory,mental task,OAI-PMH Harvest,sensorimotor systems,short-term memory,symbolic intelligence,symbolic working memory,symbolically intelligent mind,working memory

Format application/pdf (imt)

Language English

Contributor Electronically uploaded by the author (provenance)

Advisor Itti, Laurent (committee chair), Arbib, Michael A. (committee member), Aziz-Zadeh, Lisa (committee member)

Creator Email nader.noori@gmail.com

Permanent Link (DOI) https://doi.org/10.25549/usctheses-c3-356011

Unique identifier UC11295188

Identifier etd-NooriNader-2205.pdf (filename),usctheses-c3-356011 (legacy record id)

Legacy Identifier etd-NooriNader-2205.pdf

Dmrecord 356011

Document Type Dissertation

Format application/pdf (imt)

Rights Noori, Nader

Type texts

Source University of Southern California (contributing entity), University of Southern California Dissertations and Theses (collection)

Access Conditions The author retains rights to his/her dissertation, thesis or other graduate work according to U.S. copyright law. Electronic access is being provided by the USC Libraries in agreement with the a...

Repository Name University of Southern California Digital Library

Repository Location USC Digital Library, University of Southern California, University Park Campus MC 2810, 3434 South Grand Avenue, 2nd Floor, Los Angeles, California 90089-2810, USA