University of Southern California Dissertations and Theses

Perceptual and computational mechanisms of feature-based attention

(USC Thesis Other)

Perceptual and computational mechanisms of feature-based attention

PDF

Download a page range

Download transcript

Copy asset link

Request this asset

Transcript (if available)

Content PERCEPTUAL AND COMPUTATIONAL MECHANISMS OF
FEATURE-BASED ATTENTION
by
Jianwei Lu
A Dissertation Presented to the
FACULTY OF THE GRADUATE SCHOOL
UNIVERSITY OF SOUTHERN CALIFORNIA
In Partial Fulfillment of the
Requirements for the Degree
DOCTOR OF PHILOSOPHY
(NEUROSCIENCE)
December 2006
Copyright 2006 Jianwei Lu
Dedication
To my family and friends, who have always supported me.
ii
Acknowledgments
I would like to thank Laurent for being the best academic advisor during my life. Without
his advisement and help closely, it would be hard for me to make achievement during this
scientific research. He always encourages new ideas, helps closely in the detailed techniques,
helps to solve the difficulties in analyzing the data, and allows me time to learn fMRI and
programming skills. I really appreciate and enjoy his advisement style as a so nice advisor
who always encourages students to think and learn knowledge without any criticism and im-
patience. Under his advisement and help, I feel I’m improving myself step by step in every
aspect of being trained as a Ph.D. student in recent years. I thank Laurent to help in revising
the paper draft and improving my writing skills. He also encourages and allows me to get
a Master degree in Computer Science which let me get a general understanding of the most
basic computer science fundamentals at this moment. My understanding and training in the
science would definitely have large difference without him.
I would like to thank Dr. Irving Biederman and Dr. Peter Qin to continuously being as
my guidance and defense committee members. Their helpful advisement and advice during
my research, and openness and flexibility in attending my academic fairs really play the very
iii
important roles in helping me completing all the works. I appreciate them as the so nice mem-
bers for all the helps! I also thank Dr. Michael Arbib and Dr. Bosco Tjan who have been the
guidance committee member, during which they provide many helpfull advisement and sug-
gestions in my written and oral qualifying exams. I thank all the iLab members with whom
I spent five-year happy time, especially my roommates Vidhya, Ran, David and Nathan for
the technical helping and even some relaxing talks. I also would like to thank JC Zhuang and
Xiangrui Li in the USC MRI center for helping the fMRI scanning teniques and Dr. Linda
Chang, Dr. Thomas Ernst, Renat Yakupov, and C. Lozer in the Brookhave National Lab for
providing help in the MRI scanning and arrangement of visiting. I thank Professor Zuoping
Xie as my undergraduate advisor in Tsinghua University.
I can not express my gratitude to my parents and family, who always supported me in all the
possible ways, whether mentally or economically.
With all of my hearts, I thank all of my personal spiritual leaders especially the two most
important Gurus in my life: Kyabje Yongyal Rinpoche and Teacher Ren.
I thank all the friends in providing help and advice during my research.
iv
Table of Contents
Dedication ii
Acknowledgments iii
List of Tables vii
List of Figures viii
Abstract xv
Chapter 1 General Introduction 1
Chapter 2 A General Literature Review 7
2.1 General Introduction of Attention . . . . . . . . . . . . . . . . . . . . . . . 7
2.2 Spatial Attention . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.3 Feature-based attention . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
2.4 Feature-similarity gain model . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2.5 Task influences local attentional modulation . . . . . . . . . . . . . . . . . . 13
2.6 Object-based attention . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
2.7 Summary and Question . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
Chapter 3 Perceptual Consequence of Feature-based Attention 17
3.1 Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
3.2 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
3.3 Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
3.4 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
3.5 Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
3.6 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
Chapter 4 Feature-based Attention Is Not Object Based 43
4.1 Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
4.2 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
4.3 Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
4.3.1 General introduction of functional Magnetic Resonance Imaging . . . 49
v
4.3.2 Our fMRI method . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
4.4 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
4.5 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
Chapter 5 FMRI Study: Perceptual Level of Feature-based Attention using Gabor
Stimuli 66
5.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
5.2 Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
5.3 Result . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
5.3.1 Orientation discrimination task . . . . . . . . . . . . . . . . . . . . . 71
5.3.2 Drift speed task . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
5.4 Summary and Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
Chapter 6 FMRI Study: Influence of Task Relevant and Irrelevant Features in
Feature-based Attention 79
6.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
6.2 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
6.2.1 Both the task relevant and irrelevant features are different . . . . . . . 82
6.2.2 Task relevant features are different and irrelevant features are same . 84
6.2.3 Task relevant features are same and irrelevant features are different . 86
6.3 Summary and conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88
Chapter 7 Summary and Future work 89
7.1 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
7.2 Future work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
References 94
vi
List of Tables
3.1 All possible task and stimulus combinations. Here if the left side is primary, then
the right side is secondary, and conversely. The upper half of the table shows all the
task combinations when the secondary tasks are orientation discrimination tasks; the
lower half of the table shows the secondary tasks are drift discrimination tasks. Here
TRI and the other top-row labels represent relevant or irrelevant to the secondary
tasks. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
3.2 All parameters of the model fits with 10 free parameters for orientation and drift
tasks for all three subjects. A represents the fraction of attentional resources de-
noted to primary task in that condition. The corresponding attentional resources
to the secondary side in that condition is 1 A.
s
is gain to orientation or drift
thresholds for a certain task combination in the same-task conditions and
d
is gain
to orientation or drift thresholds for a certain task combination in the different-task
conditions. Note that in each condition,
s
is always less than
d
, indicating that
splitting attention across different tasks yield higher gain value and higher additional
cost. The parameters suggest that in all the eight conditions, subjects divided atten-
tional resources between both sides similarly (similar A values across conditions),
suggesting a model fit with fewer parameters as described in Table 3.3. . . . . . . . 32
3.3 All parameters of the model fits using three free parameters, for orientation and drift
tasks for all three subjects. Notations are as in Table 2. . . . . . . . . . . . . . . . 32
vii
List of Figures
2.1 Bottom-up saliency map model. Each visual features such as color, intensity and
orientation are computed in parallel, as a set of pre-attentive feature maps based on
retinal input. The feature maps are combined through all the locations, constructing
the saliency map. Figure is reprinted from L. Itti’ Ph.D. thesis. (Itti Ph.D. thesis,
Caltech, 2000) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
3.1 (a) Stimulus diagram of the direction of motion experiment. Both left and right
stimuli were composed of overlapping fields of upward and downward moving dots.
While fixating, subjects concurrently performed a speed discrimination task on one
field of dots from each side, either moving in the same direction (up or down on both
sides) or in different directions (up on one side and down on the other). (b) Task
performance was better when dividing attention across same vs. different directions
for all subjects. (c) Stimulus diagram of the same experiment without distracters.
Left and right stimuli were each composed of a single field of moving dots. Subjects
concurrently performed a speed discrimination task on the single field of dots from
each side, either moving in the same or different directions (only one example is
diagrammed here). (d) The difference in task performance when dividing attention
across same vs. different directions was reduced for all subjects. Figure is reprinted
from M. Saenz’s paper published in Vision Research. (Saenz et al., 2003) . . . . . . 21
3.2 Experimental paradigm. Stimuli were two drifting Gabor patches (parameters in
Methods) presented bilaterally to the central fixation cross. Dashed circle (not in
actual stimuli) indicates the primary stimulus (given behavioral priority and first
response). Arrows indicate drift direction and speed. Here, the primary task is a
two-interval forced-choice orientation discrimination (subjects responded whether
the stimulus was vertical then tilted, or tilted then vertical). The secondary task
is drift speed discrimination (faster then slower, or slower then faster). Using the
notations of the text, the condition shown is OVS-DHS. . . . . . . . . . . . . . . 23
viii
3.3 Orientation secondary task performance. An example task and stimulu combina-
tion is listed below the figure to show the eight conditions. In this example, the
secondary task is always OVS and the primary tasks are OVS, OVF, OHS, OHF,
DVS, DVF, DHS, and DHF corresponding to the eight conditions TRI, TRI, TRI,
TRI, TRI, TRI, TRI, and TRI. Each datapoint represents threshold elevations (ra-
tio of secondary thresholds to corresponding single-task thresholds) averaged over
left-primary, right-primary, and corresponding horizontal-secondary and vertical-
secondary conditions. Error bars represent standard error on the mean. Statistical
t-test shows p < 0:05 for most comparisons, p < 0:075 at worst for the significant
difference between thresholds described in the results. . . . . . . . . . . . . . . . 26
3.4 Drift speed secondary task performance. An example task and stimulus combination
is listed below the figure to show the eight conditions. In this example, the secondary
task is always DVS and the primary tasks are DVS, DHS, DVF, DHF, OVS, OHS,
OVF, and OHF corresponding to the eight conditions TRI, TRI, TRI, TRI, TRI,
TRI, TRI, and TRI. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
3.5 All the primary and secondary performance for both orientation (left) and drift (right)
tasks and their model predictions. Cyan wide bars represent the threshold elevation
performance and the inside narrow bars represent the corresponding model predic-
tion. The model was fitted with three free parameters (A(t): attentional resources to
primary task, so that the corresponding attentional resources to the secondary task
in that condition are A(t
0
) = 1 A(t));
s
: gain to orientation or drift thresholds
for a certain task combinations in same-task conditions;
d
: gain to orientation or
drift thresholds for certain task combinations in different-task conditions different
task thresholds, Tab. 3.3) for each of the 16 datapoints (eight primary thresholds and
eight secondary thresholds) independent of subjects. Overall model prediction accu-
racy was close to measurement error and suggested a good fit (R
2
= 0:956). Note
that for primary thresholds the TRI related labels represent relevant or irrelevant to
the primary tasks. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
ix
4.1 Feature-based attention to motion. a) Stimuli (not to scale) were circular apertures
(radius 5) of coherently moving random dots in the lower visual field (2.5 below and
centered 11 to left and right of fixation, baseline speed 10/s). Arrow, field of 50 dots
moving upward or downward. Dots were white (560 cd/m2) on a gray background
(230 cd/m2). Dashed circle (not present in the experiment), spatial focus of attention.
(b) fMRI time series of BOLD response (same versus different) to ignored stimulus
for MT+, averaged across three subjects and 24 repetitions per subject. (c) Response
amplitudes to ignored stimulus. (d) Response amplitudes to attended stimulus. (e)
Attentional response amplitudes as a percentage of stimulus-evoked response. Data
in (cCe) are mean s.e.m. During each trial, stimuli were presented for two sequen-
tial 1-s intervals separated by a 100-ms interval in which only the fixation point was
present. Trials started every 3.3 s. The order of speeds was independently random-
ized for each field of dots on every trial, and the baseline speed was randomly and
independently jittered across trials in all three fields of dots. Scans were counterbal-
anced for the attended side (left/right), the starting attended direction (up/down) and
the direction of motion on the ignored side (up/down). Three subjects with normal
visual acuity participated, and all gave written informed consent. Figure is reprinted
from the M. Saenz’s Nature Neuroscience paper in 2002. (Saenz et al. 2002) . . . . 46
4.2 Figure showing objectness visualization of the two fields of dots stimuli. (a) One-
object choice: the subject groups the two fields of dots as one single object. (b)
Two-object choice: the subject doesn’t group the two fields of stimuli, instead treat
them as two different objects. . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
4.3 A typical MRI MPRAGE image taken from our study. . . . . . . . . . . . . . . . 50
4.4 A typical MRI scanning machine setup. . . . . . . . . . . . . . . . . . . . . . . 51
4.5 Localizer stimuli in our study. We used 3x3 checkbox pattern stimuli (5HZ counter
phase flicker) to trigger as much as BOLD activity in our stimuli position (region of
interest). Since we used green and red dots as the real stimuli, so in our localizer
stimuli we also used the red and green box to match the real dot stimuli. . . . . . . 52
4.6 A 1-D view of the regions of interest (shown in the red color) triggerred by the
localizer stimuli for three of the subjects. The ROI size of each subject varied from
2455 voxels to 3572 voxels (1x1x1mm each voxel) . . . . . . . . . . . . . . . . . 54
x
4.7 Experimental stimuli and frame sequences. (a) The two-interval force-choice task
frame sequences in the same-object condition. Within one TR(Time of repetition
= 3s) period, the first stimulus frame displayed for 350ms, followed by a 150ms
inter-frame with same background but no dots stimuli(blank frame). Then the sec-
ond stimulus frame appeared for 350ms and finally the blank frame was showed
for 2150ms for the subjects to judge whether the first or second stimulus frame had
higher luminance for either red dots or green dots at the attended side (left side in
the figure). (b) The shadow effect of the stimuli. The shadow effect was made with
the same background texture but less luminance (0.6 of the textured background lu-
minance) to enhance the objectness visualization. (c) The experimental stimuli in
the different-object condition. The frame sequences are exactly with those in the
same-object condition shown in (a). (d) The localizer stimuli. The checkbox stimuli
were made of 3x3 square frames in each of which the red and green square flashed
at 5Hz frequency. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
4.8 A typical feature-based attentional enhancement. The orange clusters in the activity
map showed the feature-based enhancement at the ignored side (here left side of the
brain, right side of the visual field) when the subject attended same feature compared
to different features at the attended side. The map is contructed at the uncorrect p <
0.000215. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
4.9 The left-bottom figure showed the fMRI response change when subject attended
same feature or different feature in both same-object condition and different-objects
condition by averaging from all the four subjects. The yellow bars indicate same-
object condition. The blue bars indicate different-objects condition. The figure in-
dicates the fMRI BOLD signal when subjects attended same feature (red) (SOSF)
is higher than when they attended different feature (green) (SODF). In the different-
objects condition, the fMRI signal is much higher when subjects attended same fea-
ture (red) (DOSF) than different feature (green) (DODF). This figure also displays a
3D view of the region of interest (ROI) for one of our subject. The size of the ROI
for this subject is 3510 voxels(1x1x1mm each voxel). . . . . . . . . . . . . . . . 59
4.10 The fMRI signal response to the ignored stimuli shown as beta weight magnitude (af-
ter fitting GLM) difference when subject attended same feature minus they attended
different feature in both the same-object and different-object condition (SOSF minus
SODF, DOSF minus DODF). For all the subjects, the fMRI beta weight difference
to the ignored stimuli between same feature and different feature were significantly
higher than zero (Paired t-test :
0

0
indicates p < 0.01,
0

0
indicates p < 0.05,
0

0
indicated p < 0.08) in both the same-object and different-object conditions, sug-
gesting the feature-based attention exists in both the two conditions. . . . . . . . . 60
xi
4.11 The overall averaged results from all the four subjects. The left figure shows the
fMRI response (beta weight average) to the ignored stimuli and the right figure
shows the fMRI response to the attended stimuli (SOSF: subjects attended same
feature in the same-object condition; SODF: subjects attended different feature in
the same-object condition; DOSF: subjects attended same feature in the different-
objects condition; DODF: subjects attended different features in the different-objects
condition). Results indicated in both the same-object condition and different-objects
condition, the fMRI response to the ignored stimuli is significantly higher when sub-
ject attended same feature than when they attended different feature (left). As a
control, the fMRI response has no significant enhancement to the attended stimuli
when same feature vs. different feature. (By t-test,
0

0
indicates p < 0.008,
0

0
indicates p < 0.05, ’n.s.’ indicates no significant enhancement (p > 0.1)) . . . . . . 63
5.1 Experimental timing frame. Stimuli were two drifting Gabor patches presented bi-
laterally to the central fixation cross. Left side is the task side. Task is orientation
discrimination (subjects responded whether the stimulus was vertical then tilted, or
tilted then vertical) on faster horizontal stimulus (OHF). Right side is ignored side
and always vertical and slower patch (OVS). Arrows indicate drift direction and
speed. The two small boxes make the two Gabor stimuli look like two separate
objects. Stimuli also have a certain background which is not shown in this figure. . . 68
5.2 An illustration of different-object conditions. In this condition, the two Gabor stimuli
were casted onto two small boxes in a certain background ( here we didn’t show the
actual textured background). Similarly, in the same-object condition, the two Gabor
stimuli were displayed in one large box with the same background. In this study, we
also have shadow effect cast around the box which is not shown here. . . . . . . . . 70
5.3 Subject LI: fMRI BOLD activity with orientation discrimination task at left side
and ignored stimulus at right side of the display. Upper left showed brain activity
enhancement when subject attended same features than different features in single
object condition. Upper right showed activity enhancement in separate objects con-
dition. Comparison showed activity enhancement reduced when the two stimuli
shown as two separate objects. The enhancement cluster was marked by red ’<’.
The lower part showed the detailed cluster position and the activity strength. The
activity map was shown under uncorrected p < 0.0001. . . . . . . . . . . . . . . . 72
xii
5.4 Subject JL: fMRI BOLD activity with orientation discrimination task at left side
and ignored stimulus at right side of the display. Upper left showed brain activity
enhancement when subject attended same features than different features in single
object condition. Upper right showed activity enhancement in separate objects con-
dition. Comparison showed activity enhancement reduced when the two stimuli
shown as two separate objects. The enhancement cluster was marked by red ’<’.
The lower part showed the detailed cluster position and the activity strength. The
activity map was shown under uncorrected p < 0.0001. . . . . . . . . . . . . . . . 73
5.5 Subject RE: fMRI BOLD activity with orientation discrimination task at left side
and ignored stimulus at right side of the display. Upper left showed brain activity
enhancement when subject attended same features than different features in single
object condition. Upper right showed activity enhancement in separate objects con-
dition. Comparison showed activity enhancement reduced when the two stimuli
shown as two separate objects. The enhancement cluster was marked by red ’<’.
The lower part showed the detailed cluster position and the activity strength. The
activity map was shown under uncorrected p < 0.0001. . . . . . . . . . . . . . . . 74
5.6 Subject LI: fMRI BOLD activity with drift speed discrimination task at left side and
ignored stimulus at right side of the display. Figure shows the results for three sub-
jects. When the subjects attended same features comparing with different features
at the attended side, we didn’t found any obvious significant enhancement on the ig-
nored side. The figure shows the brain fMRI activity in the ignored side. Note at the
visual cortex, no significant enhancement was found. The activity map was shown
under uncorrected p < 0.0001. . . . . . . . . . . . . . . . . . . . . . . . . . . 75
5.7 Subject JL: fMRI BOLD activity with drift speed discrimination task at left side and
ignored stimulus at right side of the display. Figure shows the results for three sub-
jects. When the subjects attended same features comparing with different features
at the attended side, we didn’t found any obvious significant enhancement on the ig-
nored side. The figure shows the brain fMRI activity in the ignored side. Note at the
visual cortex, no significant enhancement was found. The activity map was shown
under uncorrected p < 0.0001. . . . . . . . . . . . . . . . . . . . . . . . . . . 76
5.8 Subject RE: fMRI BOLD activity with drift speed discrimination task at left side
and ignored stimulus at right side of the display. Figure shows the results for three
subjects. When the subjects attended same features comparing with different fea-
tures at the attended side, we didn’t found any obvious significant enhancement on
the ignored side. The figure shows the brain fMRI activity in the ignored side. Note
at the visual cortex, no significant enhancement was found. The activity map was
shown under uncorrected p < 0.0001. . . . . . . . . . . . . . . . . . . . . . . . 78
xiii
6.1 The results of four subjects in orientation discrimination task. The figure shows
the fMRI BOLD response at the ignored side when subjects attended same feature
(vertical and slow drift) comparing to different feature (horizontal and fast drift) at
the attended side, which means both the task relevant and irrelevant features are
different. Figure shows there is not statistically enhancement at the visual areas
(uncorrected p < 0.0001). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
6.2 The results of four subjects in drift speed discrimination task. The figure shows
the fMRI BOLD response at the ignored side when subjects attended same feature
(vertical and slow drift) comparing to different feature (horizontal and fast drift) at
the attended side, which means both the task relevant and irrelevant features are
different. Figure shows there is not statistically enhancement at the visual areas
(uncorrected p < 0.0001). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
6.3 The results of four subjects in orientation discrimination task. The figure shows
the fMRI BOLD response at the ignored side when subjects attended same feature
(vertical and slow drift) comparing to different task relevant feature (horizontal) but
same task irrelevant feature (slow) at the attended side. Figure shows there is not
statistically enhancement at the visual areas (uncorrected p < 0.0001). . . . . . . . 84
6.4 The results of four subjects in drift speed discrimination task. The figure shows
the fMRI BOLD response at the ignored side when subjects attended same feature
(vertical and slow drift) comparing to different task relevant feature (fast drift) but
same task irrelevant feature (vertical) at the attended side. Figure shows there is not
statistically enhancement at the visual areas (uncorrected p < 0.0001). . . . . . . . 85
6.5 The results of four subjects in orientation discrimination task. The figure shows
the fMRI BOLD response at the ignored side when subjects attended same feature
(vertical and slow drift) comparing to different task irrelevant feature (fast drift) but
same task relevant feature (vertical) at the attended side. Figure shows there is not
statistically enhancement at the visual areas (uncorrected p < 0.0001). . . . . . . . 86
6.6 The results of four subjects in drift speed discrimination task. The figure shows
the fMRI BOLD response at the ignored side when subjects attended same feature
(vertical and slow drift) comparing to different task irrelevant feature (horizontal)
but same task relevant feature (slow drift) at the attended side. Figure shows there is
not statistically enhancement at the visual areas (uncorrected p < 0.0001) . . . . . . 87
xiv
Abstract
Visual attention modulates visual processing along at least three dimensions: A spatial di-
mension which enhances the representation of stimuli within the focus of attention, a feature
dimension thought to enhance attended visual features throughout the visual field and an
object dimension by which attention enhance the whole object as a unit. In this thesis, we
focus on the feature dimension by studying the feature-based attention. We investigated the
overall perceptual consequences of feature-based attention, by using dual-task human psy-
chophysics and two distant drifting Gabor stimuli to systematically explore 64 combinations
of visual features (orientations and drift speeds) and tasks (discriminating orientation or drift
speed). The resulting single, consistent dataset suggests a functional model, which predicts a
maximum-rule by which only the dominant product of feature enhancement and feature ben-
efit by feature relevance may benefit perception. We also used fMRI to investigate whether
feature-based attention is a specific form of object-based attention or a new type of lower
feature based attentional selection. We studied the feature-based attentional enhancement in
two conditions: either the two stimuli appeared to belong to same object or as two different
objects. Results showed both in same-object condition and in different-object condition the
four subjects consistently had significant enhancement of the ignored stimulus in early visual
xv
areas. Hence it indicated feature-based attentional enhancement exists even when two stim-
uli belong to two different objects, suggesting it is a new type of attentional selection which
takes place in the low feature levels, not dependent on the objectness of the two features. We
also present two unpublished studies showing that Gabor stimuli are not the suitable stim-
uli to trigger the feature-based attentional enhancement using fMRI. In summary, this thesis
provides a functional model to explain the overall psychophysical perceptual mechanism of
the feature-based attention, and evidence that the feature-based attention is new kind of at-
tentional selection biasing on the lower feature level.
xvi
Chapter 1
General Introduction
Attention in three levels: What we see in the world greatly depends upon our visual atten-
tion. Visual attention is how we direct our vision towards interesting objects in our visual
fields. Attention modulates visual processing along at least three dimensions: A spatial di-
mension which enhances the representation of stimuli within the focus of attention, a feature
dimension thought to enhance attended visual features throughout the visual field and an ob-
ject dimension by which attention enhance the whole object as a unit.
Feature-based Attention: Traditionally, attention was considered that attention enhance
the visual features only at the attended position (in the receptive field). Feature-based at-
tention has been illustrated as enhancement of the attended visual features (color, orienta-
tion) throughout the visual cortex, specifically outside the receptive fields. There has been
accumulated evidence from psychophysical, electrophysiological and functional Magnetic
Resonance Imaging (fMRI) studies. A typical electrophysiological study showed attention
1
increased the gain of direction-selective neurons in visual cortical area MT outside the neu-
ron’s receptive field (Treue and Martinez Trujillo, 1999). Another study using fMRI showed
that attention enhances attended visual features throughout the visual cortex, specifically, at
the unattended side (Saenz et al., 2002). The feature-based enhancement effect is further
confirmed by the psychophysical study which revealed that subjects’ psychophysical perfor-
mance was better when they attend same features than different features (Saenz et al., 2003).
Object-based Attention: Object-based attention is firstly shown as human observers per-
formed better when concurrently discriminating two features of the same object compared to
two features of different objects (Duncan 1984). Afterwards, there has been many evidence
supporting that attentional selection takes place in the ’object’ level, selecting the whole ob-
ject as an entity. For example, studies indicated attention selection of a specific feature of
an object also enhance the processing of other features of the same object (Sohn et al., 2004,
Valdes-Sosa et al., 1998), even the features are task irrelevant (O’Craven et al., 1999). Fur-
ther study indicated that the object-based attentional selective ability is quite strong that it
can even enhance unconscious features of that object as well (Melcher et al., 2005). Some
comparative studies provided evidence that attentional selection biases on more on object
level than feature level. For example, attention can rapidly track an object through feature
space, even when distracters occupy the location of the attended object (Blaser et al., 2000).
Our Study–Perceptual mechanism of feature-based attention: In this thesis, we seek to
understand the detailed perceptual mechanism of feature-based attention both qualitatively
2
and quantitatively. In particular, we hope to understand the feature-based attention in two
different views: the perceptual consequence of feature-based attention with task and stim-
uli variation and whether feature-based attention is also object based. Although the new
global effects of visual areas due to feature-based attention have been discovered recently in
both fMRI study and psychophysical methods, these effects currently lack a complete, uni-
fied computational understanding when varying their task and feature parameters. There are
highly competing theories as to whether attention can select basic features such as colors and
shapes or particular visual objects composed of constellations of visual features. Our study
will directly bear on these theories to understand a complete perceptual consequence of the
feature-based attention, and whether attention is feature-based, or object-based, or both.
Chapter 2–A general literature review: In this chapter, we provided a detailed literature
review for understanding the feature-based attention and the related object-based attention.
Chapter 3–Perceptual consequence of feature-based attention: In this chapter, we present
our study of perceptual consequence of feature-based attention, by using dual-task human
psychophysics and two distant drifting Gabor stimuli to systematically explore 64 combina-
tions of visual features (orientations and drift speeds) and tasks (discriminating orientation
or drift speed). The resulting single, consistent dataset suggests a functional model, which
predicts a maximum-rule by which only the dominant product of feature enhancement and
feature benefit by feature relevance may benefit perception.
3
Chapter 4–Feature-based attention is not object based: In this chapter, we present the
results to demonstrate that feature-based attentional selection takes place in the lower feature
levels, e.g. not object based. The stimuli were two fields of random dots presented bilaterally
to a central fixation cross. Subjects performed luminance discrimination using a two-interval
forced choice paradigm on one side and ignored the stimuli on the other side. The ignored
stimulus was always red dots and the attended stimulus was overlapping red and green dots.
We compared visual cortical enhancement of the ignored stimulus when subjects attended
same feature (red) comparing with they attended different feature (green) on the other side in
two conditions: either the dots stimuli on both sides appeared to belong to same object or as
two separate objects. Results showed both in same-object condition and in different-object
condition the four subjects consistently had significant enhancement of the ignored stimulus
in early visual areas. Hence it indicated feature-based attentional enhancement exists even
when two stimuli belong to two different objects, suggesting it is a new type of attentional
selection which takes place in the low feature levels, not dependent on the objectness of
the two features. In summary, this thesis provide a functional model to explain the overall
psychophysical perceptual mechanism of the feature-based attention, and evidence that the
feature-based attention is new kind of attentional selection biasing on the lower feature level.
Chapter 5–FMRI Study: Perceptual level of feature-based attention using Gabor stim-
uli In this chapter, we present some unpublished results which using Gabor stimuli to study
if the feature-based attention is also object-based. The experimental strategy is same with the
previous chapter. We studied the feature-based attention in two different conditions: either
4
the two features displayed as single object, or they displayed as different objects. However,
the stimuli are same with those in Chapter 3 using Gabor stimuli. The difference is the
orientation and drift speed discrimination tasks are only engaged at one side of the stimuli,
leaving the stimulus of another side ignored. Comparing the fMRI BOLD signal strength at
the ignored side when subjects attended same features versus attended different features on
the other side in both same-object condition and different-object condition, we didn’t find the
enhancement due to feature-based attention, suggesting the Gabor stimulus is not sensitive to
trigger the feauture-based attention in the fMRI study. These results are consistent with the
unpublished data in Saenz’s thesis (M. Saenz’s Ph.D. thesis, UCSD, 2002).
Chapter 6–FMRI study: the influence of task relevant and irrelevant features in feature-
based attention: In this chapter, we present unpublished fMRI results which studied the
roles of task relevant feautures and task irrelevant features in determining the feature-based
attention. The stimuli and experimental paradigms are same with those in Chapter 5. The
orientation and drift speed discrimination tasks are only engaged at one side of the stimuli,
leaving the stimulus of another side ignored. The stimuli varied among task relevant features
and task irrelevant features, which used the same feature patterns described in Chapter 3.
According to our findings in Chapter 3, we expected that the different gain factors by the task
relevant and irrelevant features in determining the enhancement of the feature-based atten-
tion still exists in the fMRI study. Specifically, we expect that the feature-based attentional
enhancement at the ignored side should be the largest when the stimuli share both the task
relevant and irrelevant features, the second largest when they share only the task relevant,
5
and the smallest when they share only the task irrelevant features. However, no statistically
significant feature-based attentional effects were observed on all of the conditions above,
suggesting and confirming the conclusions in Chapter 5 that the Gabor stimuli may be not
the suitable and sensitive stimuli in trigger the feature-based attentional enhancement in the
fMRI study.
Chapter 7–Summary and future work: In this chapter, by summarizing all the studies
above, we conclude that the feature-based attentional effect, which was represented as the
enhancement of attended features all throughout the visual cortex, is indeed ’feature’ based.
The feature-based attention selects the single feature as the entity, not due to the traditional
thought of the object-based attention, by which the feature enhancement is because the ig-
nored features belong to the same object with the attended features at the attended location.
Furthermore, the detailed perceptual study of the feature-based attentional consequence sug-
gested that both the task relevant feautures and task irrelevant features contributed to the
enhancement effect but by a different gain factor. The overall perceptual consequence could
be unified into a simple functional computational model which predicts a winner-take-all
rule in the feature space. Finally, the Gabor stimuli are not the suitable stimuli in trigger the
feature-based attentional enhancement in the fMRI study. We also describe some possible
future works in continuing study the mechanism of the feature-based attention.
6
Chapter 2
A General Literature Review
2.1 General Introduction of Attention
Visual attention is how we direct our vision towards interesting objects in our visual fields.
In general, there are two types of controls in determining the how attention works. Bottom-
up attention is based on the image features, thus its process is very fast (up to 20 shifts per
second). Bottom up attention is also involuntary and determined by the visual input infor-
mation. A detailed computational model was proposed to simulate the bottom-up attention
called saliency map model (Koch and Ullman, 1985, Itti and Koch, 2001). In this model,
the visual input image is simultaneously encoded by different feature channels. In each of
these channels, the features are extracted and analyzed and the salient maps are constructed.
Finally the winner-take-all network detects the most location and Attention is directed to the
that location of the image. Figure 2.1 provided a graphic description of this saliency map
model. Another attention control is top-down attention. Top-down attention refers to atten-
tion can target a certain location in the visual scene, even this location may be inconspicuous.
7
Figure 2.1: Bottom-up saliency map model. Each visual features such as color, intensity and orienta-
tion are computed in parallel, as a set of pre-attentive feature maps based on retinal input. The feature
maps are combined through all the locations, constructing the saliency map. Figure is reprinted from
L. Itti’ Ph.D. thesis. (Itti Ph.D. thesis, Caltech, 2000)
Top-down attention is normally task directed, thus voluntary and slower than bottom up at-
tention.
In another views of understanding visual attention, attention is considered to modulate visual
processing at three dimensions: A spatial dimension which enhances the representation of
8
stimuli within the focus of attention, a feature dimension thought to enhance attended visual
features, for example upward motion, throughout the visual field and an object dimension by
which attention enhance the whole object as a unit.
Some early theories provided a spotlight enhancing mechanism in which attention locally
enhances neuronal responses within the attended focus (Posner, Snyder et al. 1980; Eriksen
and St James 1986), which was termed as spatial attention. In a recent observed feature di-
mension, attention enhances visual features not only in the focus of attention, but throughout
the visual field (Treue and MartinezTrujillo, 1999; McAdams and Maunsell, 2000; Saenz et
al. 2002, 2003).
2.2 Spatial Attention
Electrophysiological studies showed that visual cortical responses are enhanced by visual
attention. For example, attention can modulate neural response to attended stimuli. The
neuronal response is enhanced when attention is directed to the stimulus in its receptive field.
This enhancement becomes larger when several stimuli present to compete each other (Moran
and Desimone 1985; Luck, Chelazzi et al., 1997; Treue and Maunsell 1999). In another study,
attentional modulation of responses in areas V1, V2 and V4 were stronger when attended to
a visual object with the presence of nearby distracting objects (Motter 1993). In the neu-
roimaging studies, it was found that attention increased the activity of neurons with receptive
field even without stimulus is present (Reynolds, Chelazzi et al. 1999; Kastner, Pinsk et al.
1999). In the retinal topological view, attention was observed to modulate the responses at
9
almost all visual cortical levels including V1 (Motter 1993; Tootell, Hadjikhani et al. 1998;
Brefczynski and DeYoe 1999), and possibly more in later stages like V2, V4 and MT+. In
the neural level, the function of attention to modulate neuronal responses. For example in
McAdams and Maunsell study, a grating of different orientations was displayed in recep-
tive field of a neuron under and the animal performed the task on either the grating within
the receptive field or on another objects outside the receptive field. The results showed that
attention caused an overall gain increase in orientation tuning in V4 without changing cell
selectivity for orientation, but not the tuning peak and turning width. Another study for the
direction selective cells in MT was done by Martinez-Trujillo and Treue in 2002, in which
they concluded that attention increase stimulus contrast without changing the tuning proper-
ties of the cell. In summary of these results, attention increases the responses to stimuli in
attended focus.
2.3 Feature-based attention
Originally, some studies showed that advance knowledge about certain features of the stimu-
lus improved the detection. For example, Ball and Sekuler asked subjects to detect a visible
moving object that briefly flashed on display. Psychophysical performance was much bet-
ter if subjects acknowledge either direction or speed of the objects in advance. In this case,
attention is based on feature rather than space. In some further studies, this feature-related
attention is found in many other experiments. For example, it was found that subject can
selectively attend to a subset of stimuli within the same location that share common features
(Alais and Blake 1999) or to one of two spatially overlapping objects (Duncan 1984; Blaser
10
et al. 2000). Very recently, a non-spatial, feature-based form of attentional enhancement
has been reported, by which the activity of neurons tuned to some visual feature of a stimu-
lus (e.g., a given color or direction of motion) is increased throughout the visual field when
attention is engaged onto the stimulus. This so-called feature-based attentional modulation
has been demonstrated in human functional neuroimaging (Beauchamp et al., 1997; Saenz
et al., 2002), monkey electrophysiology (Treue and MartinezTrujillo, 1999; McAdams and
Maunsell, 2000), and human psychophysics (Shih and Sperling, 1996; Duncan and Nimmo-
Smith, 1996; Saenz et al., 2003). For example, in a single-unit macaque physiology study,
Treue and Martinez-Trujillo (1999) showed how attending to an upward-moving field of dots
on one side of the visual field increased activity in area MT, selectively for neurons tuned
to upward motion, although they could have receptive fields in the opposite hemifield from
the attended stimulus; but attending to a downward-moving field of dots did not. In a human
psychophysics study, Saenz et al. (2003) showed how performance on a dual-task was signif-
icantly better when human observers divided attention across two spatially separate stimuli
sharing a common feature (same direction of motion or same color) compared to oppos-
ing features. Of particular interest here, Vidnyanszky et al. (Proc. ECVP, 2003) recently
suggested that a motion after-effect, induced by an ignored field of dots whose dominant mo-
tion was carried by dots of a given color, appeared stronger if subjects attended, in a distant
aperture of the display, to dots of that color moving in random directions. Hence, feature-
based attention seems to affect even task-irrelevant features, in this case motion while the
task-relevant feature was color.
11
2.4 Feature-similarity gain model
In addition of the spotlight mechanism of attention, Moran and Desimone illustrated that at-
tention modulated the responses of visual neurons which depend on the feature selectivity
of the cell (Moran and Desimone 1985). In their study, a pair of stimuli was displayed in
the receptive field of the neuron in V4 and IT. One stimulus matched the cell’s preference
while the other stimulus did not. Neural firing rates increased when attention was directed
to the preferred stimulus and decreased when attention directed to non-preferred stimulus.
The results showed that the mechanism of attention is not only enhancing the spotlight of
attended locus but also depend on the neuron’s selectivity. Such trend also were found for a
wide range of stimuli like color in V2, V4 and IT (Luck, Chelazzi et al. 1997) and motion
stimuli in MT (Treue and Maunsell 1999).
The biased competition model was proposed for this feature-specific attentional modulation
(Desimore and Duncan 1995; Reynolds, Chelazzi et al. 1999). This model proposed that
multiple presented features activated competing neurons. The competition may be biased for
those neurons representing a special object through the bottom-up or top-down attentional
modulation. The biasing effects of attention could apply to multiple dimensions of stimulus
properties including spatial location, low-level features and even complex object. Thus the
model unified both spatial and feature-based attention.
Some recent studies further developed this model. McAdams and Maunsell found that the
neuronal response to an ignored grating stimulus was increased when monkey performed
12
an orientation discrimination task on a second grating compared to when it performed color
discrimination on a colored patch (McAdams et al., 2000). In another study, Treue and
Martinez-Trujillo showed feature-specific attentional modulation of responses in MT. In their
study, an ignored dots stimulus moving in the preferred direction was presented within the
receptive fields of the neuron, while attention was directed to another outside stimulus which
either moved same or opposite direction (Treue et al., 1999). The results showed neuronal
responses to the ignored stimulus increased when the monkey attended the preferred direc-
tion, and deceased when monkey attended to opposing direction. These results allowed the
bias competition model developed into a feature-similarity gain model in which feature-based
attention modulated the gain of cortical neurons that are selective for the feature property.
2.5 Task influences local attentional modulation
Previous reports have demonstrated a critical role of task onto this localized attentional mod-
ulation within the focus of attention (Corbetta et al., 1991; Huk and Heeger, 2000). For
example, Watanabe et al. (1998) showed, using one stimulus with superimposed translating
and expanding fields of dots, differential attentional modulation of fMRI signal, depending
on whether the task was to attend to the translating or the expanding feature of the stimulus.
At longer time scales, Saffell and Matthews (2003) reported task-specific perceptual learning
at a single visual location, using random dots cinematograms that always changed speed, di-
rection and luminance: Subjects trained to discriminate stimulus direction showed significant
improvement in direction discrimination, but their speed or luminance discrimination perfor-
mance did not improve; conversely, subjects trained to discriminate speed improved in speed
13
but not direction or luminance discriminations (Shiu and Pashler, 1992). Similarly, Morrone
et al. illustrated that peripheral increment thresholds for luminance constrast deteriorate when
the observer is currently performing another luminance (form or contrast) discrimination task
in central view, but not when the subject is performing a color discrimination task. Similar
results were obtained for color increment thresholds, indicating attention is parceled in inde-
pendent resources for luminance and color contrast (Morrone et al., 2004). Thus, attentional
modulation and some forms of perceptual learning appear to be specific not only for location
but also for a subset of stimulus features that are relevant to the task engaging attention.
2.6 Object-based attention
Studies have shown that human observers performed better when concurrently discriminating
two features of the same object compared to two features of different objects (Blaser et al.,
2000; Duncan 1984), which is considered as the term of object-based attention. For example,
Duncan (1984) used small (less than 1 degree), brief, foveal displays, each consisting of two
overlapping objects. It was found that two judgments that concern the same object could be
made simultaneously without loss of accuracy, whereas two judgments that concern different
objects cannot. Another fMRI study showed objects as the units of attentional selection:
Attending to one attribute of an object (such as the motion of a moving face) enhanced the
neural representation not only of that attribute but also of the other attribute of the same object
(for example, the face), compared with attributes of the other object (for example, the house)
There is a growing body of psychophysical and neurophysiological evidence that attention
can indeed select specific visual objects. For instance, Blaser et al. found that subjects can
14
keep track of a stationary item solely on the basis of its changing appearance specified by its
trajectory along color, orientation, and spatial frequency dimension even when a distracter
shared the same spatial location (Blaser et al., 2000).
2.7 Summary and Question
In summary, previous reports suggest a critical interaction between task and stimulus in shap-
ing attentional modulation and perceptual learning, for many isolated pairings of tasks and
stimuli at same or different visual locations. However, a complete quantitative understanding
of how task-relevant and task-irrelevant features are comparatively enhanced by attentional
modulation remains elusive.
On the other hand, the common features between feature-based attention and object-based
attention is attention both can enhance the same feature with attended features outside the at-
tentional receptive field. But the difference is object-based attention only enhance the features
which belong to the same object with the attended features at the attended position. Consid-
ering the previous typical studies on the feature-based attention (Saenz et al., 2002,Treue and
Martinez Trujillo 1999), the dots stimuli used were normally displayed in the blank black
background, thus the two features on the two sides were not clearly object segmented. This
raised a fundamental question: since object-based attention can also enhance the attended
features at the ignored position and it showed a so strong selective ability, is it possible that
the feature enhancement by the feature-based attention is because subjects grouping the two
features as same object so that it is actually a specific form of object-based attention, or due to
15
a completely different type of attentional selection mechanism by which attentional selection
indeed process individual features as a unit?
16
Chapter 3
Perceptual Consequence of Feature-based Attention
3.1 Abstract
Attention modulates visual processing along at least three dimensions: A spatial dimension
which enhances the representation of stimuli within the focus of attention, an object dimen-
sion which attention enhances all the features on the whole object, and a feature dimen-
sion which enhances attended visual features throughout the visual field. We investigate the
consequences of feature-based attention onto visual perception, using dual-task human psy-
chophysics and two distant drifting Gabor stimuli to systematically explore 64 combinations
of visual features (orientation and drift speed) and tasks (discriminating orientation or drift
speed). The resulting single, consistent dataset suggests a functional model, predicting a
maximum-rule by which only the dominant product of feature enhancement by feature rele-
vance may benefit perception.
17
3.2 Introduction
It has recently become clear that early visual processing is not purely bottom-up and depen-
dent on stimulus properties alone, but can be substantially modulated top-down by expecta-
tions and tasks. A first, widely observed, top-down modulatory effect of attention onto early
vision is to locally enhance the cortical representation of stimuli within the focus of attention
(Motter, 1994; Treue and Maunsell, 1996; Brefczynski and DeYoe, 1999; Lee et al., 1999).
This observation is well in line with the shiftable spotlight metaphor of attention, not only
serving to select a subregion of the visual input for further processing, but also highlighting
the neural representation of its contents.
Previous reports have demonstrated a critical role of task onto this localized attentional mod-
ulation within the focus of attention (Corbetta et al., 1991; Huk and Heeger, 2000). For
example, Watanabe et al. (Watanabe et al., 1998) showed, using one stimulus with super-
imposed translating and expanding fields of dots, differential attentional modulation of fMRI
signal, depending on whether the task was to attend to the translating or the expanding feature
of the stimulus. Thus, attentional modulation and some forms of perceptual learning appear
to be specific not only for location, but also for a subset of stimulus features that are relevant
to the task engaging attention.
More recently, a non-spatial, feature-based form of attentional enhancement has been re-
ported, by which the activity of neurons tuned to some visual feature of a stimulus (e.g.,
18
a given color or direction of motion) is increased throughout the visual field when atten-
tion is engaged onto the stimulus. This so-called feature-based attentional modulation has
been demonstrated in human functional neuroimaging (Saenz et al., 2002), monkey electro-
physiology (Treue and Martinez Trujillo, 1999; McAdams and Maunsell, 2000), and human
psychophysics (Shih et al., 1996, Duncan and Nimmo-Smith, 1996 and Saenz et al., 2003).
For example, in a single-unit macaque physiology study, Treue and Martinez-Trujillo (Treue
and Martinez Trujillo, 1999) showed how attending to an upward-moving field of dots on
one side of the visual field increased activity in area MT, selectively for neurons tuned to
upward motion, although they could have receptive fields in the opposite hemifield from the
attended stimulus; but attending to a downward-moving field of dots did not. In a human psy-
chophysics study, Saenz et al. (Saenz et al., 2003) showed how performance on a dual-task
was significantly better when human observers divided attention across two spatially separate
stimuli and attended to a same feature on both sides (same direction of motion or same color)
compared to opposing features. Of particular interest here, Vidny´ anszky et al. (Proc. ECVP,
2003) recently suggested that motion after-effect, induced by an ignored field of dots whose
dominant motion was carried by dots of a given color, appeared stronger if subjects attended,
in a distant aperture of the display, to dots of that color moving in random directions. Hence,
feature-based attention seems to affect even task-irrelevant features, in this case motion while
the task-relevant feature was color. Whether the enhancement of task-irrelevant features was
comparable to that of task-relevant features, however, was not elucidated.
19
Here we describe a psychophysical study on the feature-based attention (Saenz, et al., 2003)
which is most related to our work. In their psychophysical study, performance on a dual-task
was significantly better when human observers divided attention across two spatially sepa-
rate stimuli sharing a common feature (same direction of motion or same color) compared
to opposing features. This attentional effect was dependent upon the presence of competing
stimuli. The stimulus was composed of two spatially separate circular apertures of moving
random dots. The left and right sides of the display were identical and were each composed
of two overlapping fields of upward and downward moving dots. Subjects were instructed
to perform a threshold level speed discrimination task on one field of dots from each side at
the same time. The dual-task was performed in successive two-interval forced choice trials
(2-IFC). Subjects performed the dual-task on two fields of dots moving in either the same di-
rection (same trials) or different directions (different trials). The following plots performance
on same trials compared to different trials for each of three subjects. All subjects performed
significantly better on same trials than on different trials. The conclusion is that subjects were
better at detecting changes in a pair of spatially separated stimuli when they share a common
feature, such as a direction of motion or color, than when they did not share a common fea-
ture. This conclusion is also suggested by another set of color feature experiment in which
the feature of upward and downward motion was replaced by red and green colors of the dots
stimuli while keeping other conditions unchanged.
Combining all the studies above, in summary, previous reports suggest a critical interac-
tion between task and stimulus in shaping attentional modulation and perceptual learning,
20
Figure 3.1: (a) Stimulus diagram of the direction of motion experiment. Both left and right stimuli
were composed of overlapping fields of upward and downward moving dots. While fixating, subjects
concurrently performed a speed discrimination task on one field of dots from each side, either moving
in the same direction (up or down on both sides) or in different directions (up on one side and down
on the other). (b) Task performance was better when dividing attention across same vs. different
directions for all subjects. (c) Stimulus diagram of the same experiment without distracters. Left and
right stimuli were each composed of a single field of moving dots. Subjects concurrently performed
a speed discrimination task on the single field of dots from each side, either moving in the same or
different directions (only one example is diagrammed here). (d) The difference in task performance
when dividing attention across same vs. different directions was reduced for all subjects. Figure is
reprinted from M. Saenz’s paper published in Vision Research. (Saenz et al., 2003)
for many isolated pairings of tasks and stimuli at same or different visual locations. How-
ever, a complete quantitative understanding of how task-relevant and task-irrelevant features
are comparatively enhanced by attentional modulation remains elusive. Here, we operate
a systematic dissociation between tasks and stimulus features to elucidate their interplay in
shaping feature-based attentional modulation. We employ dual-task human psychophysics
and two distant drifting Gabor stimuli to systematically explore 64 combinations of visual
features (orientation and drift speed) and tasks (discriminating orientation or drift speed).
21
Our systematic study thus aims at casting previous and future findings on isolated task and
feature combinations into a single conceptual and computational framework.
3.3 Methods
Subjects and paradigm: Three normal volunteer human subjects simultaneously performed
two pattern discriminations (Sperling and Melchner, 1978; Lee et al., 1999), on two drifting
Gabor stimuli presented bilaterally to fixation. In each block of 100 trials, subjects divided
attention between both stimuli: one engaged their primary task (first response and behavioral
priority and dominance), presumably triggering a first set of feature-based attention effects.
The other was their secondary task (second response), presumably triggering a second set of
effects. Here we investigate how performance at the primary and secondary tasks might have
benefited from each other through possible feature-based attentional enhancement triggered
at the other site.
Stimuli and tasks: Stimuli were generated in Matlab (Mathworks, Inc.) using the Psy-
chophysics Toolbox (Brainard, 1997). Subjects were seated at a viewing distance of 80 cm
from a 22” color monitor (LaCie, Inc.; 28

21

usable field-of-view, 72 Hz) and rested on
a chin-rest. Mean screen luminance was 30 cd/m
2
, room 4 cd/m
2
. Drifting Gabor stimuli
(spatial frequency 3.6 cpd, envelope FWHM 0:55

, contrast 0.4) were horizontal or vertical
(H or V orientation feature), drifting slowly (3.33 c/s; S drift speed feature) or faster (5.00 c/s;
F drift speed feature) and were displayed at 4

eccentricity from fixation.
22
Figure 3.2: Experimental paradigm. Stimuli were two drifting Gabor patches (parameters in Meth-
ods) presented bilaterally to the central fixation cross. Dashed circle (not in actual stimuli) indicates
the primary stimulus (given behavioral priority and first response). Arrows indicate drift direction
and speed. Here, the primary task is a two-interval forced-choice orientation discrimination (subjects
responded whether the stimulus was vertical then tilted, or tilted then vertical). The secondary task is
drift speed discrimination (faster then slower, or slower then faster). Using the notations of the text,
the condition shown is OVS-DHS.
Tasks were orientation (O) or drift speed (D) discriminations, using a dual two-interval
forced-choice (2IFC) paradigm. On any given trial, only one parameter differed between
the two successive Gabor patch presentations, independently on each side of the display. In
the orientation discrimination tasks, stimuli could be oriented vertically (condition V) or hor-
izontally (condition H) vs. slightly tilted off that orientation. Subjects then reported whether
the slightly tilted patch had appeared first. In the drift speed discrimination task, stimuli
drifted at the slower (condition S) or faster (condition F) speed vs. slightly increased speed.
Subjects then reported whether the patch with slightly increased speed had appeared first.
Subjects reported the order of the alternative targets for both the left and right sides of the
23
display, by pressing two keys, and received auditory feedback (two brief tones were emitted
for incorrect answers, and three tones for correct answers).
Given 128 possible task and stimulus combinations (primary side, left or right; tasks, either
O or D on each side; stimuli, each among HS, VS, HF or VF), we measured dual-task dis-
crimination thresholds for 64 combinations (secondary stimulus speed always S).
Trials: Subjects trained on every task and stimulus combination until threshold stabilization
was reached, to factor out perceptual and other learning. After training, between 10 and 30
blocks of 100 trials were acquired for each datapoint plotted in Figs. 4.3 and 4.4 (depending
on difficulty of task combinations). Thresholds (75% correct performance) were determined
with an adaptive dual staircase procedure, adjusting the difference in patch orientation or
drift speed between the two successive alternatives for each stimulus in each trial, according
to success or failure of previous trials (Lee et al., 1999). Each threshold was computed from
a maximum-likelihood fit of a Weibull function with two degrees of freedom to the staircase
data (Itti et al., 2000). To provide a baseline for the dual-task thresholds reported in this study,
single-task thresholds were also measured, by instructing subjects to only perform one task
at one site, while keeping the stimuli at both sites identical to the dual-task stimuli.
24
3.4 Results
We focus on how different tasks and stimuli at one site modulated the thresholds at the other
site, that is, how feature-based attentional modulation triggered at the one site affected per-
ception at the other site. Because primary task thresholds didn’t exhibit large variations across
tasks, here we focus on the secondary thresholds first (The primary thresholds will be han-
dled by the modeling below). We define stimulus orientation as relevant to the orientation
discrimination task while stimulus drift speed is irrelevant, and conversely. Figs. 4.3 and 4.4
show secondary thresholds under identical or different tasks (T or T), same or different stim-
ulus feature relevant to the secondary tasks (R or R), and same or different feature irrelevant
to the tasks (I or I). Table 3.1 shows all the task combinations and corresponding relevant
properties to the secondary task. Note that stimuli always shared some task-irrelevant fea-
tures, like luminance contrast; because these were invariant, we assume no differential effect
on observations. To focus our analysis on differential threshold modulation by stimuli and
tasks at the other site, all thresholds are reported as threshold elevations, i.e., normalized
by the corresponding single-task thresholds. Also, all 64 thresholds were collapsed into 16
compound datapoints, by averaging to reduce bias across left- and right-primary and across
vertical and horizontal orientations.
With identical tasks and stimuli (TRI), perception of the secondary stimulus presumably ben-
efited maximally from feature-based attention triggered at the primary (Saenz et al., 2003).
Indeed, secondary thresholds were the lowest (best) of all conditions, yet they are higher
than the corresponding single-task thresholds (threshold elevations greater than unity), due to
25
0
2
4
Threshold Elevations
0
2
4
Threshold Elevations
0
2
4
Threshold Elevations
Subject JJ
Subject JW
Subject SQ
Example:
Figure 3.3: Orientation secondary task performance. An example task and stimulu combination is
listed below the figure to show the eight conditions. In this example, the secondary task is always
OVS and the primary tasks are OVS, OVF, OHS, OHF, DVS, DVF, DHS, and DHF corresponding
to the eight conditions TRI, TRI, TRI, TRI, TRI, TRI, TRI, and TRI. Each datapoint represents
threshold elevations (ratio of secondary thresholds to corresponding single-task thresholds) averaged
over left-primary, right-primary, and corresponding horizontal-secondary and vertical-secondary con-
ditions. Error bars represent standard error on the mean. Statistical t-test shows p < 0:05 for most
comparisons, p < 0:075 at worst for the significant difference between thresholds described in the
results.
26
T R I T R I T R I T R I T R I T R I T R I T R I
OVS-OVS OVF-OVS OHS-OVS OHF-OVS DVS-OVS DVF-OVS DHS-OVS DHF-OVS
OHS-OHS OHF-OHS OVS-OHS OVF-OHS DHS-OHS DHF-OHS DVS-OHS DVF-OHS
DVS-DVS DHS-DVS DVF-DVS DHF-DVS OVS-DVS OHS-DVS OVF-DVS OHF-DVS
DHS-DHS DVS-DHS DHF-DHS DVF-DHS OHS-DHS OVS-DHS OHF-DHS OVF-DHS
Table 3.1: All possible task and stimulus combinations. Here if the left side is primary, then the right
side is secondary, and conversely. The upper half of the table shows all the task combinations when the
secondary tasks are orientation discrimination tasks; the lower half of the table shows the secondary
tasks are drift discrimination tasks. Here TRI and the other top-row labels represent relevant or
irrelevant to the secondary tasks.
splitting of attentional resources across both tasks (Sperling and Melchner, 1978; Lee et al.,
1999). Progressively decreasing the similarity of both stimuli showed how secondary thresh-
olds benefited when stimuli shared task-relevant features (TRI better than TRI) (Treue and
Martinez-Trujillo, 1999; McAdams and Maunsell, 2000; Saenz et al., 2003), but did not ben-
efit further from additionally sharing task-irrelevant features (TRI same as TRI). However,
when no task-relevant feature was shared, sharing a task-irrelevant feature benefited thresh-
olds (TRI better than TRI), although often more weakly than sharing a task-relevant feature.
Together, our results under same-task conditions hence suggest a maximum-rule (at the basis
of our model below), by which perception improves according to the maximal product of
feature enhancement and feature benefit.
When tasks differed, sharing any feature improved secondary thresholds (TRI, TRI and TRI
better than TRI), but without distinction between task-relevant and task-irrelevant features
as found under same-task conditions (TRI, TRI and TRI not significantly different from
each other). Under different task conditions, our notations here denote as task-relevant a fea-
ture that is relevant to the secondary tasks. Different-task thresholds were often higher than
27
the corresponding same-task thresholds (less so for drift secondary), indicating an additional
cost of performing our two different tasks (indeed, subjects reported these conditions to be
harder than the same-task conditions). Some of this cost may not be considered purely atten-
tional and could be attributed to other processing limitations, including decision processes,
short-term memory, overall combined task load, or the ability of switching from primary to
secondary in different tasks.
Our secondary thresholds results suggest a complex pattern of interactions between tasks and
stimulus features in determining the perceptual consequences of feature-based attentional
modulation. Overall, these results are highly consistent with previous reports for individual
pairwise comparisons (Watanabe et al., 1998; Huk and Heeger, 2000; Saenz et al., 2003;
Saffell and Matthews, 2003). Most of the primary thresholds also exhibit similar trends with
secondary thresholds (Primary thresholds are reported together with the model). Our dataset
of 64 conditions under consistent viewing conditions, stimuli, tasks and observers, allows
us to derive a simple functional model for the observed effects. This simple model will
accommodate both the primary thresholds and secondary thresholds, which assumes primary
thresholds are also somewhat modulated by feature-based attention triggered at the secondary
tasks.
28
3.5 Model
Qualitatively, our data suggests that the interaction between attention, stimulus and task obeys
three governing principles: First, engaging attention onto a stimulus through a pattern dis-
crimination task enhances task-relevant features strongly and task-irrelevant features weakly,
throughout the visual field. Second, performance at a pattern discrimination task benefits
strongly if task-relevant features are enhanced, and weakly if task-irrelevant features are en-
hanced. Third and finally, feature enhancement and feature benefit combine multiplicatively,
with the final benefit to perception and task performance dictated by the largest of these prod-
ucts.
To further formalize these observations and derive a computational model, consider a pattern
discrimination task t and a visual feature f represented in the observer’s visual system. Let
r(f;t) 2 [0::1] denote the task-relevance of f to t, and A(t) 2 [0::1] be the fraction of the
observer’s attentional (including spatial) resources devoted to t. We express the amount of
non-spatial enhancement E(f;t) in the cortical representation of feature f resulting from de-
voting fraction A(t) of resources to task t as:
8f;t; E(f;t) = r(f;t)A(t) (3.1)
Now consider another task t
0
on a stimulus at the other site. The most parsimonious formula-
tion of how performance of t
0
may be influenced by simultaneous performance of t includes
29
at least three terms: First, there is a fraction A(t
0
) of resources available to t
0
. This A(t
0
)
should be observer-dependent, and A(t
0
) = 1 - A(t) in each dual-task condition. Second,
t
0
may benefit from possibly enhanced feature representations as attention is partly engaged
onto t. Third, we assume a simple gain value applied to the orientation or drift thresholds
to represent the subject’s ability to perform a certain combination of dual tasks. Certainly, the
gains are different when subjects perform two identical tasks compared to two different tasks.
Based on our observations, our model tests a maximum (or winner-take-all) rule where total
expected benefit is the maximum, over all features, of the product between feature enhance-
ment due to t and relevance of that feature to t
0
. Certainly, the contribution of these factors
to task performance depends on complex and nonlinear processes (Lee et al., 1999; Itti and
Koch, 2001), not fully elucidated and constrained by our data alone. Thus, our model’s for-
mulation only attempts a first-order approximation, with the main merit of clearly identifying
key parameters and of providing a simplified understanding of their interactions. Predicted
dual-task discrimination threshold Th(t
0
jt) for t
0
given t then is:
Th(t
0
jt) =
Th
0
(t
0
)
A(t
0
)

1 + max
f
r(f;t
0
)E(f;t)
(3.2)
where Th
0
(t
0
) is the single-task threshold for t
0
. Here when t is the primary task, t
0
is the
secondary task, and conversely.
30
To test the validity of this first-order formulation, we progressively reduce the number of free
parameters. First we assume that, in all eight conditions (TRI, TRI, TRI, TRI, TRI, TRI,
TRI, and TRI), the attentional allocations A(t) and A(t
0
) to both sides may differ. The fitting
procedure employed a two-dimensional Newton-Gauss optimization, with as objective func-
tion the root-mean-square difference (in units of measurement standard error) between the
sixteen model predictions (eight primary thresholds, eight secondary thresholds) and the mea-
sured thresholds. We assume a relevance of 1 for task-relevant features (e.g., r(V;OV S) =
1), 0.2 for task-irrelevant features (e.g., r(S;OV S) = 0:2), and 0 for features antagonistic to
either task-relevant or task-irrelevant features (e.g., r(H;OV S) = r(F;OV S) = 0). These
values were invariant across all conditions. Making the value for task-irrelevant features an
additional optimization parameter instead of 0:2 yielded the average values equal to 0.2 with
small variance, hence the value used here. Thus, the model has 10 free parameters (A(t) or
A(t
0
) for all eight conditions and two gain values for same and different tasks) for each set of
sixteen datapoints (eight primary and eight secondary thresholds averaged from the 64 total
task conditions). Overall, this many-parameter model suggests a good agreement with our
empirical data (R
2
= 0:981). The model fitting is done independently for each subject.
The parameters from the many-parameter fitting suggest that the different A(t) (also A(t
0
))
obtained from the eight different conditions are similar, which means that subjects divided
attentional resources into primary and secondary tasks similarly in each condition. This mo-
tivates us to collapse these eight A(t) parameters into one single parameter. Further reduction
of the number of free parameters assumes that subjects divided attention into two sides with
31
Orientation A(T R I) A(T R I) A(T R I) A(T R I) A(T R I) A(T R I) A(T R I) A(T R I) s
d
JW 0.63 0.68 0.63 0.66 0.70 0.71 0.71 0.72 1.24 1.30
JJ 0.57 0.48 0.60 0.69 0.74 0.75 0.72 0.74 0.75 0.92
SQ 0.64 0.66 0.61 0.64 0.67 0.69 0.48 0.70 0.82 0.94
Drift A(T R I) A(T R I) A(T R I) A(T R I) A(T R I) A(T R I) A(T R I) A(T R I) s
d
JW 0.59 0.57 0.60 0.61 0.64 0.66 0.65 0.66 0.78 0.98
JJ 0.50 0.53 0.59 0.60 0.58 0.56 0.59 0.56 0.92 1.01
SQ 0.55 0.54 0.58 0.59 0.57 0.56 0.56 0.58 0.76 0.79
Table 3.2: All parameters of the model fits with 10 free parameters for orientation and drift tasks for
all three subjects. A represents the fraction of attentional resources denoted to primary task in that
condition. The corresponding attentional resources to the secondary side in that condition is 1 A.

s
is gain to orientation or drift thresholds for a certain task combination in the same-task conditions
and
d
is gain to orientation or drift thresholds for a certain task combination in the different-task
conditions. Note that in each condition,
s
is always less than
d
, indicating that splitting attention
across different tasks yield higher gain value and higher additional cost. The parameters suggest that
in all the eight conditions, subjects divided attentional resources between both sides similarly (similar
A values across conditions), suggesting a model fit with fewer parameters as described in Table 3.3.
Orientation Drift
A
s

d
A
s

d
JW 0.70 1.13 1.33 0.63 0.77 1.01
JJ 0.71 0.69 1.00 0.58 0.93 1.00
SQ 0.68 0.74 0.96 0.57 0.77 0.79
Table 3.3: All parameters of the model fits using three free parameters, for orientation and drift tasks
for all three subjects. Notations are as in Table 2.
the same ratio in all conditions. This few-parameter model fitting with only three free param-
eters still results a good fitness of the model with the data (R
2
= 0:956). Table 3 lists the
fitting parameters for the three subjects.
Figure 3.5 shows model predictions when only these three free parameters were fit to each
set of 16 observations from each observer. Overall model prediction accuracy was close to
measurement error. The values obtained for A(t) suggest that, as instructed, subjects devoted
more attentional resources to the primary task (average A(t) = 64%, range [57%::71%],
32
compared to average A(t
0
) = 36%, range [29%::43%]). Since the gain parameter in same-
task conditions is always smaller than the corresponding different-task conditions, it indicates
that there is an additional impairment for performing two different tasks. We have same as-
sumption a relevance of 0.2 for task-irrelevant features. Thus, the model has three parameters
for each set of sixteen datapoints plus one global parameters for all thresholds. Each model
prediction was, on average, 2.01 measurement standard error from the corresponding dat-
apoint (range [0.008 .. 9.06]), and of 96 model predictions (3 subjects and 32 predictions
each), the model overestimated the data in 43 and underestimated in 53 cases.
3.6 Discussion
Our systematic dissociation between tasks and stimulus features in a single dataset unraveled
a seemingly complex pattern of interactions between behavioral demands and visual inputs.
Our data and model, however, suggest that the interaction between attention, stimulus and
task is well captured by a simple descriptive model. The model predicts a maximum-rule
by which only the dominant product of feature enhancement and feature benefit by feature
relevance may benefit perception. Previous studies, using isolated pairs of tasks and stimuli,
illustrated how both task-relevant and task-irrelevant features may be enhanced by attention.
Our results not only confirm this fact, but also show that task-relevant and task-irrelevant
features are enhanced with different gains, and that discrimination performance benefits dif-
ferently from possibly enhanced task-relevant and task-irrelevant features.
33
Our study of same-task conditions is consistent with recent findings from human psychophysics
employing the 2IFC paradigm that dual-task performance was better when subjects attended
to a same feature on both sides of a dual display, compared to when they attended to two
different features (Saenz et al., 2003). On the other hand, feature-based attentional effect
was also found in an fMRI experiment (Saenz et al., 2002) with the attended stimulus on
one side and an ignored stimulus on the other side, which suggested that feature-based at-
tention may not be due to the specific property of dual tasks. Further, under different-task
conditions, our results are also consistent with the literature. For example, Rossi and Par-
adiso (Rossi and Paradiso, 1995) reported how when subjects performed a primary task of
discriminating spatial frequency or orientation of a foveal grating, a secondary task of detect-
ing a near-threshold grating in the periphery benefited when the peripheral grating’s spatial
frequency or orientation matched the attended feature of the primary stimulus. While with
our tasks of orientation and speed discriminations we have observed an impairment for dif-
ferent compared to same task conditions, it is important to note that this may not necessarily
be the case for all task combinations. For example, little or no impairment has been observed
in concurrent performance of a visual and an auditory discrimination tasks (Duncan et al.,
1997). Mixing two visual tasks, Morrone et al. (Morrone et al., 2002) found no impairment
in concurrent mixed color and luminance contrast discriminations, compared to two color or
two luminance discriminations. This suggests that the amount of impairment from perform-
ing two different tasks compared to twice the same task also is highly task-dependent. In
another study, Morrone et al. reported no significant effect on the secondary luminance dis-
crimination task when they varied the primary form or contrast discrimination task relevance
34
and difficulty (Morrone et al., 2004). These results are partly consistent with our findings
(the insignificant difference among thresholds in different-task conditions TRI, TRI, and
TRI in our study) and partly inconsistent (TRI has higher thresholds than the previous three
conditions). This is possibly because Morrone et al. used a central vs. peripheral dual task
paradigm while we employed two symmetrical peripheral tasks. Another reason could be the
nature of the stimuli. In their experiment, the central task is to discriminate form of stimuli
or contrast of form stimuli while the peripheral task uses sinusoidal gratings. It is possible
that engaging attention onto a form or contrast discrimination yields a different pattern of
enhancement than the orientation and speed discrimination tasks used in our study.
Several studies have shown that human observers performed better when concurrently dis-
criminating two features of the same object compared to two features of different objects
(Blaser et al., 2000; Duncan, 1984). In our experiment, because two moving Gabor patch
stimuli could be perceived as part of a common object viewed through two apertures, our
findings could also be attributed to an object-based attention effect. But in many visual ob-
jects, the object often is defined by sharing of common features. However, we have no enough
evidence that in our study subjects perceived left and right stimuli as part of a common object
or not.
Our results contrast with those of Saenz et al. (Saenz et al., 2003) in that effects were ob-
served even though we did not require that competing stimuli be present and overlapping
with the attended stimuli. It is possible that this difference is in part due to a different choice
35
of tasks used to engage attention onto specific features of stimuli. Saenz at al. use a speed
discrimination task to engage attention. However, it could be argued that this task may not
necessarily enhance the cortical representation of stimulus features like direction of motion
or color, which seem irrelevant to the task. This could explain why no feature-based en-
hancement was observed in this simple situation when both stimuli shared the same direction
of motion or color. Adding competing stimuli (moving in the opposite direction or hav-
ing a different color), then, modified the task by adding a requirement on the observers to
separate between the component of the stimuli onto which they performed the task (e.g., up-
ward moving dots or red dots), and the competing component which they should ignore (e.g.,
downward moving dots or green dots). This separation task required active discrimination of
direction of motion and color, which were the features tested for when evaluating possible
feature-based attention effects.
Task relevant and irrelevant features in our study are defined by whether the shared feature
in the two stimuli is relevant or irrelevant to one of the dual tasks. As subjects performed
two tasks at the same time, feature-based attention would predict strongest effects when a
feature at one site that is relevant to the specific feature attended at the other site. In our study
features always were enhanced (attended) at one site, resulting feature-based attention at the
other site. But in another fMRI study, the fMRI response to the unchanging, ignored stimu-
lus in the opposite visual hemifield was increased when observers attended the same features
(direction of motion, color et) compared to the opposing features (Saenz et al., 2002). These
36
results suggested that the relevant or irrelevant features could also be defined based on a sin-
gle task. Another concern is the long exposure of the stimuli. To prevent eye movements
across the two tasks, we asked subjects focus on the central cross while they performed the
tasks. Since we do dual tasks with both tasks being difficult, the resulting thresholds with
shorter presentation of the stimuli were very poor both in primary and secondary tasks. In-
deed, subjects reported they had difficulties to perform the two tasks simultaneously when
the presentation of the stimuli was shorter. In principle, we want to keep the primary perfor-
mance as good as possible, which required that we both elongate the exposure of the stimuli
and the of the blank period between the two presentation periods. Despite this relatively long
exposure and inter-period blank of the stimuli, subjects still reported that the task was very
demanding, which limited the possibility for eye movement. It should however be noted that
our rather long stimulus presentation times may slightly blur the distinction between feature-
based attention effects during stimulus encoding and comparison with possibly other effects
occurring during stimulus maintenance. These issues will be addressed in future experiments.
Computationally, our observations and model are compatible and extend the recent study of
Huk and Heeger (Huk and Heeger, 2000), which suggested that increasing the gain of neu-
rons encoding task-relevant features may improve their signal-to-noise ratio and hence help
improve task performance. Our results are also consistent with a recent study of percep-
tual learning (Saffell and Matthews, 2003) in which subjects trained to discriminate stimu-
lus direction showed significant improvement in direction discrimination, but their speed or
37
luminance discrimination performance did not improve; conversely, subjects trained to dis-
criminate speed improved in speed but not in direction or luminance discriminations (Shiu
and Pashler, 1992). In addition, our systematic dissociation between task and features ex-
acerbates the double role of task-relevance, not only in determining the strength at which a
feature may be enhanced by engaging attention onto a stimulus and task, but also how in
determining how much a task may benefit and performance may improve from an enhanced
feature.
Thus our results suggest a neuronal mechanism by which attention enhances the activity of
cortical neurons that encode behaviorally relevant stimulus properties. This is highly remi-
niscent of the biased competition model of Desimone and Duncan (Desimone and Duncan,
1995; Reynolds et al., 1999), which proposed that multiple stimuli activate competing pop-
ulations of neurons (at one visual location) and attention biases the competition in favor of
neurons that prefer task-relevant features. This was further demonstrated in that engaging
attention onto a neuron’s preferred stimulus increased the neuron’s firing rate, while attend-
ing to a non-preferred stimulus decreased firing rate (Luck et al., 1997; Treue and Maunsell,
1996). Therefore the effect of attention on a neuron’s response (enhancement or suppression)
depends on how the features of the attended stimulus match the feature selectivity and pref-
erence of the neuron. Our results extend these findings beyond localized competing stimuli
within one neuron’s receptive field, to distant stimuli across the visual field.
38
In that sense, our model is consistent with and extends the ’feature similarity gain model’
for feature-based attentional modulation (Treue and Martinez-Trujillo, 1999), and exacer-
bates the critical role of task-relevance, not only in determining feature-based attentional
modulation strength (through r(f;t)), but also in determining possible benefits of a modu-
lation onto visual perception (through r(f;t
0
)). While task may have previously often been
chosen arbitrarily for the sole purpose of engaging attention onto or away from a stimulus,
our study suggests that it has very strong consequences onto both triggering and benefiting
from attentional modulation. For example, our model predicts that performing a luminance
increment detection task on a stimulus of given color only yields weak feature-based atten-
tional enhancement of the cortical representation of that color, because color is irrelevant to
the task; a stronger modulation would be expected if attention was engaged through a color
(hue) discrimination task; conversely, our model predicts that enhancing the representation
of a given color only weakly improves performance at a luminance increment detection task
on a stimulus of that color, but would yield greater improvement on a hue discrimination task.
We have previously (Lee et al., 1999) proposed that spatial (not feature-based) attention ac-
tivates a winner-take-all competition among neurons with overlapping receptive fields. The
maximum rule found here for feature-based attention is compatible with this mechanism. The
neuronal mechanism of the maximum rule proposed here for feature-based attention may
be that the multiplicative effect of feature enhancement and feature benefit across features
also activates some winner-take-all competition among features. This competition could take
place either at the level of early sensory coding or at a higher level, by which all the relevant
39
and irrelevant feature enhancement and feature benefit may take place but only the largest
multiplicative effect wins the competition and eventually affects behavior.
In sum, our systematic dissociation between tasks and stimulus features in a single dataset
unraveled a seemingly complex pattern of interactions between behavioral demands and vi-
sual inputs. Our data and model, however, suggest that the interaction between attention,
stimulus and task follows three simple functional principles: engaging attention onto a stim-
ulus by a task enhances task-relevant features of the stimulus strongly but its task-irrelevant
features weakly, throughout the visual field; performance at a (possibly different, distant and
concurrent) pattern discrimination task benefits strongly if visual features relevant to that task
are enhanced, but only weakly if task-irrelevant features are enhanced; and, finally, feature
enhancement and feature benefit combine multiplicatively, with the final benefit to percep-
tion and task performance dictated by the largest of these products. This model allows us to
recast a significant body of recent studies of attentional modulation within a simple unifying
computational framework. This framework may be used to guide future electrophysiology,
imaging and behavioral studies of the perceptual consequences of attention.
40
0
1
2
3
Threshold Elevations
0
1
2
3
Threshold Elevations
0
1
2
3
Threshold Elevations
Subject JJ
Subject JW
Subject SQ
Example:
Figure 3.4: Drift speed secondary task performance. An example task and stimulus combination is
listed below the figure to show the eight conditions. In this example, the secondary task is always
DVS and the primary tasks are DVS, DHS, DVF, DHF, OVS, OHS, OVF, and OHF corresponding to
the eight conditions TRI, TRI, TRI, TRI, TRI, TRI, TRI, and TRI.
41
0
2
4
0
2
4
Orientation Thresholds: Subject JW
Primary
Secondary
Threshold Elevations Threshold Elevations
T R I T R I T R I T R I T R I T R I T R I
_ _ _ _ _ _ _
0
2
4
0
2
4
Orientation Thresholds: Subject JJ
Primary
Secondary
Threshold Elevations Threshold Elevations
0
2
4
0
2
4
Orientation Thresholds: Subject SQ
Primary
Secondary
Threshold Elevations Threshold Elevations
T R I T R I T R I T R I T R I T R I T R I
_ _ _ _ _ _ _
T R I
_
T R I
_
0
1
2
3
0
1
2
3
Drift Thresholds: Subject JW
Primary
Secondary
Threshold Elevations Threshold Elevations
0
1
2
3
0
1
2
3
Drift Thresholds: Subject JJ
Primary
Secondary
Threshold Elevations Threshold Elevations
0
1
2
3
0
1
2
3
Drift Thresholds: Subject SQ
Primary
Secondary
Threshold Elevations Threshold Elevations
T R I T R I T R I T R I T R I T R I T R I
_ _ _ _ _ _ _
T R I
_
T R I T R I T R I T R I T R I T R I T R I
_ _ _ _ _ _ _
T R I
_
T R I T R I T R I T R I T R I T R I T R I
_ _ _ _ _ _ _
T R I
_
T R I T R I T R I T R I T R I T R I T R I
_ _ _ _ _ _ _
T R I
_
T R I T R I T R I T R I T R I T R I T R I
_ _ _ _ _ _ _
T R I
_
T R I T R I T R I T R I T R I T R I T R I
_ _ _ _ _ _ _
T R I
_
T R I T R I T R I T R I T R I T R I T R I
_ _ _ _ _ _ _
T R I
_
T R I T R I T R I T R I T R I T R I T R I
_ _ _ _ _ _ _
T R I
_
T R I T R I T R I T R I T R I T R I T R I
_ _ _ _ _ _ _
T R I
_
T R I T R I T R I T R I T R I T R I T R I
_ _ _ _ _ _ _
T R I
_
Figure 3.5: All the primary and secondary performance for both orientation (left) and drift (right)
tasks and their model predictions. Cyan wide bars represent the threshold elevation performance and
the inside narrow bars represent the corresponding model prediction. The model was fitted with three
free parameters (A(t): attentional resources to primary task, so that the corresponding attentional
resources to the secondary task in that condition are A(t
0
) = 1 A(t));
s
: gain to orientation or
drift thresholds for a certain task combinations in same-task conditions;
d
: gain to orientation or drift
thresholds for certain task combinations in different-task conditions different task thresholds, Tab. 3.3)
for each of the 16 datapoints (eight primary thresholds and eight secondary thresholds) independent of
subjects. Overall model prediction accuracy was close to measurement error and suggested a good fit
(R
2
= 0:956). Note that for primary thresholds the TRI related labels represent relevant or irrelevant
to the primary tasks.
42
Chapter 4
Feature-based Attention Is Not Object Based
4.1 Abstract
Feature-based attention revealed that attention can enhance attended features at the ignored
locations. Object-based attention showed that attention can enhance the ignored features on
the same object with the features at the attended positions. We used fMRI to investigate
whether feature-based attention is also object-based, i.e., feature-based attention is a specific
form of object-based attention or a new type of lower feature based attentional selection. The
stimuli were two fields of random dots presented bilaterally to a central fixation cross. Sub-
jects performed luminance discrimination using a two-interval forced choice paradigm on
one side and ignored the stimuli on another side. The ignored stimulus was always red dots
and the attended stimulus was overlapping red and green dots. We compared visual cortical
enhancement of the ignored stimulus when subjects attended same feature (red) comparing
with they attended different feature (green) on the other side in two conditions: either the dots
stimuli on both sides appeared to belong to same object or as two separate objects. Results
43
showed both in same-object condition and in different-object condition the four subjects con-
sistently had significant enhancement of the ignored stimulus in early visual areas. Hence it
indicated feature-based attentional enhancement exists even when two stimuli belong to two
different objects, suggesting it is a new type of attentional selection which takes place in the
low feature levels, not dependent on the objectness of the two features.
4.2 Introduction
Attentional selection is processed by three different strategies: A space selection which at-
tention enhances a spotlight at the position of attention (Motter 1994; Treue and Maunsell,
1996; Brefczynski and DeYoe, 1999; Lee et al., 1999), an object selection which objects are
selected as an entity (Desimone and Duncan, 1995; Duncan 1984), and a feature selection
which attentional modulation enhance the attended feature in the global visual cortex (Saenz
et al., 2002, 2003; Lu and Itti, 2005).
Here we describe a recent study which observed that attention to a stimulus feature (color
or direction of motion) increased the response of cortical visual areas to a spatially distant,
ignored stimulus that shared the same feature (Saenz et al., 2002). This study was done by
using functional magnetic resonance imaging (fMRI). In experiment subjects were asked to
attend to one direction of motion within two overlapping fields of upward and downward
moving dots on one side of a central fixation point, then ignore a single field of dots moving
up or down on the other side. Dots on the ignored side did not change direction. The task
is speed discrimination. The result illustrated that all visual areas responded more strongly
44
to the ignored stimulus when it moved in the same direction as the target side. This global
effect was generalized by using color as the attended feature in a second experiment with
three subjects. The results were similar: fMRI response amplitudes to the ignored stimulus
were stronger during the ’same’ condition, when its color matched the attended color.
In another typical physiological study, feature-based attention was demonstrated to increase
the gain of neural responses in direction selective cells of middle temporal areas (MT) in
Macaque (Treue and Martinez-Trujillo, 1999). This feature-based attention was also sup-
ported by many psychophysical studies (Saenz et al., 2003; Sohn and Vidnyanszky 2005; Lu
and Itti, 2005). On the other hand, there have been broad psychophysical and physiological
evidences supporting that attentional selection takes place in the object level. Psychophysical
studies showed that performance on two features of same objects is significantly better than
two features of different objects (Blaser et al., 2000; Duncan 1984; Duncan 1996; Reynolds
et al., 2003). Studies also indicated attention selection of a specific feature of an object also
enhance the processing of other features of the same object (Sohn et al., 2004; Sosa et al.,
1998), even the features are task irrelevant (Craven et al., 1999). Further study indicated that
the object-based attentional selective ability is quite strong that it can even enhance uncon-
scious features of that object as well (Melcher and Vidnyanszky, 2005). Comparative studies
provided evidence that attentional selection biases on more on object level than feature level.
For example, attention can rapidly track an object through feature space, even when dis-
tracters occupy the location of the attended object (Blaser et al., 2000).
45
Figure 4.1: Feature-based attention to motion. a) Stimuli (not to scale) were circular apertures (radius
5) of coherently moving random dots in the lower visual field (2.5 below and centered 11 to left and
right of fixation, baseline speed 10/s). Arrow, field of 50 dots moving upward or downward. Dots were
white (560 cd/m2) on a gray background (230 cd/m2). Dashed circle (not present in the experiment),
spatial focus of attention. (b) fMRI time series of BOLD response (same versus different) to ignored
stimulus for MT+, averaged across three subjects and 24 repetitions per subject. (c) Response am-
plitudes to ignored stimulus. (d) Response amplitudes to attended stimulus. (e) Attentional response
amplitudes as a percentage of stimulus-evoked response. Data in (cCe) are mean s.e.m. During each
trial, stimuli were presented for two sequential 1-s intervals separated by a 100-ms interval in which
only the fixation point was present. Trials started every 3.3 s. The order of speeds was independently
randomized for each field of dots on every trial, and the baseline speed was randomly and indepen-
dently jittered across trials in all three fields of dots. Scans were counterbalanced for the attended
side (left/right), the starting attended direction (up/down) and the direction of motion on the ignored
side (up/down). Three subjects with normal visual acuity participated, and all gave written informed
consent. Figure is reprinted from the M. Saenz’s Nature Neuroscience paper in 2002. (Saenz et al.
2002)
The common features between feature-based attention and object-based attention is attention
both can enhance the same feature with attended features outside the attentional receptive
field. But the difference is object-based attention only enhance the features which belong to
46
D E Figure 4.2: Figure showing objectness visualization of the two fields of dots stimuli. (a) One-object
choice: the subject groups the two fields of dots as one single object. (b) Two-object choice: the
subject doesn’t group the two fields of stimuli, instead treat them as two different objects.
the same object with the attended features at the attended position. Considering the previous
typical studies on the feature-based attention (Saenz et al., 2002; Treue and Martinez-Trujillo,
1999), the dots stimuli used were normally displayed in the blank black background, thus the
47
two features on the two sides were not clearly object segmented. Especially, previous study
also implied that features belonging to the same object are bound and selected jointly outside
the focus of attention (Sohn and Vidnyanszky, 2005). This raised a fundamental question:
since object-based attention can also enhance the attended features at the ignored position
and it showed a so strong selective ability, is it possible that the feature enhancement by the
feature-based attention is because subjects grouping the two features as same object so that
it is actually a specific form of object-based attention, or due to a completely different type
of attentional selection mechanism by which attentional selection indeed process individual
features as a unit? In the neural level, the question would be for the feature-based attention,
does attention boost the attended features in all the visual perceptual field or just boost the
attended features which belong to the same object containing the attended features at the at-
tended position.
Using fMRI study, we duplicated the color dots experiment in the previous study on the
feature-based attention (Saenz et al., 2002) in two conditions: either the two features dis-
played as in same object or they appeared as different objects. Based on the two possible
hypotheses, if attentional selection is processed on the object level, the feature enhancement
modulated by feature-based attention should become lower or disappear when the features
displayed as different objects. In contrast, the feature-based attentional enhancement may
still exist in the different-object conditions. Our results below showed that the effect of
feature enhancement consistently exist in both the same-object condition and different-object
48
condition, with even larger enhancement in different-object condition, suggesting the feature-
based attention takes place indeed at the low feature levels.
4.3 Methods
4.3.1 General introduction of functional Magnetic Resonance Imaging
Magnetic Resonance Imaging techniques won the Nobel Prize in 2003. It is primarily a
method to record and visualize the inside of the living organisms. The MRI technique is built
based on the relaxation properties of excited hydrogen nuclei. The hydrogen nuclei have the
intrinsic quantum property called spin. When the tissue is placed in the magnetic field inside
the MRI machine, the spins of the atomic nuclei within the tissue will align in parallel to
the magnetic field or anti-parallel. During the high-energy nuclei relax and realigned, they
provide energy which can be recorded. The realigment of nuclear spins within the magnetic
field is called longitudinal relaxation. The image acquired from this relaxation is called T1-
weighted image. T2-weighted image is relied on local dephasing of spins of the transverse
pulse. The strength of the signal is different for different tissues, thus creating the contrast in
the MRI image. The image is reconstructed by the inverse Fourier transform to the signal.
FMRI is a technology developed not long ago. In 1990 the study showed that in the rat brain,
the blood oxygenation can be measured on the MRI signal (Ogawa, Lee et al. 1990). After
that, other studies started to show that the MRI signal increased in the visual and motor cortex
49
Figure 4.3: A typical MRI MPRAGE image taken from our study.
if the sensory neurons were stimulated (Kwong, Belliveau et al. 1992; Ogawa, Tank et al.
1992). After improving the imaging speed by some techniques, the MRI scanning started to
be used in the current medical center and hospitals in the form of the functional MRI.
Functional MRI meansures the signal change due to the level of blood oxygen, therefore
reflecting the neural activity in the brain. Increasing neural activity will result increasing
the demand of the oxygen, thus the vascular system will compensate for this demand, result-
ing the increasing tof the oxygenated hemoglobin. There have been many evidence indicating
that the blood oxygenation dependent signal (BOLD) response is approximately a linear func-
tion of he neural activity. Further study also showed that the BOLD response to the visual
50
Figure 4.4: A typical MRI scanning machine setup.
stimuli is linear function with the repeated individual stimulus (Dale and Buckner 1997) and
also with the stimulus duration (Boynton, Engel et al. 1996). In another study, Heeger et al
showed the human V1 contrast response functions linearly with the single-unit data, there-
fore providing good evidence for the linear property between the BOLD signal the neuronal
activity (Heeger, Huk et al., 2000).
Our fMRI scanning took place in the Dana and David Dornsife Cognitive Neuroscience Imag-
ing Center at USC (http://brainimaging.usc.edu).
4.3.2 Our fMRI method
Subject: Five adults participated in the fMRI experiments (three male). One subject were
excluded because of technical problems, leaving four subjects for final analysis. The study
was approved by the University of Southern California Institutional Review Board. Written
51
6KLIWLQJDW+= Figure 4.5: Localizer stimuli in our study. We used 3x3 checkbox pattern stimuli (5HZ counter phase
flicker) to trigger as much as BOLD activity in our stimuli position (region of interest). Since we used
green and red dots as the real stimuli, so in our localizer stimuli we also used the red and green box to
match the real dot stimuli.
informed consent was obtained from all participants.
Localizer: In the fMRI study, the localizer is referred to the way we map our stimuli position
on the computer screen display to the position in our visual cortex. According to the theory of
the retinotopic organization, the visual input information of the outside world was mapped to
52
visual cortex in a specific retinotopic organization. As one of the typical visual topographical
studies, Brefczynski and DeYoe found that cortical topography of the purely attention-driven
activity is precisely matching the topography of the pure visual targets without attention (Bre-
fczynski et al., 1999). In our study, we used the 3x3 checkbox pattern stimuli (5HZ counter
phase flicker) to trigger as much as BOLD activity in our stimuli position (region of interest).
Because in our study we use green and red dots as the real stimuli, so in our localizer stimuli
we also used the red and green box to match the real dot stimuli.
Stimuli and Task: The stimuli were two fields of random distributed stationary dots pre-
sented bilaterally to central fixation cross. On the attended side, the display was overlapping
red and green dots (120 dots respectively). The ignored side always had 120 red dots. To
enhance the visualization of the dots, each dot was enlarged to a disk covering four(?) pixels.
The distance between any two disks was restricted to a minimal value so that dots would not
overlapping each other and spread into the whole field space widely. The dots displayed in
the limited lifetime (200ms) so that they appeared as a flicker to prevent the eye movement.
The baseline luminance on the two sides was randomized across trials, so subjects could not
compare luminance between two sides to benefit the task performances. The center of the
dots fields was at 6 degree eccentricity from fixation, and the radius of the dots field were at 4
degree eccentricity from the field center. In the same-object condition, the two fields of dots
were displayed in single whole gray box appearing on top of a textured background. In the
different-object condition, they were displayed respectively in two small boxes on the same
53
Figure 4.6: A 1-D view of the regions of interest (shown in the red color) triggerred by the localizer
stimuli for three of the subjects. The ROI size of each subject varied from 2455 voxels to 3572 voxels
(1x1x1mm each voxel)
background. A shadow effect was cast around every box to enhance the objectness visualiza-
tion. The shadow was composed of same texture of the background but with less luminance.
Subjects were instructed to perform a two-interval forced-choice (2IFC) luminance discrim-
ination task on either red dots or green dots on the attended side, while fixated on the central
54
D E F G PV PV PV PV 5HVSRQVH Figure 4.7: Experimental stimuli and frame sequences. (a) The two-interval force-choice task frame
sequences in the same-object condition. Within one TR(Time of repetition = 3s) period, the first
stimulus frame displayed for 350ms, followed by a 150ms inter-frame with same background but no
dots stimuli(blank frame). Then the second stimulus frame appeared for 350ms and finally the blank
frame was showed for 2150ms for the subjects to judge whether the first or second stimulus frame
had higher luminance for either red dots or green dots at the attended side (left side in the figure). (b)
The shadow effect of the stimuli. The shadow effect was made with the same background texture but
less luminance (0.6 of the textured background luminance) to enhance the objectness visualization.
(c) The experimental stimuli in the different-object condition. The frame sequences are exactly with
those in the same-object condition shown in (a). (d) The localizer stimuli. The checkbox stimuli were
made of 3x3 square frames in each of which the red and green square flashed at 5Hz frequency.
55
cross. There were no tasks on the ignored side. In every 2IFC trial, subjects report whether
a higher luminance occurred first or a lower luminance occurred first. The initial luminance
difference between the higher luminance and the lower luminance was set to the threshold
level by pre-training the subjects individually, so that the task was hard enough to engage
strong attention on the attended side. The task difficulty during the scanning was also con-
trolled by a staircase to maintain the threshold level of the luminance discrimination. The
scanning sessions were selected as valid scans if the subjects’ psychophysical performance
maintaining their individual’s threshold levels, which made sure their attention was engaged
on the attended side.
fMRI Scanning and Analysis: The fMRI data were acquired on a Siemens 3T MRI scanner
(MAGNETOM Trio). BOLD-sensitive images were collected by echo-planar imaging (EPI)
sequence with a repetition time (TR) of 3 seconds. We recorded fMRI activity using 17 slices
to cover all the early visual cortex (3x3x3mm each voxel, TE = 30 ms). Before the functional
scanning, the MPRAGE structure of the whole brain was scanned using 192 slices (1x1x1mm
each voxel). Block design was used to study the BOLD activity on the ignored side for four
different blocks: 1. Subject attended same features (red dots) on the attended side when
the two dots fields appeared as same object (SOSF). 2. Subject attended different features
(green dots) on the attended side in the same-object condition (SODF). 3. Subjects attended
same features when the two dots fields appeared as different object (DOSF). 4. Subject at-
tended different features in the different-objects condition (DODF). Each block contained a
56
30-second task period (10 scans) plus a 3-second message period at the beginning of each ses-
sion for the task instruction (instruct the subjects to either attend on red or attend on green).
During each block the task remained identical. The task changed every 30 seconds to avoid
the fMRI adaptation. Prior to real task, each subject was scanned by displaying the localizer
stimuli to extract the region of interest (ROI) corresponding to the stimuli. A total number of
396 to 512 trials were acquired for the four blocks each subject.
The fMRI data were pre-processed (Mean intensity adjustment, Slice scan time correction,
3D motion correction, Spatial smoothing and Temporal filter) by BrainV oyager QX software.
The general linear model was employed to estimate the beta weight magnitude of BOLD
response for each of the four block conditions. A region of interest (ROI) was defined for
each individual subject by using the localizer stimuli, then the general linear model was fit
within the region of interest (see Results). Paired t tests were finally performed using the
regression coefficients for each subject.
4.4 Results
The figure showed a typical region of interest area selected for one of the subjects in the 3D
views. The region of interest was chosen by computing the linear correlation maps when
displaying the localizer stimuli versus displaying only the background and boxes without
stimuli. A minimal p < 0.00001 was used to define the region of interest for each subject.
The ROI size of each subject varied from 2455 voxels to 3572 voxels (1x1x1mm each voxel)
57
Figure 4.8: A typical feature-based attentional enhancement. The orange clusters in the activity map
showed the feature-based enhancement at the ignored side (here left side of the brain, right side of
the visual field) when the subject attended same feature compared to different features at the attended
side. The map is contructed at the uncorrect p < 0.000215.
The fMRI response time serials to the ignored stimulus after averaging all the scans of the four
subjects into a 132-second time period was plotted at the left-bottom of the figure. In the same
object condition, the fMRI signal response to ignored stimulus was stronger when subjects
attended same features (SOSF) comparing to when they attended different features (SODF).
This effect confirms the previous feature-based attentional enhancement. Interestingly, in
the different-object condition, the fMRI response enhancement to the ignored stimulus when
subjects attended same features (DOSF) comparing to different features (DODF) still existed
58
Time (s)
33 66 99 132
-0.2
-0.1
0
0.1
0.2
fMRI signal change (%)
0
SOSF SODF DOSF DODF
Attend
Red
Green
Red
Green
Figure 4.9: The left-bottom figure showed the fMRI response change when subject attended same
feature or different feature in both same-object condition and different-objects condition by averag-
ing from all the four subjects. The yellow bars indicate same-object condition. The blue bars indi-
cate different-objects condition. The figure indicates the fMRI BOLD signal when subjects attended
same feature (red) (SOSF) is higher than when they attended different feature (green) (SODF). In the
different-objects condition, the fMRI signal is much higher when subjects attended same feature (red)
(DOSF) than different feature (green) (DODF). This figure also displays a 3D view of the region of
interest (ROI) for one of our subject. The size of the ROI for this subject is 3510 voxels(1x1x1mm
each voxel).
and even showed much larger effect than in the same-object conditions. This result indicated
that the enhancement modulated by feature-based attention existed even when the two fea-
tures appeared as two different objects, suggesting feature-based attentional enhancement is
59
0
0.1
0.2
0
0.1
0.2
0
0.1
0.2
0
0.1
0.2
Subject JL Subject BH
Subject VN Subject ZL
beta weight differece beta weight differece
beta weight differece beta weight differece
SOSF-SODF
SOSF-SODF
SOSF-SODF
SOSF-SODF
DOSF-DODF DOSF-DODF
DOSF-DODF DOSF-DODF
**
***
**
***
**
***
*
*
Figure 4.10: The fMRI signal response to the ignored stimuli shown as beta weight magnitude (after
fitting GLM) difference when subject attended same feature minus they attended different feature in
both the same-object and different-object condition (SOSF minus SODF, DOSF minus DODF). For
all the subjects, the fMRI beta weight difference to the ignored stimuli between same feature and
different feature were significantly higher than zero (Paired t-test :
0

0
indicates p < 0.01,
0

0
indicates p < 0.05,
0

0
indicated p < 0.08) in both the same-object and different-object conditions,
suggesting the feature-based attention exists in both the two conditions.
not object-based. The following further results analysis confirmed this conclusion.
The results of individual subject consistently illustrated the non-object-based effect of the
feature-based attention. The beta weight difference between subject attended same features
and different features were plotted for each subject after general linear model (GLM) fitting
60
procedure. The GLM was fitted for each subject by combining all the scanning study ses-
sions within their individual region of interest. To do the paired t-test, we also fitted GLM
on every session of each subject (9-12 sessions per subject). Each session is one single cycle
of the four conditions (SOSF, SODF, DOSF and DODF). The paired t-test was performed
on the beta gain collected from GLM on each session between same feature and different
in both same-object and different object conditions, indicating a p < 0.08 in the worst case.
Thus the difference between same feature and different feature in both same-object condition
and different-object condition were significant for most of the four subjects. In the same-
object condition, there were significant enhancement for all the four subjects. This effect
is consistent with the previous fMRI study which revealed the feature-based attentional en-
hancement in blank black background (Saenz 2002). More importantly, we also observed
feature-based attentional enhancement in the different-object condition for all the subjects,
with the enhancement even much larger (plotted as blue bars in the figure. Statistical test
in the different-object conditions indicated all subjects showed a more significant effect in
the beta weight difference between subjects attended same feature versus different feature
(paired t-test p < 0.05 at worst). The beta weight were obtained by using 396 to 512 scan
points for each subject. These results indicated that the feature-based attentional effect exist
consistently across subjects in both same-object and different-object condition, suggesting
the feature-based attention is not dependent on the objectness of the two features.
61
The figure 4.11 showed the overall averaged fMRI BOLD response to the ignored stimu-
lus across all the subjects after the GLM fitting procedure. As a control, the final subject-
averaged BOLD response to the attended stimulus was also plotted in (b). There were signif-
icant enhancement between subjects attended same features and attended different features
both in same-object condition and different-object condition on the ignored side (unpaired t-
test performed on 39 datapoints on each condition from all the four subjects showed p < 0.05
in the same-object condition and p < 0.008 in the different-object condition). On the other
hand, the fMRI response on the attended side doesn’t enhance significantly when subjects
attended same features comparing with they attended different features for both conditions (p
> 0.1 in both conditions by same unpaired t-test as above), indicating attention was equally
engaged onto the attended task in all the conditions. The control fMRI response results to
the attended stimulus were consistent with the previous study on the feature-based attention
(Saenz et al., 2002) and provided evidence that the attentional enhancement in our study
was identical with the previously revealed feature-based attention. Our results above demon-
strated that the global effect of the feature-based attentional selection takes place in the low
feature levels, not dependent on the objectness of the two features.
62
SOSF SODF DOSF DODF
SOSF SODF DOSF DODF
fMRI response (% BOLD signal)
0
0.2
0.4
0.6
fMRI response (% BOLD signal)
0
0.2
0.4
0.6
Ignored Side Attended Side a.
b.
*
**
QV QV Figure 4.11: The overall averaged results from all the four subjects. The left figure shows the fMRI
response (beta weight average) to the ignored stimuli and the right figure shows the fMRI response
to the attended stimuli (SOSF: subjects attended same feature in the same-object condition; SODF:
subjects attended different feature in the same-object condition; DOSF: subjects attended same feature
in the different-objects condition; DODF: subjects attended different features in the different-objects
condition). Results indicated in both the same-object condition and different-objects condition, the
fMRI response to the ignored stimuli is significantly higher when subject attended same feature than
when they attended different feature (left). As a control, the fMRI response has no significant en-
hancement to the attended stimuli when same feature vs. different feature. (By t-test,
0

0
indicates
p < 0.008,
0

0
indicates p < 0.05, ’n.s.’ indicates no significant enhancement (p > 0.1))
4.5 Discussion
Feature-based attention has been widely studied as the ability to enhance the features through-
out the visual cortex (Maunsell and Treue, 2006). Our study confirmed the previous feature-
based attentional fMRI response enhancement to the ignored stimuli (Saenz et al., 2002). Es-
pecially, our new findings revealed that the feature-based attentional enhancement still exist
63
in the condition when two features appeared as different objects, suggesting the feature-based
attentional enhancement takes place in the low feature levels, not due to subject grouping the
two features as single object. Therefore, the global enhancement effect of the feature-based
attention seems to employ a new early mechanism biasing on single feature.
Our study is consistent with the previous related studies on the feature-based attention. For
example psychophysically, studies showed that the subjects’ dual-task performance is better
when they attended same features compared to opposite features (Saenz et al., 2003). Fur-
thermore, our previous study showed that attention enhances both task-relevant features and
task-irrelevant features, but by different gain factors (Lu and Itti, 2005).Our study is also con-
sistent with some recent behavior evidence that attention can bias on feature level against the
object level in several psychophysical experimental paradigm. For example, Katzner et al.
showed the integration of color and motion features of random dots occurred when they ap-
peared across superimposed surfaces, which could not be accounted by object-based (Katzner
et al., 2006). In a recent visual search task, neurons were found to give enhanced response
whenever a preferred stimulus in their receptive field matched the target feature (Bichot et
al., 2005). Comparing these previous study, the new findings of our study is we extend this
feature bias attentional selection to the unattended stimuli, thus provide a more wide under-
standing of the feature-based attention.
In the neural levels, our study supported the recently developed Feature-Similarity Model. In
this model, attention was proposed to increase the gain for the neurons preferring the attended
64
feature and decrease the gain for the neurons with opposite preference (Maunsell and Treue,
2006). By showing the feature-based attention is indeed not object-based, our study cast a
clear boundary to this model that attention increased the gain for the neurons preferring the
attended feature throughout the visual cortex, not only the neurons which belong to the same
object with the attended features at the attended location.
Interestingly, our study showed that in the different-object condition, the feature-based at-
tentional enhancement was even much larger than in the same-object condition. One of the
possible reason may be because we only used the localizer stimuli displaying in the different-
objects condition to find the region of interest. Thus the region of interest we selected would
more bias on the dots stimuli in the different-object condition. Except for this factor, whether
the feature-based attention triggering more enhancement when the two features appearing as
different objects remains unknown and needs further study.
Overall by studying the recently observed feature-based attention both in the same-object
condition and in the different-object condition, we showed that the feature-based attentional
enhancement exists in both conditions. Therefore, our results indicate that the feature-based
attention, instead because subjects group the two features as a single object by the object-
based attention, is actually employing a completely new mechanism that its selection takes
place in the low feature levels.
65
Chapter 5
FMRI Study: Perceptual Level of Feature-based Attention using
Gabor Stimuli
5.1 Overview
In this chapter, we present some unpublished results which using Gabor stimuli to study if
the feature-based attention is also object-based. However, our preliminary data suggested the
feature-based attentional enhancement is hard to be extracted using the Gabor stimuli, con-
sistent with Saenz’s unpublished data in her thesis (M. Saenz’s thesis, UCSD, 2002). These
findings suggested the Gabor stimuli maybe are not the good stimuli to trigger feature-based
attentional enhancement in the fMRI study.
The experimental paradigm and stimuli are very similar with the study in Chapter 2 using
Gabor stimuli. The only main difference is that in this study we did single task instead of
dual task during the fMRI scanning, which is consistent with the previous chapter. The rea-
son we use single task is that we wanted to duplicate the previous feature-based attentional
66
enhancement by setting one side of the stimulus as attention task side, and another side as
ignored side. The question is same with the previous chapter: since object-based attention
can also enhance the attended features at the ignored position and it showed a so strong se-
lective ability, is it possible that the feature enhancement by the feature-based attention is
because subjects grouping the two features as same object so that it is actually a specific form
of object-based attention, or due to a completely different type of attentional selection mech-
anism by which attentional selection indeed process individual features as a unit?
Using fMRI study, we did the single-task experiment using Gabor stimuli in two conditions:
either the two features displayed as in same object or they appeared as different objects. The
objectness designation is also same with previous chapter: The two features appearing in the
same large box represents the same object, and the two features appearing in two different
small boxes represents different object condition. Based on the two possible hypotheses,
if attentional selection is processed on the object level, the feature enhancement modulated
by feature-based attention should become lower or disappear when the features displayed as
different objects. In contrast, the feature-based attentional enhancement may still exist in
the different-object conditions. However, we even didn’t find clear feature-based attentional
enhancement in the same-object condition in this study. There is only slightly enhancement
in the Broadman 18 and 19 areas at the ignored side, close to human MT areas when sub-
ject attended same features comparing to different features when they performed orientation
discrimination task at the attended side. This slight enhancement was not observed in the
different-object conditions. But since we didn’t run the localizer stimulus in this study, we
67
PV PV PV PV Figure 5.1: Experimental timing frame. Stimuli were two drifting Gabor patches presented bilater-
ally to the central fixation cross. Left side is the task side. Task is orientation discrimination (subjects
responded whether the stimulus was vertical then tilted, or tilted then vertical) on faster horizontal
stimulus (OHF). Right side is ignored side and always vertical and slower patch (OVS). Arrows indi-
cate drift direction and speed. The two small boxes make the two Gabor stimuli look like two separate
objects. Stimuli also have a certain background which is not shown in this figure.
could not make sure that the enhancement is really corresponding to the stimulus position,
hence is hard to identify that enhancement is the real feature-based attentional effect. When
subject performed drift speed discrimination task, we even were not able to observe any sig-
nificant feature-based enhancement in both single-object and different-object conditions. The
overall results suggested that Gabor stimuli may not be the good stimuli to trigger the feature-
based attention, which is consistent with the unpublished data in Saenz’s thesis (M. Saenz’
thesis, UCSD).
5.2 Method
Stimuli and tasks: Stimuli were generated by the C++ Neuromorphic Vision Toolkit devel-
oped in iLab. The stimuli are two drifting Gabor stimuli presented bilaterally to the central
68
fixation across. Drifting Gabor stimuli (spatial frequency 3.6 cpd, envelope FWHM 0:55

,
contrast 0.4) were horizontal or vertical (H or V orientation feature), drifting slowly (3.33 c/s;
S drift speed feature) or faster (5.00 c/s; F drift speed feature) and were displayed at 4

ec-
centricity from fixation. For each trial the time of repetition is 3 second and the timing of
each trial is shown in figure.
The stimuli are same to the previous study with two drifting Gabor stimuli presented bi-
laterally to fixation except the timing of frame sequences (Fig. 16). The tasks are also
orientation discrimination (O) or speed (D) discrimination, using a dual two-interval forced-
choice (2IFC) paradigm. Similarly, in the orientation discrimination tasks, stimuli could be
oriented vertically (condition V) or horizontally (H) vs. slightly tilted off that orientation,
while keeping the drift speed unchanged; subjects then reported whether the slightly tilted
patch had appeared first. In the drift speed discrimination task, stimuli drifted at the slower
(condition S) or at the faster (F) speed vs. slightly increased speed; subjects then reported
whether the patch with slightly increased speed had appeared first. Differently with previous
study, subjects only performed single task at one side, while ignoring the stimulus at the other
side (no task). The ignored side always keeps display as slower speed and vertical orienta-
tion. Two conditions were studied as single-object condition and different-object condition.
In different-object condition, the two Gabor patch stimuli were cased inside two small boxes
with certain background outside within which the two Gabor stimuli looked like two sepa-
rate or different ’objects’. In single-object condition, the Gabor stimuli are cast into a large
box with same background, so they looked like single object. The shadow effects were also
69
Figure 5.2: An illustration of different-object conditions. In this condition, the two Gabor stimuli
were casted onto two small boxes in a certain background ( here we didn’t show the actual textured
background). Similarly, in the same-object condition, the two Gabor stimuli were displayed in one
large box with the same background. In this study, we also have shadow effect cast around the box
which is not shown here.
used to enhance the visualization as an object. Echo-planar imaging (EPI) was done with a
new developed 4 Tesla Varian magnetic resonance imaging system at Brookhaven National
Laboratory. Stimuli were displayed in the goggle glasses worn by subjects inside the fMRI
machine. The scanning voxel size is 3x3x3 mm. The repetition time (TR) is 3 seconds. The
two conditions were displayed in alternative sequence in sessions.
70
5.3 Result
The fMRI scanning results were analyzed by SPM99 under Linux after realignment, smooth-
ness, and normalization. Preliminary analysis of the data suggested brain areas for the ig-
nored side reduced activity in part of Broadman 18 and 19 areas when stimulus displayed as
different separate objects than as single object in the ignored side, when subject performed
orientation discrimination task. But such property was not observed significantly in the drift
speed discrimination task.
5.3.1 Orientation discrimination task
In this session, the task is orientation discrimination task. When the two sides of the stimuli
shared the same feature, both the Gabor orientation and the drift speed are same. When
they displayed as different features, both the orientation and the drift speed are different.
In such condition, if the feature-based attentional effect exists, it should trigger the largest
enhancement when subjects attended same features comparing with the different features,
since the feature difference exists in two kinds of features (orientation and drift speed). The
following figures showed the result of three subjects. The results suggested there are feature-
based enhancement clusters in the Broadman area 18 and 19 when subject attended same
features comparing with different features.
5.3.2 Drift speed task
The figure shows the feature-based attentional effect when task is drift speed discrimination
while the features also differ both in orientation and drift speed, e.g. when the two sides of
71
Figure 5.3: Subject LI: fMRI BOLD activity with orientation discrimination task at left side and ig-
nored stimulus at right side of the display. Upper left showed brain activity enhancement when subject
attended same features than different features in single object condition. Upper right showed activity
enhancement in separate objects condition. Comparison showed activity enhancement reduced when
the two stimuli shown as two separate objects. The enhancement cluster was marked by red ’<’. The
lower part showed the detailed cluster position and the activity strength. The activity map was shown
under uncorrected p < 0.0001.
stimuli have different features, for the orientation features one side is horizontal and anther
side is vertical; for the drift speed features one side is slow drift and another side is fast drift.
In summary, the features property is same with previous figures. But the task is drift speed
discrimination instead of orientation discrimination.
Unlike the feature-based attentional effect in the orientation discrimination above, when sub-
jects did drift speed discrimination task, the feature-based attentional enhancement was not
72
Figure 5.4: Subject JL: fMRI BOLD activity with orientation discrimination task at left side and ig-
nored stimulus at right side of the display. Upper left showed brain activity enhancement when subject
attended same features than different features in single object condition. Upper right showed activity
enhancement in separate objects condition. Comparison showed activity enhancement reduced when
the two stimuli shown as two separate objects. The enhancement cluster was marked by red ’<’. The
lower part showed the detailed cluster position and the activity strength. The activity map was shown
under uncorrected p < 0.0001.
observed in all the three subjects in any visual areas, even in the Broadman 18 and 19 areas.
This findings in further suggest that the feature-based attentional enhancement above shown
in the orientation discrimination task is not so reliable, may not be in the MT areas. When we
ran the scanning in this study, we didn’t run the clear localizer stimuli to find the correspond-
ing visual cortical areas for our stimuli. Without the localizer stimuli, it’s hard to identify
73
Figure 5.5: Subject RE: fMRI BOLD activity with orientation discrimination task at left side and ig-
nored stimulus at right side of the display. Upper left showed brain activity enhancement when subject
attended same features than different features in single object condition. Upper right showed activity
enhancement in separate objects condition. Comparison showed activity enhancement reduced when
the two stimuli shown as two separate objects. The enhancement cluster was marked by red ’<’. The
lower part showed the detailed cluster position and the activity strength. The activity map was shown
under uncorrected p < 0.0001.
the enhancement effect we found in the orientation discrimination task above is really the
feature-based attentional enhancement or not.
5.4 Summary and Conclusion
As the supplement of the previous session, in this chapter we present the results using Gabor
stimuli to study whether the feature-based attention is also object based. However, the re-
sults indicated that the Gabor stimuli is not a good type of stimuli to trigger the feature-based
attentional effect in the fMRI study, although they work well in the dual task psychophysi-
cal study in Chapter 3. In summary, in the orientation discrimination task, all the subjects
displayed a significant cluster enhancement at the ignored stimulus side when they attended
74
6DPH2EMHFW6DPH)HDWXUHPLQXV'LIIHUHQW)HDWXUH 'LIIHUHQW2EMHFW6DPH)HDWXUHPLQXV'LIIHUHQW)HDWXUH 6XEMHFW/, Figure 5.6: Subject LI: fMRI BOLD activity with drift speed discrimination task at left side and
ignored stimulus at right side of the display. Figure shows the results for three subjects. When the
subjects attended same features comparing with different features at the attended side, we didn’t found
any obvious significant enhancement on the ignored side. The figure shows the brain fMRI activity in
the ignored side. Note at the visual cortex, no significant enhancement was found. The activity map
was shown under uncorrected p < 0.0001.
75
6DPH2EMHFW6DPH)HDWXUHPLQXV'LIIHUHQW)HDWXUH 'LIIHUHQW2EMHFW6DPH)HDWXUHPLQXV'LIIHUHQW)HDWXUH 6XEMHFW-/ Figure 5.7: Subject JL: fMRI BOLD activity with drift speed discrimination task at left side and
ignored stimulus at right side of the display. Figure shows the results for three subjects. When the
subjects attended same features comparing with different features at the attended side, we didn’t found
any obvious significant enhancement on the ignored side. The figure shows the brain fMRI activity in
the ignored side. Note at the visual cortex, no significant enhancement was found. The activity map
was shown under uncorrected p < 0.0001.
76
same features comparing to attending the different features in the part of Broadman 18 and
19 areas in the same object condition. But this effect disappeared when the two stimuli dis-
played as the different objects. However, since we didn’t run the localizer stimuli, it’s hard to
say if that enhanced clusters are corresponding to the real ignored features positions. On the
other hand, if the subject did drift speed task, in both the same object condition and different
object condition, we didn’t find any significant enhancement effect in the visual cortical ar-
eas. Since our original purpose to study if the feature-based attention is object based or not,
we must first be able to duplicate the original feature-based attentional effect in the previous
Saenz et al.’ study. However, combining our two studies above, it’s almost clear that it’s hard
to duplicate the previous feature-based attentional effect even in the same object condition,
suggesting the Gabor stimuli are not good type of stimuli in the fMRI study of feature-based
attention, which pushed us to switch to the dots stimuli in the Chapter 4.
77
6DPH2EMHFW6DPH)HDWXUHPLQXV'LIIHUHQW)HDWXUH 'LIIHUHQW2EMHFW6DPH)HDWXUHPLQXV'LIIHUHQW)HDWXUH 6XEMHFW5( Figure 5.8: Subject RE: fMRI BOLD activity with drift speed discrimination task at left side and
ignored stimulus at right side of the display. Figure shows the results for three subjects. When the
subjects attended same features comparing with different features at the attended side, we didn’t found
any obvious significant enhancement on the ignored side. The figure shows the brain fMRI activity in
the ignored side. Note at the visual cortex, no significant enhancement was found. The activity map
was shown under uncorrected p < 0.0001.
78
Chapter 6
FMRI Study: Influence of Task Relevant and Irrelevant Features
in Feature-based Attention
6.1 Overview
In Chapter 3, we observed that task relevant and irrelevant features both influence the per-
ceptual consequence of the feature-based attention, but with a different gain factor. The gain
factor of task irrelevant features in determining the feature-based attentional enhancement is
about one fifth of that of the task relevant features. Thus we expect that this task relevance
pattern may also appear in the fMRI study. In this chapter, we present our unpublished results
to study the influence pattern of feature based attention by the task relevant and irrelevant fea-
tures. Feature based attention is referred to the enhancement of the attended features when
subjects attended same features comparing to attending different features. According to our
study in Chapter 3, we assume the different gain pattern also exist in the fMRI study of the
feature-based attention. Specifically, we assume:
79
1. When the attended side and ignored side share the totally same features, the enhancement
caused by the feature-based attention should be the largest.
2. When the attended side and ignored side share the same task relevant features (e.g. ver-
tical or horizontal in orientation discrimination task; slow or fast in the drift speed task), but
not share the same task irrelevant features (e.g. fast or slow in orientation discrimination
task; vertical or horizontal in drift speed discrimination task), the enhancement caused by the
feature-based attention should be the second largest.
3. When the attended side and ignored side share the same task irrelevant features but not
share the same task relevant features, the enhancement caused by the feature-based attention
should be the third largest.
4. When the attended side and ignored side share neither the same task irrelevant features nor
the same task irrelevant features, there would be no enhancement caused by the feature-based
attention.
The stimuli are same to the previous chapter with two drifting Gabor stimuli presented bilat-
erally to fixation except the timing of frame sequences. The tasks are also orientation dis-
crimination (O) or speed (D) discrimination, using a dual two-interval forced-choice (2IFC)
paradigm. Similarly, in the orientation discrimination tasks, stimuli could be oriented verti-
cally (condition V) or horizontally (H) vs. slightly tilted off that orientation, while keeping
80
the drift speed unchanged; subjects then reported whether the slightly tilted patch had ap-
peared first. In the drift speed discrimination task, stimuli drifted at the slower (condition
S) or at the faster (F) speed vs. slightly increased speed; subjects then reported whether the
patch with slightly increased speed had appeared first. Subjects only do single task at one
side, while ignoring the stimulus at the other side (no task). The ignored side always keeps
same, displaying as slower speed and vertical orientation. The attended side, specifically, is
the combination of the following different conditions:
1. Orientation discrimination task or drift speed discrimination task.
2. Vertical or horizontal in orientation features.
3. Fast or slow drift in drift speed features.
We want to examine how different tasks and stimuli condition in the task side influence the
fMRI activity to the ignored side, which always keeps same as vertical and slow drift Gabor
stimulus.
However, our results didn’t catch our original assumption shown above. After analysis of four
subjects for each of the conditions above, the overall results indicated that the enhancement of
the fMRI BOLD response of ignored side when subject attended same features (whether task
relevant feature, task irrelevant feature or both) comparing to different features is not so sta-
tistically significant. Furthermore, since the overall results didn’t show significant enhance-
ment by the feature-based attention, we also can not obviously differentiate the influence of
different task relevant feature properties. Thus the results suggested that the feature-based
81
6XEMHFW/, 6XEMHFW-/ 6XEMHFW5( 6XEMHFW81 Figure 6.1: The results of four subjects in orientation discrimination task. The figure shows the
fMRI BOLD response at the ignored side when subjects attended same feature (vertical and slow drift)
comparing to different feature (horizontal and fast drift) at the attended side, which means both the
task relevant and irrelevant features are different. Figure shows there is not statistically enhancement
at the visual areas (uncorrected p < 0.0001).
attentional fMRI BOLD response enhancement is hard to extract using Gabor stimuli, con-
firming the study in the previous chapter and consistent with Saenz’s unpublished data in her
thesis (M. Saenz’s Ph.D. thesis, UCSD).
6.2 Results
6.2.1 Both the task relevant and irrelevant features are different
In this session, the task is orientation discrimination task or drift speed discrimination task.
When the two sides of the stimuli shared the same feature, both the Gabor orientation and the
drift speed are same (vertical and slow). When they displayed as different features, both the
82
6XEMHFW-/ 6XEMHFW/, 6XEMHFW81 6XEMHFW5( Figure 6.2: The results of four subjects in drift speed discrimination task. The figure shows the fMRI
BOLD response at the ignored side when subjects attended same feature (vertical and slow drift)
comparing to different feature (horizontal and fast drift) at the attended side, which means both the
task relevant and irrelevant features are different. Figure shows there is not statistically enhancement
at the visual areas (uncorrected p < 0.0001).
orientation and the drift speed are different (the attended side is horizontal and fast speed).
In such condition, if the feature-based attentional effect exists, it should trigger the largest
enhancement when subjects attended same features comparing with the different features,
since the feature difference exists in two kinds of features (orientation and drift speed). The
figures showed the result of three subjects. The results showed that there were not significant
enhancement (uncorrected p < 0.001) when subjects attended same features comparing to
when they attended different features, both in the orientation discrimination task and drift
speed discrimination task.
83
6XEMHFW5( 6XEMHFW/, 6XEMHFW-/ 6XEMHFW81 Figure 6.3: The results of four subjects in orientation discrimination task. The figure shows the
fMRI BOLD response at the ignored side when subjects attended same feature (vertical and slow drift)
comparing to different task relevant feature (horizontal) but same task irrelevant feature (slow) at the
attended side. Figure shows there is not statistically enhancement at the visual areas (uncorrected p <
0.0001).
6.2.2 Task relevant features are different and irrelevant features are same
In this session, the task is also orientation discrimination task or drift speed discrimination
task. When the two sides of the stimuli shared the same feature, both the Gabor orientation
and the drift speed are same (vertical and slow). When they displayed as different features,
only the task relevant features are different while the task irrelevant features keeping same,
e.g. in the orientation discrimination task the attended side is horizontal and slow speed
drifting Gabor stimulus and in the drift speed discrimination task the attended side is fast
speed and vertical Gabor stimulus. Certainly, the ignored side is always vertical and slow
drift speed. In such condition, if the feature-based attentional effect exists, it should trigger
84
6XEMHFW5( 6XEMHFW81 6XEMHFW-/ 6XEMHFW/, Figure 6.4: The results of four subjects in drift speed discrimination task. The figure shows the fMRI
BOLD response at the ignored side when subjects attended same feature (vertical and slow drift)
comparing to different task relevant feature (fast drift) but same task irrelevant feature (vertical) at the
attended side. Figure shows there is not statistically enhancement at the visual areas (uncorrected p <
0.0001).
the second largest enhancement (comparing the condition in 6.2.1) when subjects attended
same features comparing with the different features, since the feature difference exists in
the task relevant features, which should have a larger gain factor to determine the feature-
based attention than the task irrelevant features according to our study in Chapter 3. The
following two figures showed the result of three subjects. The results showed that there were
not significant enhancement (uncorrected p < 0.001) when subjects attended same features
comparing to when they attended different features, both in the orientation discrimination
task and drift speed discrimination task.
85
6XEMHFW/, 6XEMHFW5( 6XEMHFW81 6XEMHFW-/ Figure 6.5: The results of four subjects in orientation discrimination task. The figure shows the
fMRI BOLD response at the ignored side when subjects attended same feature (vertical and slow drift)
comparing to different task irrelevant feature (fast drift) but same task relevant feature (vertical) at the
attended side. Figure shows there is not statistically enhancement at the visual areas (uncorrected p <
0.0001).
6.2.3 Task relevant features are same and irrelevant features are different
In this session, the task is orientation discrimination task or drift speed discrimination task
too. When the two sides of the stimuli shared the same feature, both the Gabor orientation
and the drift speed are same (vertical and slow). When they displayed as different features,
only the task irrelevant features are different while the task relevant features keeping same,
e.g. in the orientation discrimination task the attended side is vertical and fast speed drifting
Gabor stimulus and in the drift speed discrimination task the attended side is slow speed and
horizontal Gabor stimulus. The ignored side is always vertical and slow drift speed. In such
86
6XEMHFW-/ 6XEMHFW/, 6XEMHFW81 6XEMHFW5( Figure 6.6: The results of four subjects in drift speed discrimination task. The figure shows the fMRI
BOLD response at the ignored side when subjects attended same feature (vertical and slow drift)
comparing to different task irrelevant feature (horizontal) but same task relevant feature (slow drift) at
the attended side. Figure shows there is not statistically enhancement at the visual areas (uncorrected
p < 0.0001)
condition, if the feature-based attentional effect exists, it should trigger the smallest enhance-
ment (comparing to the condition in 6.2.1 and 6.2.2) when subjects attended same features
comparing with the different features, since the feature difference exists in the task irrelevant
features, which should have a smaller gain factor to determine the feature-based attention
than the task relevant features according to our study in Chapter 3. The following two fig-
ures showed the result of three subjects. The results showed that there were not significant
enhancement (uncorrected p < 0.001) when subjects attended same features comparing to
when they attended different features, both in the orientation discrimination task and drift
speed discrimination task.
87
6.3 Summary and conclusion
In summary, although we originally proposed that task relevant features and task irrelevant
features should play different roles in determine the enhancement caused by the feature-based
attention in the fMRI study, our results didn’t meet our original assumption. The overall
results indicated that the enhancement of the fMRI BOLD response of ignored side when
subject attended same features (whether task relevant feature, task irrelevant feature or both)
comparing to different features is not so statistically significant. Since the overall results
didn’t show significant enhancement by the feature-based attention, we also can not obvi-
ously differentiate the influence of different task relevant feature properties. Thus the results
suggested that the feature-based attentional fMRI BOLD response enhancement is hard to
extract using Gabor stimuli, confirming the study in the previous chapter and consistent with
Saenz’s unpublished data in her thesis (M. Saenz’s Ph.D. thesis, UCSD).
88
Chapter 7
Summary and Future work
7.1 Summary
Attention modulates visual processing at three different dimensions: A spatial dimension
which enhances the representation of stimuli within the focus of attention, a feature dimen-
sion thought to enhance attended visual features throughout the visual field and an object
dimension by which attention enhance the whole object as a unit.
In our study, we investigated the detailed perceptual mechanism of feature-based attention
both qualitatively and quantitatively. In particular, we hope to understand the perceptual
consequence of feature-based attention with task and stimuli variation and whether feature-
based attention is also object based. Although the new global effects of visual areas due to
feature-based attention have been discovered recently in both fMRI study and psychophysical
methods, these effects currently lack a complete, unified computational understanding when
89
varying their task and feature parameters. In another aspect, there are highly competing theo-
ries as to whether attention can select basic features such as colors and shapes (feature-based)
or particular visual objects composed of constellations of visual features (object-based). Our
study directly bear on these theories to understand a complete perceptual consequence of the
feature-based attention, and whether attention is feature-based, or object-based, or both.
By summmarizing our studies, we conclude that the feature-based attentional effect, which
was represented as the enhancement of attended features all throughout the visual cortex, is
indeed ’feature’ based. The feature-based attention selects the single feature as the entity, not
due to the traditional thought of the object-based attention, by which the feature enhance-
ment is because the ignored features belong to the same object with the attended features at
the attended location. Furthermore, the detailed perceptual study of the feature-based atten-
tional consequence suggested that both the task relevant features and task irrelevant features
contributed to the enhancement effect but by a different gain factor. The overall perceptual
consequence could be unified into a simple functional computational model which predicts a
winner-take-all rule in the feature space.
Specifically, we make three conclusion on the perceptual and computational mechanism of
the feature-based attention:
90
1. The perceptual level of the feature-based attention takes place on the ’feature’ level, not
on the ’object’ level.
2. The perceptual consequence of the feature-based attention is determined by both the task
relevant features and the task irrelevant, but by a different gain factors. The overall con-
sequence could be explained by a simple functional computational model which predicts a
winner-take-all rule in the feature space.
3. The Gabor stimuli are not the suitable stimuli in triggering the feature-based attentional
enhancement in the fMRI study.
7.2 Future work
The roles of task relevant and irrelevant features in determining the fMRI BOLD re-
sponse enhancement of the feature-based attention: As we observed in the Chapter 5 and
Chapter 6, the Gabor stimuli is not suitable to trigger the feature-based attentional enhance-
ment in the fMRI study. This raised the question: does the feature relevance pattern we
observed in the psychophysical study still exists in the fMRI study? Since we successfully
duplicated the feature-based attention using the color dots stimuli, the next step may be us-
ing such stimuli to study the feature relevance pattern by fMRI. For example, if we consider
the red-green dots stimuli and performed luminance discrimination task as in Chapter 4, we
can set the feature difference both in the color feature and luminance feature. Then we can
91
investigate how the scale and pattern of feature-based attentional fMRI BOLD response en-
hancement will change corresponding to the different task relevant and irrelevant features.
The influence of the objectness of the stimuli on the feature-based attention: In our study
in Chapter 4, we showed the feature-based attention both exist in single-object and different-
object conditions, suggesting the feature-based attentional selection takes place in the lower
feature levels, not object-based. To enhancement the visualization of objectness in our stim-
uli, we cast a shadow effect at the right-bottom side of the boxes. There are many different
geometric methods to build the 3D effect of the object, in which our study only provided a
simpler one. For example, the other possible method may include casting the shadow at the
different location of the stimulus boxes, like one shadow on the upper-left of the left stimulus
and another shadow on the bottom-right corner of the right stimulus, which makes the two
stimuli appear at the different depth in the space and enhances the visualization of the object-
ness in further. More studies should be done with this aspect to ensure the non-object based
effect of the feature-based attention.
The study of feature-based attention in nature scene stimuli: After a series of studies
using electrophysiology, psychophysics and fMRI, the perceptual and computational mech-
anism has been observed in many different aspects. However, so far all the studies of the
feature-based attention have been using the simple dots or Gabor stimuli. This raised an in-
teresting question: is there the feature-based attentional effect also in the real nature scene
92
stimuli, e.g. in our real visual world? In our previous studies, we defined the feature rele-
vance in the very simple way: the vertical or horizontal features are relevant to the orientation
discrimination task while the slow or fast drift features are relevant to the drift speed discrim-
ination task. But using the real nature scene stimuli, since there were many complicated
feature channels combining together, many more investigation should be carried out to de-
sign and run the psychophysical and fMRI studies.
93
References
Alais D, Blake R. Grouping visual features during binocular rivalry. Vision Res. 1999 Oct;
39(26):4341-53.
Alais D., Blake R. Neural strength of visual attention gauged by motion adaptation. Nat Neu-
rosci. 1999 Nov; 2(11):1015-8.
Ball K., Sekuler R. Direction-specific improvement in motion discrimination. Vision Res.
1987; 27(6):953-65
Beauchamp, M. S., Cox, R. W., and DeYoe, E. A. Graded effects of spatial and featural atten-
tion on human area mt and associated motion processing areas. J Neurophysiol, 78(1):516–
520, Jul 1997.
Bichot N. P., Rossi, A. F. and Desimone, R. Parallel and serial neural mechanisms for visual
search in macaque area V4. Science, 308(5721):529-34, Apr 2005.
Blaser, E., Pylyshyn, Z. W., and Holcombe, A. O. Tracking an object through feature space.
Nature, 408(6809):196-199, Nov 2000.
Brefczynski, J. A. and DeYoe, E. A. A physiological correlate of the ’spotlight’ of visual
attention. Nat Neurosci, 2(4):370–374, Apr 1999.
Brainard, D. H. The Psychophysics Toolbox, Spatial Vision, 10(4):433-6, 1997
Chawla, D., Rees, G. and Friston, K. J.. The physiological basis of attentional modulation in
extrastriate visual areas. Nat Neurosci, 2(7):671–676, Jul 1999.
Corbetta, M., Miezin, F. M., Dobmeyer S., Shulman, G. L. and Petersen, S. E.. Selective
and divided attention during visual discriminations of shape, color, and speed: functional
anatomy by positron emission tomography. J Neurosci, 11(8):2383–2402, Aug 1991.
Desimone, R. and Duncan, J. Neural mechanisms of selective visual attention. Annu Rev
Neurosci, 18:193–222, 1995.
94
Duncan, J. Selective attention and the organization of visual information. J Exp Psychol Gen,
113(4):501-517, Dec 1984.
Duncan, J. Martens S., and Ward, R. Restricted attentional capacity within but not between
sensory modalities. Nature, 387(6635):808–810, Jun 1997.
Duncan, J. and Nimmo-Smith, I. Objects and attributes in divided attention: surface and
boundary systems. Percept Psychophys, 58(7):1076–1084, Oct 1996.
Eriksen C. W., St James J. D. Visual attention within and around the field of focal attention:
a zoom lens model. Percept Psychophys. 1986 Oct; 40(4):225-40.
Hillyard S. A., Anllo-Vento L. Event-related brain potentials in the study of visual selective
attention. Proc Natl Acad Sci U S A. 1998 Feb; 95(3):781-7.
Huk, A. C. and Heeger, D. J.. Task-related modulation of visual cortex. J Neurophysiol,
83(6):3525–3536, Jun 2000.
Itti, L. and Koch C.. Computational modelling of visual attention. Nat Rev Neurosci, 2(3):194–
203, Mar 2001.
Itti, L., Koch. C. and Braun, J. Revisiting spatial vision: towards a unifiying model. Journal
of the Optical Society of America, 17(11):1899-917, Nov. 2000.
Itti, L. Ph.D Thesis, 2000. California Institute of Technology
Kastner S, Pinsk M. A., De Weerd P, Desimone R, Ungerleider LG. Increased activity in
human visual cortex during directed attention in the absence of visual stimulation. Neuron
1999 Apr; 22(4):751-61.
Katzner, S., Busse, L. and Treue, S. Feature-based attentional integration of color and visual
motion. Journal of Vision, 6(3):269-84, Mar 2006.
Lee D. K., Itti L., Koch C., and J. Braun. Attention activates winner-take-all competition
among visual filters. Nat Neurosci, 2(4):375–381, Apr 1999.
Lu. J. and Itti, L. Perceptual consequence of the feature-based attention. Journal of Vision,
5(7):622-31, Aug. 2005.
Luck, S. J. Chelazzi, L. Hillyard, S. A. and Desimone R. Neural mechanisms of spatial selec-
tive attention in areas V1, V2, and V4 of macaque visual cortex. J Neurophysiol, 77(1):24–42,
Jan 1997.
95
McAdams C. J. and Maunsell J. H.. Attention to both space and feature modulates neuronal
responses in macaque area v4. J Neurophysiol, 83(3):1751–1755, Mar 2000.
Melcher, D., Papathomas, T. V ., Vidnyanszky, Z. Implicit attentional selection of bound vi-
sual features. Neuron, 46(5):723-9, Jun 2005.
Moran J, Desimone R. Selective attention gates visual processing in the extrastriate cortex.
Science. 1985 Aug 23;229(4715):782-4.
Morrone, M. C. Denti V ., and Spinelli D.. Color and luminance contrasts attract independent
attention. Curr Biol, 12(13):1134–1137, Jul 2002.
Morrone, M. C. Denti, V . Spinelli D. Different attentional resources modulate the gain mech-
anisms for color and luminance contrast. Vision Res, 44(12):1389-1401, Jun 2004.
Motter B. C. Focal attention produces spatially selective processing in visual cortical areas
V1, V2, and V4 in the presence of competing stimuli. J Neurophysiol. 1993 Sep;70(3):909-
19.
Motter, B. C. Neural correlates of attentive selection for color or luminance in extrastriate
area V4. J Neurosci., 14(4):2178–2189, Apr 1994.
O’Craven K. M., Downing P. E. and Kanwisher N. fMRI evidence of objects as the units of
attentional selection. Nautre, 401(6753):584-7, May 1999.
Posner, M. I., Snyder, C. R., and Davidson, B. J. Attention and the detection of signals. J Exp
Psychol, 109(2):160–174, Jun 1980.
Reynolds, J. H., Chelazzi, L., and Desimone, R. Competitive mechanisms subserve attention
in macaque areas V2 and V4. J Neurosci., 19(5):1736–1753, Mar 1999.
Rossi, A. F. and Paradiso, M. A. Feature-specific effects of selective visual attention. Vision
Res, 35(5):621–634, Mar 1995.
Saenz, M., Buracas, G. T., and Boynton, G. M.. Global effects of feature-based attention in
human visual cortex. Nat Neurosci, 5(7):631–632, Jul 2002.
Saenz, M., Buracas, G. T., and Boynton, G. M.. Global feature-based attention for motion
and color. Vision Res, 43(6):629–637, Mar 2003.
Saenz. M. Ph.D. thesis, University of California, San Diego, 2002.
96
Saffell, T. and Matthews, N.. Task-specific perceptual learning on speed and direction dis-
crimination. Vision Res, 43(12):1365–1374, Jun 2003.
Shih, S. I. and Sperling, G. Is there feature-based attentional selection in visual search? J Exp
Psychol Hum Percept Perform, 22(3):758–779, Jun 1996.
Shiu, L. P. and Pashler, H. Improvement in line orientation discrimination is retinally local
but dependent on cognitive set. Percept Psychophys, 52(5):582–588, Nov 1992.
Sohn, W., Papathomas T. V ., Blaser E. and Vidnyanszky Z. Object-based cross-feature atten-
tional modulation from color to motion. Vision Res., 44(12):1437-43, Jun 2004.
Sperling, G. and Melchner, M. J. The attention operating characteristic: examples from vi-
sual search. Science, 202(4365):315–318, Oct 1978.
Treue, S. Neural correlates of attention in primate visual cortex. Trends Neurosci, 24(5):295–
300, May 2001.
Treue, S. and Martinez Trujillo, J. C.. Feature-based attention influences motion processing
gain in macaque visual cortex. Nature, 399(6736):575–579, Jun 1999.
Treue, S. and Maunsell, J. H.. Attentional modulation of visual motion processing in cortical
areas. Nature, 382(6591):539–541, Aug 1996.
Valdes-Sosa, M., Cobo A. and Pinilla T. Transparent motion and object-based attention. Cog-
nition, 66(2):B13-B23, May 1998.
Watanabe, T., Harner, A. M., Miyauchi, S., Sasaki, Y ., Nielsen, M., Palomo, D., and Mukai,
I. Task-dependent influences of attention on the activation of human primary visual cortex.
Proc Natl Acad Sci U S A, 95(19):11489–11492, Sep 1998.
Weichselgartner, E. and Sperling, G.. Dynamics of automatic and controlled visual attention.
Science, 238(4828):778–780, Nov 1987.
97

Abstract (if available)

Abstract Visual attention modulates visual processing along at least three dimensions: A spatial dimension which enhances the representation of stimuli within the focus of attention, a feature dimension thought to enhance attended visual features throughout the visual field and an object dimension by which attention enhance the whole object as a unit. In this thesis, we focus on the feature dimension by studying the feature-based attention.We investigated the overall perceptual consequences of feature-based attention, by using dual-task human psychophysics and two distant drifting Gabor stimuli to systematically explore 64 combinations of visual features (orientation and drift speed) and tasks (discriminating orientation or drift speed). The resulting single, consistent dataset suggests a functional model, which predicts a maximum-rule by which only the dominant product of feature enhancement and feature benefit by feature relevance may benefit perception. We also used fMRI to investigate whether feature-based attention is a specific form of object-based attention or a new type of lower feature based attentional selection. We studied the feature-based attentional enhancement in two conditions: either the two stimuli appeared to belong to same object or as two different objects. Results showed both in same-object condition and in different-object condition the four subjects consistently had significant enhancement of the ignored stimulus in early visual areas. Hence it indicated feature-based attentional enhancement exists even when two stimuli belong to two different objects, suggesting it is a new type of attentional selection which takes place in the low feature levels, not dependent on the objectness of the two features. We also present two unpublished studies showing that Gabor stimuli are not the suitable stimuli to trigger the feature-based attentional enhancement using fMRI.

Linked assets

University of Southern California Dissertations and Theses

Conceptually similar

PDF

Computational modeling and utilization of attention, surprise and attention gating

PDF

Computational modeling and utilization of attention, surprise and attention gating [slides]

PDF

Integrating top-down and bottom-up visual attention

PDF

Temporal dynamics of attention: attention gating in rapid serial visual presentation

PDF

Explicit encoding of spatial relations in the human visual system: evidence from functional neuroimaging

PDF

Functional models of fMRI BOLD signal in the visual cortex

PDF

Part based object detection, segmentation, and tracking by boosting simple shape feature based weak classifiers

PDF

Advanced techniques for object classification: methodologies and performance evaluation

PDF

Toward situation awareness: activity and object recognition

PDF

Vision-based studies for structural health monitoring and condition assesment

PDF

Towards health-conscious spaces: building for human well-being and performance

Asset Metadata

Creator Lu, Jianwei (author)

Core Title Perceptual and computational mechanisms of feature-based attention

School College of Letters, Arts and Sciences

Degree Doctor of Philosophy

Degree Program Neuroscience

Publication Date 11/21/2006

Defense Date 10/12/2006

Publisher University of Southern California (original), University of Southern California. Libraries (digital)

Tag attention,feature-based attention,fMRI,OAI-PMH Harvest,object-based attention,psychophysics

Language English

Advisor Itti, Laurent (committee chair), Biederman, Irving (committee member), Qin, Peter Z. (committee member)

Creator Email jianweil@usc.edu

Permanent Link (DOI) https://doi.org/10.25549/usctheses-m186

Unique identifier UC160862

Identifier etd-Lu-20061121 (filename),usctheses-m40 (legacy collection record id),usctheses-c127-36959 (legacy record id),usctheses-m186 (legacy record id)

Legacy Identifier etd-Lu-20061121.pdf

Dmrecord 36959

Document Type Thesis

Rights Lu, Jianwei

Type texts

Source University of Southern California (contributing entity), University of Southern California Dissertations and Theses (collection)

Repository Name Libraries, University of Southern California

Repository Location Los Angeles, California

Repository Email cisadmin@lib.usc.edu