University of Southern California Dissertations and Theses

A Monte Carlo Evaluation Of A Computer-Interactive Extended Transitivity Dominance Scaling Model

(USC Thesis Other)

A Monte Carlo Evaluation Of A Computer-Interactive Extended Transitivity Dominance Scaling Model

PDF

Download a page range

Download transcript

Copy asset link

Request this asset

Transcript (if available)

Content INFORMATION TO USERS
This material was produced from a microfilm copy of the original document. While
the most advanced technological means to photograph and reproduce this document
have been used, the quality is heavily dependent upon the quality of the original
submitted.
The following explanation of techniques is provided to help you understand
markings or patterns which may appear on this reproduction.
1. The sign or "target" for pages apparently lacking from the document
photographed is "Missing Page(s)". If it was possible to obtain the missing
page(s) or section, they are spliced into the film along with adjacent pages.
This may have necessitated cutting thru an image and duplicating adjacent
pages to insure you complete continuity.
2. When an image on the film is obliterated with a large round black mark, it
is an indication that the photographer suspected that the copy may have
moved during exposure and thus cause a blurred image. You will find a
good image of the page in the adjacent frame.
3. When a map, drawing or chart, etc., was part of the material being
photographed the photographer followed a definite method in
"sectioning" the material. It is customary to begin photoing at the upper
left hand corner of a large sheet and to continue photoing from left to
right in equal sections with a small overlap. If necessary, sectioning is
continued again — beginning below the first row and continuing on until
complete.
4. The majority of users indicate that the textual content is of greatest value,
however, a somewhat higher quality reproduction could be made from
"photographs" if essential to the understanding of the dissertation. Silver
prints of "photographs" may be ordered at additional charge by writing
the Order Department, giving the catalog number, title, author and
specific pages you wish reproduced.
5. PLEASE NOTE: Some pages may have indistinct print. Filmed as
received.
Xerox University Microfilms
300 North Zeeb Road
Ann Arbor, M ichigan 48106
75-6417
GREEN, Rex Stanley, 1941-
A M O N T E C A R LO EVALUATION O F A COMPUTER-INTERACTIVE
EXTENDED TRANSITIVITY D O M INA N CE SCALING MODEL.
University o f Southern C a lifo rn ia , Ph.D., 1974
Psychology, general
Xerox University Microfilms f Ann Arbor, Michigan 48106
THIS DISSERTATION HAS BEEN MICROFILMED EXACTLY AS RECEIVED.
A M ONTE CARLO E V A L U A T IO N O F A
C O M P U T E R -IN T E R A C T IV E E X T E N D E D
TRANSITIVITY DOM INANCE SC ALIN G M O D EL
by
R ex S ta n ley G reen
A D is s e r ta tio n P r e s e n te d to the
FA C U L T Y OF THE G RAD UA TE SCHOOL
UNIVERSITY OF SO UTH ERN C A LIFO R N IA
In P a r tia l F u lfillm e n t of th e
R eq u ir e m e n ts fo r th e D e g r e e
DOCTOR OF PH ILO SO PH Y
(P sy c h o lo g y )
A u gu st 1974
UNIVERSITY O F SO U T H E R N C A LIFO RN IA
TH E G R A D U A T E SC H O O L
U N IV E R S IT Y PARK
LOS A N G E L E S . C A L IF O R N IA 9 0 0 0 7
This dissertation, written by
Rex Stanley Green
under the direction of AA?.... Dissertation C om
mittee, and approved by all its members, has
been presented to and accepted by The Graduate
School, in partial fulfillment of requirements of
the degree of
D O C T O R O F P H I L O S O P H Y
Dean
D a te August..22^.. 1974
DISSERTATION COMMITTEE
TABLE OF CONTENTS
page
LIST O F T A B L E S ................................................................................................ iv
LIST O F F IG U R E S ............................................................................................. v
C H A P T E R I. BA CK G R O U ND................................................................... 1
R e p r e s e n tin g O rd ers of
S t im u li.................................................................................. 2
C o m p le te O rd ers fr o m
In c o m p le te D a ta .............................................................. 3
S u m m a r y ........................................................................... 8
C H A P T E R H. H Y P O T H E S E S ................................................................... 9
T h e S im u la tio n M o d e l ................................................ 9
S itu a tio n V a r ia b le s ....................................................... 10
A lg o r ith m V a r ia b le s ................................................... 12
P u r p o s e ............................................................................... 13
C H A P T E R III. M E T H O D ............................................................................. 15
P r o c e d u r e ........................................................................... 15
D e s ig n s .............................................................. 18
C H A P T E R IV. R E S U L T S ............................................................................. 26
S e le c tio n o f P e r fo r m a n c e
I n d ic e s .......................................................................... 28
T h e P e r fo r m a n c e o f IR IS ......................................... 31
C o m p a r iso n of C o m p lete
J u d g m en ts to In ter a c tiv e
A p p r o a c h ........................................................................... 37
C o m p a r iso n of In ter a c tiv e
to N o n - I n t e r a c t iv e ....................................................... 39
C H A P T E R V. D IS C U S S IO N ...................................................................... 46
C H A P T E R VI. CO NC LU SIO N S................................................................... 51
R E F E R E N C E S ....................................................................................................... 52
ii
page
A P P E N D IX .............................................................................................................. 53
A c r o n y m s ........................................................................ 54
S u m m ary of R e su lts —
T ab le A ............................................................................... 55
iii
LIST OF TABLES
T able
1 C o m p a r iso n o f Stopping R u le s ...............
2 L o ca tin g P e a k s o f Tgt fo r S e le c te d
R u n s ....................................................................
3 I n te r c o r r e la tio n s of P e r fo r m a n c e
In d ices .................................................................
4 P e r fo r m a n c e of the In ter a c tiv e
A p p r o a c h ..........................................................
5 C om p arin g the O c c u r r e n c e of P e a k s
o f G O F , w ith th o se of t ............................
3 s t
6 E ffe c ts o f F u r th e r P o w e r in g .................
7 S ta r t-U p s w ith P r io r In form ation . .
8 C om p arin g the Two A p p ro a ch es:
In te r a c tiv e v s . N o n -In te r a c tiv e . . . .
9 B e s t P o w e r s o f In te r a c tiv e v s.
N o n - I n t e r a c t iv e ............................................
LIST OF FIGURES
Figure page
1 T w o-W ay In te r a c tio n s of S tim u lu s
S e t S iz e w ith F o c u s Size o n Tau
and J E .......................................................................................................... 33
2 T w o-W ay In te r a c tio n s of L e v e l
o f E r r o r w ith F o c u s Size on GOF
and J E .......................................................................................................... 34
3 In te r a c tiv e R e s u lts for e a c h F ocus
S iz e C om pared to a ll Ju d gm en ts by
L e v e l of E r r o r ...................................................................................... 38
4 T w o-W ay In te r a c tio n of L e v e l of
P o w er in g w ith M ethod on T a u .................................................... 42
5 T w o-W ay In te r a c tio n of M ethod
w ith L e v e l of Ju d gm en ts o n T a u ................................................... 43
v
C H A PT E R I - BACKGROUND
R e s e a r c h e r s in the b e h a v io r a l s c ie n c e s o ften n eed in fo r m a tio n
about the o r d e r of so m e s e t of s tim u li. P r e fe r e n c e s am on g c o n s u
m e r p ro d u cts, jo b s, p eo p le w ith w hom w e m ig h t a s s o c ia t e , for e x
a m p le , a r e stu d ied in an e ffo r t to d is c e r n p a tte r n s of e v a lu a tiv e
b eh a v io r. P e o p le m a y be o r d e r e d a s to how happy, frie n d ly o r in
te llig e n t th ey a r e . O rd er in g s o f p h y s ic a l stim u li su ch a s lig h ts ,
to n e s, w e ig h ts, e tc . p ro v id e the b a s is fo r r e la tin g p s c h o lo g ic a l p r o
p e r tie s to p r o p e r tie s o f p h y s ic a l o b je c ts . O rd erin g s o f p eop le or
o r g a n iza tio n s a id in m akin g d e c is io n s r e g a rd in g e m p lo y e e s e le c tio n
and a d v a n cem en t, ed u c a tio n a l a d v a n c e m e n t, to u rn a m en t p a ir in g s ,
and so forth .
When an in v e stig a to r r e q u ir e s o r d e r in g s fo r in c r e a s in g ly la r
g e r s e ts of s tim u li, th e num b er o f p a ir w ise c o m p a r is o n s each s u b
j e c t m u st m ak e so o n e x c e e d s the s u b je c t's c a p a c ity or tim e lim it a
tio n s on the stu d y . The p r e s e n t stu d y is c o n c e r n e d w ith evalu atin g
a n ap p roach w h ich p r e s e r v e s the e ff e c tiv e n e s s o f p a ir w ise c o m p a r i
so n p r o c e d u r e s fo r d e te r m in in g o r d in a l s c a le s , w h ile s u b sta n tia lly
red u cin g the ju d gm en t ta sk fa cin g ea ch su b je c t. C liff (in p r e s s ) has
p ro p o sed a c o m p u te r -in te r a c tiv e a p p ro a ch to ob tain in g o rd in a l
s c a le s . S u b jects su p p ly ju d gm en ts to c o m p u te r -lin k e d r e m o te t e r
m in a ls . E ach su b je ct in te r a c ts w ith th e co m p u ter to p ro v id e th o se
ju d gm en ts the c o m p u ter p r o g r a m d e te r m in e s a r e n eed ed to s p e c ify
1
2
an o r d e r in g of the s tim u li. A s la r g e r s e t s of s tim u li a r e s c a le d ,
the p e r c e n ta g e of ju d gm en ts req u ire d d e c lin e s . The a im o f th is
stu d y w as to e v a lu a te a p a rticu la r in te r a c tiv e o r d e r in g a lg o r ith m by
sim u la tin g the ju d gm en ts of s u b je c ts . V a r ia b le s w e r e h y p o th e siz e d
to r e p r e s e n t co n d itio n s w h ich ty p ic a lly ch an ge a c r o s s d a ta -g a th e r in g
situ a tio n s. T h ese v a r ia b le s w e r e m a n ip u la ted so that the e f f e c t iv e
n e s s o f in te r a c tiv e o r d e r in g cou ld be co m p a red to a n o n -in te r a c tiv e
ap p roach for in c o m p le te s e t s o f data a s w e ll a s fo r c o m p le te s e ts of
ju d g m en ts.
R e p r e se n tin g O rd er s of S tim u li
O rd er s o f s tim u li a r e b a se d on the c o lle c tio n of m an y b in ary
d om in an ce r e la tio n s , iP j, in w h ich the r e la tio n £ in d ic a te s s tim u
lu s i is p r e fe r r e d to o r p r e c e d e s j. C liff (in p r e s s ) h a s s u g g e ste d
r e p r e se n tin g the d e te r m in a tio n of th e s e b in a r y r e la tio n s by a g rap h -
th e o r e tic ap p ro a ch (H a ra ry , F . , N orm an , R . , and C a rtw rig h t, D . ,
1965; B u sa c k e r , R. and S a a ty , T. , 1965). The s e t o f s tim u li, r e
fe r r e d to a s v e r t ic e s , fo rm the row s and co lu m n s o f an a d ja c e n c y
m a tr ix , A . E ach e n try in the m a tr ix , a „ , r e p r e s e n ts a c o n n e c tio n
o r la ck of c o n n e c tio n b etw een s tim u li i and j. W h en ever iP j, a „ =
1; o th e r w ise a ^ = 0. A fu lly c o n n e c te d a d ja c en cy m a tr ix fo r a c o m
p le te o r d e r co n ta in s at le a s t n ( n - l) /2 l ' s . In the s p e c ia l c a s e iP i,
the b in a r y d o m in a n ce r e la tio n is r e fle x iv e and n l ' s f ill the
d ia g o n a l.
A s so m e fo rm of d o m in a n ce data is obtain ed fro m ea ch p e r so n
3
(p a ir w ise ju d g m en ts, rank o r d e r s , r a tin g s of s u b s e ts , e t c .) the o b
s e r v e d r e la tio n s can be con n ected in the a d ja c e n c y m a tr ix by e n t e r
ing l ' s fo r the a p p ro p ria te a. . he o r d e r o f the s tim u li m ay be
r e p r e s e n te d by a v e c to r con tain in g the w o n -lo s t r e c o r d s o r s c o r e s
for e a ch s tim u lu s . E ach s c o r e is c a lc u la te d by fin d in g the "w ins"
for i, w. = E ^ a .., the " lo s s e s , " I .= z f a , . and the s c o r e , s .= w .-£ ..
' l j i j 1 h hi ' i l l
The s tim u li m a y be a r r a n g e d in d e sc e n d in g o r a sc e n d in g o r d e r b a se d
on th e ir s c o r e s to fo rm the c o m p le te o r d e r . The row s and co lu m n s
of the a d ja c e n c y m a tr ix can be p erm u ted a c c o r d in g ly to v is u a lly r e
fle c t the e x te n t to w h ich the o b s e r v e d c o n n e c tio n s c o n fo r m to the
p r o p e r tie s of a c o m p le te o r d e r . F o r that m a tte r o th er ty p es of
o r d e r s m a y be s im ila r ly stu d ie d . The p r e s e n t in v e stig a tio n is lim
ite d to the stu d y of c o m p le te o r sim p le o r d e r s w h ich id e a lly have
th r ee p r o p e r tie s .
P I: T r a n s itiv ity , i f h P i and iP j, th en h P j.
P 2: A s y m m e tr y , if iP j, th en not j P i.
P 3: C o n n e c tiv ity , iP j or j P i.
The n u m b er o f a s y m m e tr ic a lly c o n n ected r e la tio n s , th e num b er o f
in tr a n s itiv e s e t s o f th r e e s tim u li, and the n u m b er of c o n n ected r e
la tio n s a ll m a y be d e te r m in e d by in s p e c tio n o f A .
C o m p lete O rd er s fro m In co m p lete D ata
F o r in s ta n c e s in w h ich the n u m b er of s tim u li b e c o m e s so
la r g e that too m an y ju d gm en ts o r ran k in gs a r e r e q u ir e d fro m a p e r
so n to fu lly c o n n e c t a ll the b in a r y r e la tio n s, s o m e fo rm o f in c o m -
4
p le te data m u s t s u ffic e . In se e k in g o r d e r s fo r in d iv id u a ls, the
m eth od e m p lo y ed for g a th erin g the data m u st a s s i s t the in d iv id u a l to
d e te r m in e enough of the c o n n e c tio n s to p e r m it r e c o v e r in g the c o m
p le te o r d e r . Ju d gm en ts that a r e g a th ered sh ou ld fa c ilita te d e t e r
m in in g fu r th e r c o n n e c tio n s by so m e m ea n s o th er than ob tain in g
m o r e ju d g m en ts.
A m eth od for exten d in g th e c o n n e c tio n s su p p lied by an in d iv id
u a l su b je ct h a s b een d e v e lo p e d by C liff (in p r e s s ) . The tr a n s itiv ity
p r o p e r ty is exten d ed fr o m c o n n e c tio n s su p p lied by a p e r s o n 's ju d g
m e n ts to o th er r e la tio n s in A by finding s u c c e s s iv e p o w er s o f A ,
2 3
e . g . , A and A . B o o le a n a r ith m e tic is p e r fo r m e d on the a d ja c e n -
2
cy m a tr ix su ch that 1 + 1 = 1, e tc . M a trix m u ltip lic a tio n of A A = A
2 3
and A A =A . F o r any s e t of th r ee v e r t ic e s , h, i, and j, w h en a
ch ain o f len g th 3 e x is t s , the "pow ering" o f A f ills in the c o n n e c tio n s
im p lie d by the tr a n s itiv ity p r o p e r ty . In the e x a m p le b elo w an id e n
tity m a tr ix w a s added to A so that su b se q u e n t p o w er s of A w ill r e
f le c t a ll the r e la tio n s w h ich a r e d e te r m in e d .
j
1
1
1
The ju d g m en ts p r e v io u sly ob tain ed w e r e hjPi and iP j . B y p ow erin g
A the c o n n e c tio n hl^j is d e te r m in e d . F o r la r g e r stim u lu s s e t s c o n -
h i j h i
1 1 0 11
O i l 01
0 0 1 0 0
5
s id e r a b ly few e r than n(n-l)/2 ju d g m en ts a r e req u ire d fro m an in d i
v id u a l. M any of the c o n n e c tio n s a r e m a d e by the p o w erin g p r o c e s s
in ste a d .
T h is p ow erin g p r o c e s s fo r c o m p le tin g c o n n e c tio n s im p lie d by
the tr a n s itiv ity p r o p e r ty p r o v id e s a m e a n s fo r d e c id in g w h ich ju d g
m en ts to obtain fro m s u b je c ts . A fte r s o m e in itia l p a ir s have b een
jud ged , an in c o m p le te a d ja c e n c y m a tr ix , V, can be p o w ered and r e
la tio n s c a n be d e te r m in e d w ith ou t r e q u ir in g ju d g m e n ts. If so m e r e
la tio n s a r e s t ill not c o n n e c te d , w ill c o n ta in v. .= v..= 0. T h ese
ij J*
u n fille d e n tr ie s in V in d ic a te w h ich ju d g m en ts a r e su b se q u e n tly
n e e d e d . T his type of p r o c e d u r e r e q u ir e s th at s o m e in te r a c tio n
o c c u r b etw een the su b je c t and the c o lle c t o r o f h is ju d g m e n ts. A
c o m p u te r -in te r a c tiv e a p p ro a ch i s r e p o r te d in th is stu d y that e n a b le s
su b je c ts to su p p ly ju d g m en ts to a c o m p u te r -b a s e d a lg o r ith m through
r e m o te te r m in a ls . The a lg o r ith m p e r fo r m s the p o w er in g , s e le c t s
the next su b s e t o f ju d g m en ts, then p r o m p ts the su b je c t fo r th em .
A lte r n a tiv e ly , a ll the ju d g m en ts th at w ill be n eed ed fro m the
su b je c t m a y be s e le c te d in ad v a n ce and o b ta in ed fr o m the su b je c t
n o n -in te r a c tiv e ly . V ariou s ty p e s o f in c o m p le te b lo c k d e s ig n s ca n
s e r v e to id en tify w h ich p a ir s to p r e s e n t to the su b je c t. A fte r a ll
th e data h a s b een e n te r e d in an a d ja c e n c y m a tr ix , p o w er s o f the
m a tr ix ca n be c a lc u la te d p r io r to c o m p u tin g the s c o r e s fo r ea ch
s tim u lu s .
The in te r a c tiv e a p p r o a c h . The in te r a c tiv e ap p ro a ch a lte r n a te s
c y c le s o f g a th erin g ju d gm en ts w ith p o w e r in g s o f the in c o m p le te a d
ja c e n c y m a tr ix , V. A fte r e a c h c y c le the in fo r m a tio n in V is u se d
to s e le c t the n ex t s e t of ju d g m en ts. S u c c e s s iv e ly c lo s e r a p p r o x im a -
3
tio n s to the c o m p le te a d ja c e n c y m a tr ix , A , a r e g e n e r a te d un til
3
n e a r ly a ll the r e la tio n s a r e c o n n ected in V .
Im p lem en tin g su ch an in te r a c tiv e p r o c e s s fo r la r g e stim u lu s
s e t s su ch a s n = 50 is m ad e rapid and e ffic ie n t th rou gh a c o m p u te r -
p r o g r a m m e d a lg o r ith m w ith w h ich Ss in te r a c t a t a r e m o te te r m in a l.
S e v e r a l a d ja c e n c y m a tr ic e s a r e s to r e d a lo n g w ith a s c o r e v e c to r and
o th e r ta b u la tio n s o f the p r o g r e s s o f fillin g in c o n n e c tio n s . The a l
g o r ith m c o lle c t s the data and d e c id e s w hat fu r th e r data is n eed ed a s
w e ll a s w hen to sto p c o lle c tin g ju d g m en ts. The im p le m e n ta tio n of
su c h an in te r a c tiv e o r d e r in g r o u tin e , In te r a c tiv e R a n k -O rd ers for
Individ ual S u b jects (IR IS), h a s b een r e p o r te d by C liff & R eyn old s
(1974).
P o w er in g a d ja c e n c y m a tr ic e s m ay be p e r fo r m e d w h en ev e r
d e s ir e d to fill in c o n n e c tio n s im p lie d by the tr a n s itiv ity p r o p e r ty .
C o n n ectio n s thus im p lie d ought not be redundant to the in d iv id u a l's
ju d g m en ts. U sin g p o w erin g in an in te r a c tiv e a p p ro a ch , e a c h s e t of
ob tain ed ju d gm en ts is p o w ered p r io r to s e le c tin g a d d itio n a l ju d g
m e n ts . F e w e r d u p lic a tio n s o f c o n n e c tio n s sh ou ld r e s u lt. One a im
of the p r e s e n t in v e s tig a tio n w as to c o m p a re the e ffe c ts o f p ow erin g
w h en a p p lied in te r a c tiv e ly to one c y c le o f p o w erin g for in c o m p le te
data g a th ered n o n -in te r a c tiv e ly .
A n o n -in te r a c tiv e a p p r o a c h . A n o n -in te r a c tiv e m eth od for s e
le c tin g a su b se t o f a ll n(n-l)/2 p a ir w ise ju d g m en ts m u s t d e te r m in e
in ad v a n ce w h ich p a ir s of stim u li to p r e s e n t. One su ch m eth od fo r
g a th erin g in c o m p le te data e m p h a s iz e s the c o n n e c tiv ity b e tw e e n o b
tain ed ju d g m en ts. C y c lic d e s ig n s , w h ich a r e one fo rm o f p a r tia lly
b a la n ced in c o m p le te b lo ck d e s ig n s , r e q u ire s u b s e ts of ju d gm en ts
to be g a th ered so that h is co n n ected to i, i to j, and s o fo rth . E ach
s e t {x} o f p a ir w ise ju d g m en ts c o n s is ts o f p a ir s in w h ich the f ir s t
m e m b e r is stim u lu s 1, 2 , ..., y , .. ., n - 1 and the se co n d m e m b e r is
x + l , x + 2 , . . . , x + y, . . . , x + n - 1. F o r e a c h s e t {x} a ll n p a ir s w ill
be c o n n ected a s long a s the g r e a te s t c o m m o n d iv is o r for x and n is
u n ity. F o r la r g e s e ts o f s tim u li s e v e r a l s e ts {x} m u st be d e s ig n a
ted in a d van ce of g a th erin g the data. In c a s e s w h er e n, X j, . . .
a ll h ave g r e a te s t co m m o n d iv is o r of u n ity , a ll s e t s {x} w ill be c o n
n e c te d . C y c lic d e s ig n s a r e a ls o b ala n ced d e s ig n s . E ach stim u lu s
a p p e a r s an eq u a l n u m b er of tim e s r e su ltin g in eq u a l n u m b ers of
c o n n e c tio n s fo r each s tim u lu s .
In s e le c tin g a n o n -in te r a c tiv e m eth o d , s p e c ia l e m p h a s is sh ou ld
be p la c ed on the c o n n e c tiv ity w ithin and a c r o s s s e t s o f p a ir s of
ju d g m en ts. G re a ter c o n n e c tiv ity a c r o s s r e la tio n s should im p r o v e
the e ffe c t o f p o w erin g the in c o m p le te a d ja c e n c y m a tr ix o n ce a ll the
ju d g m en ts re q u ire d b y the c y c lic d e s ig n a r e o b ta in ed . Im p lyin g
r e la tio n s by p o w erin g r e q u ir e s ch ain s of len gth 3 to be p r e s e n t in
V^. C on n ected s u b s e ts of stim u li can fo rm su ch c h a in s. F o r s t i m
u li h, i, j, w h en h P i and iP j,o n e p o w erin g w ill p r o v id e h P j in V^+ l
L o n g er ch a in s m u st e x is t in for a th ird p o w erin g to add c o n n e c
tio n s in V^+^.
8
M any d iffe r e n t c y c lic d e s ig n s m a y be fo rm e d for la r g e r s t i m
u lu s s e t s . A m ea n s of s e le c tin g am on g th o se c y c lic d e s ig n s w h ich
a r e m a x im a lly co n n ected h a s b een in v e stig a te d by S p en ce and
D o m on ey (in press). T hey r e co m m en d c a lc u la tin g the d e s ig n e f
fic ie n c y (DE) for s e le c te d c y c lic d e s ig n s and p ick in g that d e s ig n
w ith the m a x im u m DE. DE is the r a tio of the b etw e e n tr e a tm e n ts
v a r ia n c e for the c o m p le te d e s ig n to th at of the in c o m p le te d e s ig n .
S p en ce and D om on ey found th is m e a s u r e to be r e la te d to o th er in
d ic e s of c o n n e c te d n e ss and found that e m p lo y in g th is m eth od of s e
le c tin g s e t s {x} w as m o re s u c c e s s f u l in a M onte C a r lo e v a lu a tio n of
in c o m p le te d e s ig n s for m u ltid im e n s io n a l s c a lin g .
S u m m ary
O rd ers o f s tim u li a r e o ften n eed ed in b e h a v io r a l r e s e a r c h .
N u m ero u s b in a r y d om in an ce r e la tio n s m u st be d e te r m in e d to d e s
c r ib e a c o m p le te o r d e r . P a ir in g s tim u li to o b s e r v e th e se r e la tio n s
is the m o st a p p ro p ria te p r o c e d u r e . O rd erin g la r g e s e t s o f s tim u li
in th is m an n er fo r each su b je ct r e q u ir e s too m an y p a ir in g s . C on
se q u e n tly , in c o m p le te s e t s of ju d g m en ts, w h ich m a y be exten d ed
to a d e q u a te ly d e te r m in e c o m p le te o r d e r s , m u s t s u ffic e . A p o w e r
ing p r o c e s s w as d e s c r ib e d w h ich ca n be a p p lied in te r a c tiv e ly to s e
le c t the ju g m e n ts a s w e ll a s ex ten d th em a c c o r d in g to the t r a n s it i
v ity p r o p e r ty . The in te r a c tiv e a p p ro a ch w as c o n tr a ste d w ith a n o n -
in te r a c tiv e ap p roach in w h ich the in c o m p le te s e t o f ju d g m en ts is d e
te r m in e d a p r io r i and o n ly on e c y c le o f p o w erin g a p p lied to exten d
the ju d g m en ts.
C H A P T E R II — H Y PO T H E SE S
T h is in v e s tig a tio n w ill c o m p a re r e s u lts fo r the tw o a p p r o a c h e s
ju s t d e s c r ib e d u sin g a M onte C a r lo tec h n iq u e. S im u latin g ju d g
m ents o f p eo p le r e q u ir e s h y p o th e siz in g a m o d e l fo r th e ir b e h a v io r
and the situ a tio n a l co n d itio n s under w h ich th ey w ill m ak e ju d g m en ts.
P a r a m e te r s fo r th is s im u la tio n m o d e l m a y be v a r ie d to r e p r e s e n t
m an y o f the o c c a s io n s in w h ich in te r a c tiv e o r d e r in g w ill be e m
p lo y ed . O nce the p a r a m e te r s and th e ir v a lu e s a r e d e te r m in e d , a
M onte C a r lo v e r s io n of IRIS can g e n e r a te ju d g m en ts m uch lik e the
v e r s io n w h ich in te r a c ts w ith p e o p le . In stead of c o lle c tin g a c tu a l
ju d g m en ts, the d ir e c tio n of e a c h ju d gm en t is ran d o m ly d e te r m in e d
in a M onte C a rlo fa sio n .
The S im u la tio n M od el
E ach ju d gm en t w as d e te r m in e d a c c o r d in g to T h u r sto n e 's L aw
o f C o m p a ra tiv e Ju d gm en t (T o r g e r s o n , 1958). E ach stim u lu s is a s
su m ed to have a tru e s c a le v a lu e , L , and a d is c r im in a l p r o c e s s r e
fle c te d in an e r r o r te r m e, •, w ith N(t.,a ). A c c o r d in g to th is
J 3 0j
m o d e l, la r g e r a r e fle c t g r e a te r v a r ia b ility about the tru e s c a le
G j
v a lu e , L . The e^j m a y b e c o m e la r g e r e la tiv e to the L , c a u sin g
j P i to b e o b s e r v e d w h en t. > t .. The e ffe c t on the ob tain ed o r d e r
J— i J
w ill be to a lte r the ob tain ed s c o r e s , Sj, so th at th e ir o r d e r d o e s not
10
c o r r e sp o n d e x a c tly to the o r d e r of the tru e s c a le v a lu e s . F o r e x
a m p le , i m a y h a v e the la r g e s t tru e s c a le v a lu e but lo s e to o th er
s tim u li a s the b in a r y d o m in a n ce r e la tio n s a r e d e te r m in e d . The o b
tain ed s c o r e m a y th en be lo w e r than for so m e o th er s tim u lu s , j,
fo r w h ich no l o s s e s o c c u r r e d . C o n seq u en tly , Sj > s^ and one o r
m o r e in v e r s io n s w ill s e p a r a te the ob tain ed o r d e r fro m the tru e
o r d e r .
The T h u rston e M odel p r o v id e s fo r d iffe r e n t s e ts o f a s s u m p
tio n s r e g a rd in g g . P e r h a p s so m e s tim u li a r e m o re c o n s is te n tly
judged than o th e r s . G r e a te r c o n s is te n c y fo r i m ig h t be r e p r e s e n
ted by c < a « T he e x te n t to w hich the a d iffe r can be s p e c ifie d
' e. e . e
i J
to r e p r e s e n t a s r e a lis t ic a lly a s p o s s ib le how d iffe r e n t s tim u li m a y
a c t w h en b ein g c o m p a re d to one a n o th e r . V a rio u s op tio n s m a y be
in c o r p o r a te d in in te r a c tiv e o r d e r in g r o u tin e s to b e s t r e c o v e r the
tru e o r d e r w h en a t a •
e. e .
i J
B y c h o o sin g to r e p r e s e n t the tru e o r d e r of the s tim u li w ith a
d is ta n c e m o d e l, the d is tr ib u tio n o f the tru e s c a le v a lu e s m a y be
m a n ip u la ted to r e fle c t the d iffe r e n t ty p e s o f stim u lu s s e t s for w h ich
r e a l data is g a th e r e d . E q u a lly sp a c e d p o in ts can be r e p r e s e n te d by
a u n ifo rm d istr ib u tio n . S tim u lu s s e ts w h ich a r e c h a r a c te r is t ic a lly
u n ev en ly d istr ib u te d m ig h t be r e p r e s e n te d by a n o rm a l d istr ib u tio n ,
2
p o is s o n d istr ib u tio n , \ d istr ib u tio n , e tc .
S itu a tio n V a r ia b le s
T h ree situ a tio n a l v a r ia b le s w e r e h y p o th e siz e d to a ffe c t the
11
p e r fo r m a n c e o f IRIS: (a) e r r o r s in ju d g m en ts m ad e by s u b je c ts ,
(b) the n u m b er o f s tim u li b ein g o r d e r e d , and (c) the le v e l o f p r io r
k n ow led ge about the o r d e r o f the s tim u li. The le v e l o f e r r o r in the
T h u rston e m o d e l m a y be m a n ip u la ted a c c o r d in g to the r a tio o f the
stan d ard d e v ia tio n o f the e r r o r to the stan d ard d e v ia tio n of the tru e
s c a le v a lu e s . A s a is in c r e a s e d r e la tiv e to at , the p e r tu rb a tio n
6 I
of th e tru e s c a le v a lu e s in c r e a s e s the p r o b a b ility o f jP i w h en h >
tj. D iffe r e n t le v e ls of e r r o r sh ou ld c o n s is te n tly y ield d iffe r in g
am ou n ts of in v e r s io n s o f th e s^ r e la tiv e to the t .. B oth the in t e r
a c tiv e and n o n -in te r a c tiv e a p p r o a c h e s w ill be l e s s a b le to r e c o v e r
the tru e o r d e r a s h ig h e r le v e ls of e r r o r a r e ad d ed . The h ig h e s t and
lo w e s t e r r o r le v e ls sh ou ld r e la te to the p e r fo r m a n c e o f r e a l data a s
w e ll.
The m o s t s a lie n t fea tu r e o f IRIS sh ou ld be its a b ility to r e
c o v e r an o r d e r fo r v e r y la r g e s e t s of s tim u li w h ile req u irin g no
m o r e ju d g m en ts than so m e in d iv id u a l can be e x p e c te d to m ak e during
a d a ta -g a th e r in g s e s s io n . The n u m b er o f ju d g m en ts fo r c o m p le te
data r is e s e x p o n e n tia lly r e la tiv e to n. IRIS m u s t be a b le to p e r fo r m
a d e q u a te ly fo r d e c r e a s in g p e r c e n ta g e s o f ju d g m en ts a s n in c r e a s e s .
D iffe r e n t le v e ls fo r the n u m b er o f s tim u li stu d ied h e r e sh ou ld c o v e r
the ran ge fr o m th o s e m a r g in a lly la r g e enough to req u ire a c o m p u
t e r iz e d a p p ro a ch to th o se req u irin g the m a x im u m n u m b er o f ju d g
m e n ts an in d iv id u a l ca n p r o v id e . O ver th is ran ge o f n the r e c o v e r y
o f a n o r d e r sh ou ld not d e g e n e r a te due to la r g e r s tim u lu s s e t s . The
p e r c e n ta g e o f ju d gm en ts sh ou ld d e c r e a s e m a r k e d ly r e fle c tin g g r e a t-
12
d e g r e e s o f in c o m p le te n e s s .
A s a n in v e s tig a to r g a th e rs data fro m m o r e in d iv id u a ls, he
m a y be a b le to d e te r m in e a r e la tiv e ly good o r d e r o f the stim u li
a c r o s s s u b je c ts . A fte r s e v e r a l su b je c ts h ave p ro v id ed o r d e r s , the
o r d e r m a y be know n a p r io r i to s o m e e x te n t. P r io r k n ow led ge o f
the o r d e r m ig h t a s s i s t IRIS in r e c o v e r in g the o r d e r during the s t a r t
up p h a se o f d a ta -g a th e r in g . P o w er in g V to exten d tr a n s itiv ity r e
q u ir e s so m e p r io r ju d g m en ts. A n in itia l s e t of ju d gm en ts m u st be
ob tain ed b e fo r e c y c le s o f p ow erin g b eg in . S om e r e s t r ic t io n s on
w h ich ju d g m en ts a r e p r e s e n te d a t th is tim e cou ld p r e v e n t e a r ly e x
te n sio n s o f t r a n s it iv ity in the "w rong" d ir e c tio n . S p acin g o f the
s tim u li w h ich a r e in itia lly p a ir ed w ould red u ce the p r o b a b ility th at
ju d g m en ts in the r e v e r s e d ir e c tio n to the tru e o r d e r w ould be o b
ta in e d . S in c e th e r e m ig h t be d isa d v a n ta g e s to adopting su ch a p r o
c e d u r e , th is s ta r t-u p op tion w as in v e stig a te d a s a situ a tio n a l v a r i
a b le .
A lg o r ith m V a r ia b le s
Tw o v a r ia b le s a ffe c tin g the o p e r a tio n o f IRIS w e r e e x a m in e d .
The n u m b er o f ju d g m en ts ob tain ed b etw e e n c y c le s o f p ow erin g m a y
be v a r ie d . One m eth od o f d e te r m in in g how m an y and w h ich ju d g
m e n ts to o b tain is to fo rm fo c u s e s o f s tim u li. E ach fo cu s w ill c o n
ta in k s tim u li. A ll p a ir s of th e se s tim u li, k(k-l)/2, a r e c o lle c te d
b e tw e e n c y c le s o f p o w er in g . If one ju d gm en t is g a th ered e a c h tim e ,
e a c h fo cu s o f s tim u li that is fo rm e d co n ta in s only tw o m e m b e r s .
13
L a r g e r fo c u s e s sh ou ld tend to in c r e a s e the n u m b er of ju d gm en ts
req u ired to ob tain a c o m p le te o r d e r . The in c r e a s e in red u n d an cy of
the ju d g m en ts m ig h t c o m p en sa te fo r h ig h e r le v e ls o f e r r o r . A n i m
p r o v e m e n t in p e r fo r m a n c e could rep a y the in c r e a s e in ju d g m en ts
and p ro v id e a fu rth er sa v in g s in c o s t a s the n u m b er of p o w er in g s is
red u ced .
The se co n d v a r ia b le to be e x p lo r e d w as the n u m b er o f p o w e r
in g s on ea ch c y c le . E ach tim e the a d ja c e n c y m a tr ix , V, is p o w ered ,
a ll c o n n e c tio n s hPj a r e added fo r s e t s of r e la tio n s h P i and iP j. If
a ll h P j a r e c o n n e c te d , fu rth er p ow erin g d o e s not a lte r V. S in ce the
p o w erin g p r o c e s s can be q u ite c o s tly in c o m p u ter tim e , s o m e lim it
m u st be p la c e d on fu rth er p o w er in g . On the o th er hand in s u ffic ie n t
p o w erin g s o f V w ill r e q u ire fillin g in c o n n e c tio n s w ith m o r e ju d g
m e n ts . P o w er in g cou ld be in c r e a s e d to f ill in r e la tio n s m o r e r a p id
ly and red u ce the in c o n s is te n c y in h e r en t in d e te r m in in g r e la tio n s
fro m ju d g m e n ts. T h ere cou ld be a sa v in g s o v e r a ll if the o r d e r w e r e
ob tain ed in fe w e r p o w erin g c y c le s .
P u r p o se
The a im of th is in v e stig a tio n w as to e v a lu a te a c o m p u te r -
in te r a c tiv e o r d e r in g a lg o r ith m . A m o d e l of the ju d gm en t p r o c e s s for
p a ir w ise c o m p a r is o n s of s tim u li w a s s e le c te d and data g e n e r a te d
u sin g a M onte C a rlo tec h n iq u e. V a lu es w e r e s e le c te d fo r the p a r
a m e te r s in th is ju d gm en t m o d e l to r e p r e s e n t c e r ta in ty p e s o f r e a l
data that m ig h t be c o lle c te d . T h ree situ a tio n a l v a r ia b le s w e r e m a n
14
ipulated, le v e l of e r ro r , num ber of stim u li, and p rior know ledge of
the o rd er, to r e p r ese n t a c r o s s se ctio n of exp erim en tal conditions.
Two v ariab les that affect the operation of the algorith m w ere a lso
studied, focus s iz e and num ber of pow erings in each c y c le .
This d a ta -sim u la tio n p roced u re w as applied in in tera ctiv e and
n o n -in teractive approaches to gathering the judgm en ts. R esu lts for
the two approaches w ere com p ared to d eterm in e when the in te r a c
tive approach is to be p r e fe r r ed .
CH APTER IH - METHOD
P ro ced u re
An A P L v e r sio n of IRIS (C liff and R eynolds, 1974) was c o n
verted to FO RTRAN IV and the judgm ent gathering routine replaced
with a judgm ent sim u la to r . C ase V of the Thurstone Law of C om
parative Judgm ent was se le c te d a s a m od el for sim ulating judgm ents
of su b jects,
d _ . j + v - w _
hlJ 2 \ f ?
A p seu d o-ran d om num ber gen erator supplied values from a uniform
distribution w hich the judgm ent sim u la to r tran sform ed to n orm ally
distributed e, . and e, .. The standard deviation of the e r r o r , a ,
hi hj e
w as the sa m e for a ll stim u li. The e^ w e r e added to the true sc a le
v a lu e s, L and L , which w e r e u n iform ly distributed with at = 1. The
sig n of each d ^ j d eterm in ed w hether iPj or jP i.
Judgm ents w ere generated until a su b -m a tr ix of the in com p lete
adjacen cy m a trix , V, w as filled . The num ber of stim u li in the fo
c u s , k, d eterm in ed the num ber of judgm en ts, k ( k - l) /2 , b etw een
pow ering c y c le s , c, of the V,. A fter a ll th ese judgm ents w ere gen
erated , any sy m m e tr ic connections , v .. = v.. = 1, in the focus m atrix
J ^
w ere rem oved . Then the a s y m m e tr ic connections w e r e en tered in
15
16
V . v f w as obtained by adding an identity m a trix , I, to V, and
2
m ultiplying by V, to obtain V, , and so forth up to the lim it of p
p o w erin g s. The te st of betw een n ess (O 'N eil and O 'N eil, 1973) was
om itted and in stead n u m eric inform ation was c o lle cte d in V^. E n
tr ie s w ere not r e str ic te d to l 's and 0 's. A s two or m o re n o n -z e ro
e n trie s in vj co rresp on d ed a c r o s s row s and co lu m n s, the en tries
in b eca m e g r e a te r than one. The n u m eric value r e flec te d the
extent to which a ll relation s im p lied determ in in g any one relation .
Two s c o r e v e c to r s w e r e calcu lated , one b ased on the p r e se n c e of
n o n -z e r o e n tr ie s , the other on the m e tr ic in form ation in each entry.
The s c o r e , Sj, for a given stim u lu s w as calcu lated from a c c o r
ding to "wins" m inus " lo s s e s ," s. = w . - I . , in which w. = E^v.. and
J J J J 1 J1
jf c j = E^ The n u m eric s c o r e w as s im ila r ly calcu lated excep t the
m e tr ic in form ation in each entry w as included. The stim u lu s p r e
sentation index, IP, w as a function of m a tc h e s,
m . = E"(v.. v . .),
J Ji ij
or the num ber of s y m m e tr ic connections in V*5, a s w ell a s w and A,
IP. = n - (w. + A . - m .) - 1 .
J J J J
When IPj = n -1 , no in form ation about the relation s of j is known. If
IPj = 0, a ll relation s a r e d eterm in ed for j.
S electio n of stim u li for the next focus o f judgm ents began by
locating m ax IPj. Then the p resen tation in d ices of the other stim u li
w ere d e c r e a s e d by 1 if they w ere p r e v io u sly paired with j. The
17
focus of stim u li was constructed by repeating this p r o c e s s , each
tim e adding the stim ulu s with m ax IP^ until k stim u li w e r e s e le c te d .
Additional fo cu ses w ere form ed and c y c le s of pow ering continued
until a ll the relation s in V*3 w e r e d eterm in ed .
Since relation s could be eith er sy m m e tr ic o r a s y m m e tr ic ,
the p r o g r e ss the a lgorith m m ade in filling the m a tr ix with a s y m
m e tr ic relation s w as record ed . The num ber of a s y m m e tr ie s (ASY)
in VP is ASY = E^(w. + J L ■ - 2 m .)/2 . ASY w as ca lcu la ted for each Vp.
J J J J c
To d eterm in e when the algorith m w as no lo n g e r m aking su fficien t
p r o g r e ss in filling the m a trix with a s y m m e tr ic r e la tio n s to w arrant
continuing, four stopping ru les w ere studied. T h ree ru les defined
p r o g r e ss in ter m s of adding a s y m m e tr ie s from one c y c le of p o w e r
ing to the next: (a) Stopping Rule for M onotonicity (SR-M ) of the
values calculated for ASY, (b) Stopping Rule for T rends of T hree
(SR -T3) c y c le s of ASY, and (c) Stopping Rule for T rends of F ive
(SR -T5) c y c le s of ASY. Both trend in d ices w e r e adjusted to a llo w
for so m e sid ew ays m o v em en t and w ere b ased only on judgm ents of
p airs which had not p rev io u sly been obtained. The Stopping Rule
for P ow erin g (SR -P) required that s u c c e s s iv e p ow ers w ithin a c y c le
m aintain or in c r e a s e the a s y m m e tr ie s . The th ree r u le s applied b e
tw een c y c le s w ere:
SR-M : If ASY < ASY ., stop;
c C -1 ^
SR -T 3: If (-A SY , + A S Y J /( S N R . / 3 ) < 1 , stop,
c C i = 1 1
NR. = new ly d eterm in ed relation s in focu s i;
1 5
SR -T 5: If (-2A SY . - ASY ., + ASY . + 2ASY ) / S NR.< l,s to p .
C“4 C -O C - 1 C > _ | 1
18
The stopping rule applied b etw een pow erings of V was:
SR -P : If ASY < ASY stop, p = 2 , 3, or 4.
p p — l
SR -M and S R -P a r e m onotonic functions betw een c y c le s and betw een
p ow erin gs, r e sp ec tiv e ly . P r o g r e s s is defined a s m aintaining or i n
c r ea sin g ASY. SR -T 3 and SR -T 5 denote an unacceptable downward
or sid ew ays shift in the trend of ASY a c r o s s eith er th ree or five
c y c le s , r e sp ec tiv e ly . U sing new relation s per focus in stead of
k ( k - l) /2 reco g n iz e s the p o ssib ility that so m e judgm ents in the focus
m a y have been m ade in an e a r lie r focus and a r e not expected to form
new a s y m m e tr ie s . It w as exp ected that so m e stopping ru les would
not be activated on so m e runs p rior to a ll relation s being determ ined.
D esign s
Interactive approach. F ir s t, the in tera ctiv e approach was
evaluated se p a r a te ly from the n o n -in tera ctiv e approach. F ive fa c
to rs that m ight influence the p erfo rm a n ce of IRIS w ere m anipulated
in three sep arate a n a ly ses of varian ce. The extent of the error,
E = a /a was se t to .2 0 , .4 0 , and . 60. T h ese th ree le v e ls of e r r o r
6 S
w ere s e le c te d after num erous te s t runs indicated: (a) the low e r r o r
condition differed from E = .0 0 , (b) the th ree le v e ls w ere unlikely
to o verlap in their effect, (c) E > . 60 quickly produced v e r y sy m
m e tr ic which did not differ c o n siste n tly for differen t le v e ls of
e r r o r , and (d) co rresp on d en ce of the obtained o rd ers to their true
o rd ers e n co m p a ssed the range of in te r e s t. The s iz e of each focus,
K, was varied from 3 to 6 causin g the num ber of judgm ents gen er-
19
ated in each focus to vary fro m 3 to 15. K = 3 w as s e le c te d a s the
s m a lle s t focus s iz e to be as com p arab le as p o ssib le with K = 2.
K = 2 w as not studied b eca u se of the s p e c ia l a lg eb ra ic pow ering p r o
c e s s required to keep pow ering c o sts down. P ilo t runs at K = 6 gave
indications that the num ber of judgm ents required w as not e x c e s s iv e
and far few er pow erin gs w ere needed to obtain a solution. The num
ber of stim u li, N, w as fixed at 25 and 49. The s iz e s ch osen for the
stim ulu s s e ts extended a c r o s s the range from ju st la rg e enough to
require an in com p lete d e sig n to potentially requiring too m any judg
m en ts. The values 25 and 49 p erm itted constructin g the w id e st
variety of c y c lic d e sig n s that would be fully balanced a s w ell a s m a x
im iz e con n ected n ess a c r o s s s e ts {x} of n p a ir s. T h ese th ree v a r
iab les w ere fully c r o s s e d and three rep lication s of each condition
run in an E(3) xK(2) X N(2) d esign . Each run w as analogous to a
new su b ject m aking judgm ents which d eterm in e a sim p le ord er for
eith er 25 or 49 stim u li.
Two other v a r ia b le s w e r e exam ined by taking a portion of this
d esig n and adding new conditions which w ere fully c r o s s e d . Since P,
the num ber of p o w er in g s, w as alw ays th ree, two additional c e lls
w ere studied in w hich P = 4. In ord er for additional pow erings to
add m o r e a s y m m e tr ic , rather than sy m m e tr ic , r e la tio n s, E m u st
be r e la tiv e ly low. Since adding m o re relation s by pow ering ought to
reduce the num ber of judgm ents needed, additional pow erings should
be m o st helpful for N = 49. In this s m a lle r a n a ly sis, N = 49 and E =
. 20. Only K was v a ried in addition to P in a K(2) x P(2) d esig n .
20
The fifth variab le w as type of start. Random (Rnd) or Known
(Kwn). F o r so m e o c c a sio n s in which the in teractive ordering a p
proach is app licab le, so m e prior inform ation m ay be av a ila b le a s
to the true ordering of the stim u li for a subject. This inform ation
m ight be helpful in the sta rt-u p period in which fo cu se s are form ed,
but no pow ering o c c u r s. Only a fte r each stim ulu s w as connected to
another by a judgm ent was pow ering initiated. H igher le v e ls of e r
ror can cause m any of th ese e a r ly judgm ents to be the r e v e r s e of
the true o rd er. Obtaining these r e v e r s a ls during the sta rt-u p phase
w ill r e su lt in in c o r r e c tly im plying other relation s throughout m o st
of the pow ering c y c le s . To o ffset this effect, the stim u li can be
align ed in th eir exp ected ord er and ea rly fo c u se s filled with stim u li
a s far apart in the true order as p o ssib le . F or K = 3 the f ir s t focus
would contain {l, n /3 + 1, 2n /3 + l ) . The second focus would contain
[2, n / 3 + 2, 2 n /3 + 2}, etc.
Since the true ord er is not known with com p lete certain ty, this
stim u lu s s e le c tio n procedure was a lte r ed to r eflect the sam e le v e l
of e r r o r that e x iste d in the judgm ents gathered for that run. F ir s t ,
the true sc a le valu es w ere perturbed ju st a s in the sim u lation of
ju d gm en ts. Then, the ord er of perturbed sc a le valu es was d e te r
m ined. F ro m this o r d e r the sta rt-u p focu ses w ere form ed by s e
lectin g e v e r y n /k -th judgm ent. The Random start involved random
ly generating a s e t of s c a le values for the stim u li, ord erin g them ,
then se le c tin g e v e r y n /k -th stim u lu s for the e a r ly fo c u se s . The
d e sig n for this a n a ly sis was E(2) X S(2) with N = 2 5 , K = 3, P = 3 , and
21
E = . 40 or . 60. E ach c e ll was replicated three tim e s.
Within this sa m e fram ew ork of a n a ly se s of varian ce, each run
included a com p lete judgm ents p rocedure for co m p a riso n p u rp oses.
Once a ll relation s w ere determ in ed in V^, the p rogram generated a ll
the judgm ents n e c e s s a r y to fill V. This m a trix w-.s not pow ered
sin c e a ll relation s w ere d eterm in ed by judgm ents alon e.
M e a su re s of p e r fo r m a n c e . V arious in d ices w ere constructed
to a s s e s s the p erform an ce of IRIS. They each fell into one of five
c a te g o r ie s . The f ir s t c a teg o ry included m e a s u r e s of the c lo s e n e s s
of the obtained ord er to the true order. The obtained ord er was
rep resen ted by eith er the sc o r e v ector, s, or the n u m eric s c o r e
v e c to r , b, which re p r ese n te d n u m erica lly the extent to which filled
en trie s in w ere im p lied by the netw ork of rela tio n s. Both
K endall's t and P e a r s o n 's r w e r e calcu lated betw een each s c o r e
v e c to r , s or b, and the true s c o r e s , t. The in d ices w ere labeled
T ., the ordinal c o r re la tio n betw een the s c o r e s and the true s c a le
st
v a lu e s, T^t, the ordinal c o r re la tio n betw een the n u m eric s c o r e s and
the true s c a le v a lu e s, r gt and r ^ , the corresp on d in g P e a r s o n c o r
rela tio n s. The s c o r e v e c to r and num eric s c o r e v ecto r w ere id e n
tic a l for the com p lete judgm ents m a tr ix . C onsequently, the two
m e a s u r e s of c o r re la tio n betw een the s c o r e s calcu lated for the c o m
plete judgm ents ad jacen cy m atrix, A , and the true sc a le values
w ere d esign ated and ra ^» rep resen tin g all judgm ents.
The secon d c a te g o r y d e scrib ed the extent of in tran sitivity in
the obtained o rd er. The P e r cent of Intran sitivity (PI) w as c a lc u la -
22
ted from V^. Since s y m m e tr ie s a s w ell a s a s y m m e tr ie s occu r in
V^, a m ethod for deciding on the in tran sitivity of each s e t of three
stim u li was develop ed to be as com p arab le as p o ssib le to m a tr ic e s
containing only a s y m m e tr ic rela tio n s. E ach se t of th ree stim u li w as
d eterm in ed to be in tra n sitiv e if one chain ex isted which w as in tran
s itiv e . The num ber of in tr a n sitiv ities w as then in crem en ted by one.
The m a xim u m num ber of in tr a n sitiv ities rem ain ed the sa m e a s for
chains w as not reco rd ed . The index for the rate of in tra n sitiv ity for
solu tions d eriv ed from a ll ( _ ^ judgm ents was d esign ated PI n .
The third c a teg o ry contained m e a s u r e s of a s y m m e tr y . It
se e m e d p o ssib le that the d e g r e e of a sy m m e tr y m ight be em ployed
to d e sc r ib e the good n ess of an obtained ordering w hen the true o r
d er is not known. When m o re relation s fill the ad jacen cy m atrix,
the s c o r e v e c to r is m o re overd eterm in ed . A m o r e filled m a trix
m ight r e p r e se n t a m o re a ccu ra te obtained ord er. This a cc u r a cy
m ight d eclin e, though, if sy m m e tr ic connections a r e being added.
The p er cen t of a s y m m e tr ic relation s in the ad jacen cy m a tr ix V1
w as th erefo re lab eled G o o d n e s s -o f-F ill (GOF^).
The fourth ca te g o r y of in d ices d e sc rib e d the extent to which
connections am ong the v e r tic e s o r stim u li have been m ade. In the
in tera ctiv e ord erin g p r o c e s s both the subject and the a lgorith m c o n
n ect v e r tic e s . The connections m ade by the su b ject a r e the judg
m en ts he su p p lies. F o r co m p lete judgm ents data he su p p lies 100%
of the con n ection s. A n index of Judgm ent E fficien cy (JE) w as c a l-
a s y m m e tr ic a d jacen cy m a tr ic e s , The num ber of tra n sitiv e
23
culated for each c y c le to indicate the extent to which judgm ents w ere
generated by the algorith m , rep resen tin g judgm ents a subject m a k es.
E very judgm ent generated including repeated judgm ents w ere s u m
m ed and the ratio of this sum to the Po s s ible p airs m ultiplied
by 100 to form a per cent. A secon d index rep resen ted the P e r cent
of F ille d R elations (PFR ) in V*5. This index was based on the c o n
nections m ade by both judgm ents and pow erings of them by the a l
gorithm . When all ^ ) connections have been m ade, w hether a s y m
m etr ic or s y m m e tr ic , a ll r e la tio n s a r e d eterm in ed in and P F R =
100.
The la st categ o ry of m e a s u r e s w as d esign ed to re flec t the r e
liab ility of the judgm ents being m a d e , in a repeated m e a s u r e s s e n s e ,
and the p red ictive validity in h eren t in the obtained o rd er. R e lia b il
ity of the judgm ents m a y be in d icated by how frequently a secon d
judgm ent for one pair r e v e r s e s the f ir s t judgm ent. F o r ty judgm ents
w ere gen erated on each run w hich repeated so m e of the judgm ents
needed to obtain the o r d e r . The ratio of the num ber of r e v e r s a ls to
the num ber of p a ir s, 40, form ed the R e v e r s a l of Judgm ents Index
(RJI). A secon d r e v e r s a l rate w a s ca lcu la ted to form an index of
P r e d ictiv e V alidity (PV ). (A l is t of a c r o n y m s appears in the A ppen
dix, page 5 5 ). The v e c to r of s c o r e s obtained from "predicts"
the d irectio n of future jud gm en ts. If > s^, then a subject judging
this p a ir of stim u li should rep ort i £ j . If jl?i is rep orted , the judg
m ent is the r e v e r s e of the p red icted judgm ent. The num ber of r e
v e r sa ls of p red icted judgm ents w a s calcu lated for 40 judgm ents gen
24
erated p rior to judgm ents on w h ich the o r d e r w as based . The ratio
of r e v e r s a ls to judgm ents w as subtracted from 1. 0 to form PV .
Since this v e r sio n of IRIS req u ired eith er iPj or jP i to be g e n e r a
ted. no r e v e r s a l w as counted if s. = s ..
i J
C om p arison s with a n o n -in te r a c tiv e approach. Since in te r a c
tive ordering r eq u ires a c c e s s to com p u ters through exp en sive t e r
m in als and a s p e c ia l algorithm , the r e su lts from the Monte C arlo
runs of IRIS w e r e com pared to an in com p lete n o n -in tera ctiv e a p
proach. The co m p a riso n m ethod u tilized se ts of connected ju d g
ments s e le c te d for a cyclic d e s ig n (Spence and D om oney, in p r e s s ).
C ertain c y c lic d e sig n s are m o re fully connected than o th e rs. They
a re a ll balanced in that each stim u lu s appears q tim e s . The le v e l
of q d eterm in es the per cent of judgm en ts required by the d esig n .
Two le v e ls of q w ere s e le c te d b ased on the r e su lts of the runs
of IRIS for N of 49. F or the nine in tera ctiv e runs at K = 3, the
m ean of JE was 25. 1% and for K of 6, 34.7% . By settin g q= 12
and 16, two c y c lic design s w ere co n stru cted requiring 25% and 33§%
of a ll judgm ents. The p rocedure recom m en d ed by Spence and
D om oney (in p r e s s ) was follow ed in se le c tin g the c y c lic s e ts . In the
d esig n v ecto r, g, there are 24 e le m e n ts when n = 4 9 . E ntering a 1
in g d esign ates the value of ( x ) , the sep aration factor. If g^= 1,
adjacent stim u li a r e paired, i . e . , lw /2 , 2 w /3 , e tc . F o r q = l6
eight such s e ts of judgm ents m u s t be gathered, each se t containing
n judgm ents. The d esign v e c to r with the h ig h est d e sig n e fficie n c y
of those sam p led c o n sisted of altern atin g a 1 follow ed by two 0 's,
25
beginning with g^ = 1. The d e sig n efficien cy (DE) w as .9 6 5 . F or
q= 12 only s ix c y c lic se ts a r e needed. The repeated pattern of l's
and 0's was changed to a 1 follow ed by four 0 's. DE was .9 4 2 .
An a lgorith m was p rogram m ed in FORTRAN IV to s e le c t and
generate the n e c e s s a r y judgm ents for a replication of the K (2)x E(3)
a n a ly sis of varia n ce at N = 4 9 , P=3, S = Rnd. T h ese c e lls based on
the n o n -in tera ctiv e approach w e r e com bined with the correspond ing
c e lls from the in teractive a n a ly s is . K was changed to J to r e p r e
sen t the per cent of judgm ents required. K = 3 corresp on d ed to J =
25% and K = 6 to J = 33§%. The com p lete design for the a n a ly sis of
varian ce w as a repeated m e a s u r e s d esign in wi.'ch for each m ethod
(M), in te r a c tiv e v s. n o n -in te r a c tiv e , each pow er of V w as c o m
pared. T h e d e sig n J(2)xE(3)xM(2)xP(3) was analyzed based on three
rep lication s p er c e ll.
CH APTER IV — RESULTS
B efore the in tera ctiv e ordering a lg o rith m 's p erform an ce could
be exam in ed , v f had to be se le c te d from am ong a ll the pow ering
c y c le s to d eterm in e the ordering of the stim u li. The ideal rule
would alw ays locate that V*3 for which t . was h igh est and had fir s t
C S t
o ccu rred . The m ean s of the two m e a s u r e s which tend to r e fle c t
3
th ese two c o n sid er a tio n s, t . (calcu lated from V ) and JE , w ere
s c
tabulated for each stopping rule. Four stopping ru les and the d e
fault stop, P F R = 100, w ere com p ared a c r o s s a ll 48 runs, 36 for the
E x K X N d e sig n and 6 m o re each for the K X P and the E x S d e
sig n s. In num erous in sta n ces the values o f T and JE for the
3
stopping ru les w ere the sa m e a s those for the default stop, when
the rule w as not a ctivated . Excluding the default stop, the ord er of
operation of ru les b ased on the h igh est T . w as a lm o s t the r e v e r s e
3
TA BLE 1
C om p arison of Stopping R ules
M ean t . M ean JE
3
SR -T 3 .7 9 5 3 6 .2
SR -M .8 0 6 4 1 .4
SR -T 5 .8 2 5 4 1 .0
S R -P .8 4 0 4 7 .5
D efault Stop . 839 4 9 .7
Note: T hree runs stopped just p rior to P F R = 100.
26
27
of the ord er for the lo w e st JE. The only ex cep tio n was SR -T 5. It
w as not the b e s t for e ith e r c r ite r io n , but it w as alw ays second b est.
T h ese a v e r a g e s did not n e c e s s a r ily r e fle c t how w ell IRIS did
on t a c r o s s a ll c y c le s . The m axim u m t obtained on a given
St^ s
run could occur on so m e c y c le other than the one picked by a stop
ping rule. This m a x im u m could a ls o o ccu r on m o re than one cy cle.
The peak c y c le s of t w e r e id en tified for each of 27 s e le c te d runs.
3
E ach peak contained a ll the in sta n c e s in which t . was h igh est.
3
Som e peaks contained lo w e r valu es of Tgt which w ere in te r sp e r se d
am ong the h igh est t . • None d iffered by m o re than .0 1 . The best
3
stopping rule for t . , the b e st for JE , and SR -T 5 w e r e com p ared
8
2
in a X te s t. Table 2 d en otes w hether the c y c le s e le c te d by the
rule fell b efo re, during, or a fter the peak c y c le s of T g^ . on that run.
TA BLE 2
Locating P eak s of t for S elected Runs
3
B efore During A fter
SR -T 5 7 16 4
SR -T3 18 6 3
S R -P 2 16 9
Since X = 24. 02, d. f. = 4, the te s t rev ea led sign ifican t dif-
2
fe r e n c e s betw een stopping r u le s, p c . 0 1 . On the b a sis of the \
te s t and the pattern of m ea n s on T > and JE, S R -T 5 w as s e le c te d
3 *
a s the b est stopping rule. It tended to locate the b e st Tgt a s often
a s S R -P and u su a lly required few er judgm en ts. Under this stopping
28
rule the range of the num ber of judgm ents over a ll 48 runs w as
108-555. The m a xim u m for K = 3 w as con sid era b ly few er, 387.
S election of P e r fo r m a n c e Indices
F iv e c a te g o r ie s of p erform an ce of the in tera ctiv e procedure
w ere identified. H ow ever, four c a te g o r ie s contained m ore than one
m e a su r e . Since it m ight be the c a s e that so m e m e a s u r e s within
c a te g o r ie s w ere quite redundant, a s e le c tio n procedure was d e v ise d
for d eterm in in g which in d ices to u se a s dependent v a ria b les in the
a n a ly ses of va ria n ce.
A c o r re la tio n m a trix of a ll the m e a s u r e s w as form ed and the
c a te g o r ie s of in d ices iso la ted . The m e a s u r e s of p erform an ce for
the in com p lete judgm ents solu tion w ere included to provide b a s e
lin es for the in tera ctiv e p ro ced u re. The c a te g o r ie s with in d ice s a p
pearing to be m o re highly related to the f ir s t categ o ry w ere p o s i
tioned c lo s e r to the f ir s t in the m a trix . T . and G O F, which a r e
st j 1
com puted from V w e r e om itted, t . w as not exp ected to be a
1
good m e a su r e of c o rresp o n d en ce to the true ord er b eca u se of the
n e c e s s ity of pow ering the judgm ent m a trix , V. G O Fj was n early
id en tical to JE sin c e , for the m o st part, each new judgm ent added
another a sy m m e tr y .
Each c o r r e la tio n r e fle c ts the extent to which two in d ices v a r
ied s im ila r ly under the m anipulations of the five independent v a r i
ab les E, K, N, P , and S a c r o s s n = 48 runs. E xam ining ca te g o r y I,
T . and r . w e r e the m o st highly related , t . and r a s w e ll as
Sv<2 St St0 St
TABLE 3
In tercorrelation s of P erfo rm a n ce Indices
C ategory
I
j
V II i n IV
rbt rst Tbt
Tst8
7 * T „
st3 all rall
PV RJI PI
p I 11
all
GOF a GOF3 JE PFR
rbt
84 87 85 83 60 55 60 -48 -37 -64 46 38 33 18
r st
- 84 86 97 80 72 71 -53 -59 -80 33 45 14 -07
Tbt
- 90 88 62 52 68 -40 -35 -63 56 42 . 46 26
Ts ts
- 90 67 59 62 -48 -42 -69 64 56 46 32
Tst3
79
70 73 -49
-55 -82 43 46 26 -01
Tall
- 91 75 -51 -52 -77 14 26 03 -24
rall
- 67 -42 -36 -67 08 18 -02 -17
PV - -41 -32 -72 28 22 21 -08
RJI - 48 56 -39 -4 4 -25 -06
PI - 73 -21 -71 13 35
PIa ll
- -41
-49 -21 21
GOF2 - 69 79 68
g o f 3 - 23 35
JE
_
59
P F R
Note: D ecim a l points w e r e om itted.
30
a n d r ^ t d i f f e r e d o n l y i n t h e c o r r e l a t i o n a l m e a s u r e s u s e d , t v s . r .
a n d ra ^ w e r e n e x t m o s t h i g h l y r e l a t e d , a g a i n d i f f e r i n g o n l y i n
t h e m e t h o d o f c o r r e l a t i o n u s e d , r a n d t w e r e v e r y s i m i l a r l y
St st^
related to a ll the other in d ic e s, t . w as p r e fe r r ed to r due to
St^ s
the th e o re tic a l ap p rop riaten ess of the ordinal m e a su r e and to the
m o re favorable distributional p ro p erties of K endall's t .
T . w as com pared to T to d eterm in e from which V1 the ob-
3 2
tained order should be determ in ed . U sing Ta j j and PIa j^ a s a b a sis
of co m p a riso n , t . varied m ore s im ila r ly to them than did t . .
8 3 8 2
The a v e r a g e s over a ll runs w ere ch eck ed , T . = . 825 and T . =
3 8 2
.8 2 2 . t . was s e le c te d over t . a s it w as higher and m o re c lo s e -
3 8 2
ly related to
The n u m eric s c o r e in d ices co rresp o n d ed l e s s to m o st of the
o t h e r i n d i c e s t h a n d i d t . , p a r t i c u l a r l y t o t h e c o m p l e t e j u d g m e n t s
3
in d ic e s . T heir patterns of c o r r e la tio n s w ere m o re like T . than
8 2
T t • T , w as a ls o s e le c te d over them a s the dependent variab le
3 3
to r e p r e se n t the c lo s e n e s s of the obtained ord er to the true o rd er.
Since t w as s e le c te d as a dependent va ria b le, G O F , w as
3
c h o se n a s the m e a su r e of a s y m m e tr y in stead of G O F 2 for c a teg o ry
III. In ca teg o ry IV JE and P F R w e r e som ew h at related , thus redun
dant. Since P F R was le s s related to ca te g o r y I m e a s u r e s , it w as
not an alyzed further. JE w as se le c te d a s the dependent variab le
from th is c a teg o ry . It r e fle c ts the e ffe c ts of the independent v a r i
a b les on the am ount of judgm ents Ss would have to m ak e. PV and
RJI w e r e not highly related and perhaps c o m p rise two c a te g o r ie s .
31
Since RJI did not relate as highly to Tgt , only PV w as ch o se n as
a dependent variab le for further a n a ly s is .
The P erfo rm a n ce of IRIS
The five dependent va ria b les w e r e an alyzed sep a r a te ly for
E (3 )x K(2) x N(2). Trend a n a ly se s w e r e p erform ed on E. The lin
e a r effect for E w as highly sig n ifica n t on all m e a s u r e s . In c r e
m en ts in E produced d e c r em e n ts in each m e a su r e including JE.
(Table A in the Appendix, page 55, lis t s all the c e ll m e a n s. )
TABLE 4
P e r fo r m a n c e of the Interactive A pproach
Source df F
T8t3
PV PI g o f 3 JE
E r r o r (E)
L inear 1 93. 60 53. 19 62. 17 6 2 .0 7 * 65. 08
Q uadratic
Stim ulus Set
1 .8 3 . 34 .3 7 .3 1 . 53
(N)
F o c u s Size
1 . 80 1. 94 1. 84 .0 0 6 2 9 .8 9
(K) 1 3 .9 9 4. 70 16.08* 5. 58 + 3 2 6 .7 6
E x N 2 2. 56 2. 54 1. 05 2. 23 3. 20
E x K 2 1. 12 . 27 2. 84 9 .9 1 * 9. 11
N x K 1 7 .6 4 * .4 7 1.76 3 .9 6 68. 09
E x N x K 2 1. 27 3. 11 1. 78 2. 20 . 45
*p < . 01
*p < .0 5
32
JE w as the m o st affected of the five dependent m e a s u r e s .
M ain e ffe c ts for N and K w ere sign ifican t, occu rin g in the e x
pected d irectio n . S m a lle r num bers of stim u li or la r g e r fo cu se s
in c r e a se d the p ercen t of judgm ents needed. Both PI and GOF^
a ls o had sign ifican t m ain effe c ts for K. The le v e l of in tran sitivity
w as in c r e a se d and tended to be l e s s a s y m m e tr ic a s K in c r e a se d .
T . w as not a ffected by e ith e r the focus s iz e or num ber of stim u li.
s 3
T here w ere only four sign ifican t in te r a c tio n s, a ll tw o-w ay.
T f and JE had sign ifican t in tera ctio n s for N x K. The e ffe c ts for
3
K w e r e s im ila r at N= 25 on both m e a s u r e s . The in c r e a s e in K at
N = 49 cau sed t . to go down w hile in c r e a sin g JE . At N = 49 the
3
only plus for in c r e a s in g the focus s iz e was in reducing the num ber
of p ow erin gs. t . dropped and m o re judgm ents w ere required.
3
Both GOF^ and JE had sign ifican t E x K in te r a c tio n s. F o r the low
le v e l of e r r o r , in c r e a s in g the focus s iz e im p roved the a sy m m e tr y of
VP. F o r m od erate and high e r r o r the redundancy in judgm ents of
the la r g e r focus resu lted in l e s s a sy m m e tr y . At K = 3 the low and
m o d erate le v e ls of e r r o r yielded a lm o s t the sa m e GOF^. A s K in
c r e a s e d , JE in c r e a s e d a c r o s s a ll le v e ls of e r r o r . The in c r e a s e was
g r e a te r for E = . 20. T here w as little d ifferen ce b etw een the low and
m o d erate le v e ls of e r r o r on JE at K = 3. Looking at the two higher
e r r o r le v e ls , a s K in c r e a s e d , GOF^ d e c r e a s e d w hile JE in c r e a se d .
Instead of m o re judgm ents yielding m o r e a s y m m e tr y , low er
g o o d n e s s -o f-fill resu lted .
A ll the in d ice s w e r e affected by the e r r o r s im ila r ly to Tgt .
Beyond that, P V was a ffected m o st s im ila r ly to r . a c r o s s a ll the
33
34
T / i
■ ' - Af- r i ' . r
’ r, v ' 1
I. U :: 1 1 p j T r M
35
e ffe c ts . T h ese two m e a s u r e s d iffered in that PV had no sign ificant
effect for N x K. In te r m s of an index of the rela tiv e good n ess of
an obtained ord er, P V appeared to provide the b e st approxim ation
to r for real data. One additional a n a ly s is w as p erform ed on
3
GOF^ to d eterm in e if it, too, w as indicative of the c lo s e n e s s of an
obtained ord er to the true o rd er. A nother exam in ation w as m ade
of the 27 runs for which the peaks of t w e r e lo cated . The peak
8 t n
2
of GOF^ w a s d eterm in ed in a s im ila r m an n er. A nother \ te st
w as p erform ed to d eterm in e if the peak of GOF^ overlapp ed that
for t or o ccu rred equally as often b efore and a fte r the peak of
S I a
2
T , • X = 11* 6, d. f. = 2, was sig n ifica n t for p< .0 1 . About the
st^
sa m e num ber of peaks of G O F , overlapp ed th ose of t a s w ere
J S t - j
2
captured by SR -T 5 in the p reviou s \ te s t. GOF^ could have
spanned s e v e r a l c y c le s , though, w hile SR -T 5 applied to only one
c y c le on each run.
TA BLE 5
C om paring the O ccu rren ce of P ea k s of GOF^ with T hose of
B efo re During A fter
7 17 3
The K(2) x P(2) a n a ly sis of varian ce r e v ea le d no sign ifican t
e ffe c ts involving pow ering excep t for P V . PI w as not included b e
c a u se v ery little in tra n sitiv ity o cc u r re d at E = . 20. The in c r e a s e
in pow ering produced an im p rovem en t in P V , e s p e c ia lly . The s ig -
36
TA BLE 6
E ffects of F urther P ow erin g
t s , 3
PV G OF3 JE
F ocu s Size (F) 1 4. 26 13. 20* 12. 88* 8 5 .0 9
P ow erin g (P) 1 1. 39 1 9 .2 8 * . 30 . 02
K X P 1 1. 39 8. 08 * 3. 73 1. 50
Reps 8
*p<. 01
^ p < . 05
nificant m ain effects for focus s iz e on GOF^ and JE reaffirm ed
those in the previou s a n a ly s is .
In the E(3) x S(2) d e sig n a ll the dependent m e a s u r e s had s ig
nificant e ffects for e r r o r ex cep t PI. In a ll c a s e s a s the e r r o r in
c r e a s e d , the dependent m e a s u r e d e c r e a s e d . In this a n a ly s is , both
Tgt and GOF^ w ere sign ifican tly im p roved by em p loying a known
sta rt. JE w as not affected by the m anipulation of the start.
TA BLE 7
S tart-U p s with P r io r Inform ation
Source df F
T s t 3
P V P I g o f 3 JE
E r r o r (E) 1 1 9 . 0 0 * 9 3 . 8 3 * 3 . 9 2 1 3 . 1 2 * 8. 33
S t a r t (S) 1 5. 9 8 * . 0 5 1 . 3 4 6. 3 8 * 2. 08
E x S 1 . 0 5 . 2 6 . 8 0 . 9 7 . 2 9
Reps 8
*p < . 01, ^p < . 05
37
C om p arison of C om plete Judgm ents to Interactive A pproach
One standard of c o m p a riso n for IRIS w as the com p lete judg
m en ts approach to r e co v e r in g an o r d e r . Undoubtedly, subjects w ill
not be asked to provide the 1, 176 judgm ents required for 49 stim u li.
If the Monte C arlo sim u la tio n of £^'s judgm ents is reason ab ly valid,
som e idea can be gained of what a com p lete judgm ents solutions
m ight be like. F or p u rp o ses of com p arin g com p lete to in com plete
d e sig n s, only the le v e l of e r r o r w as exp ected to co n sisten tly a f
fect the com p lete judgm ents so lu tio n s. In the E x K x N d esign
there w as an effect on t . for the N x K in teraction . The c o m -
st^
plexity of this in tera ctio n s e e m e d unsuited to the p u rp oses of this
com p arison . H en ce, the la r g e r m ain e ffe c t, focus s iz e , was e m
ployed here to d istin gu ish w h eth er the le v e ls of K d iffered s y s t e m
a tic a lly from each oth er as w e ll a s fro m the com p lete judgm ents
s olution.
Both le v e l of in tr a n sitiv ity and c o rresp o n d en ce to the true
order a re depicted . 1 F ig u r e 3. On both m e a s u r e s the com p lete
judgm ents solu tio.. w a s highly lin e a r , d e c r e a sin g on t and in
c rea sin g on PIan fro m low to high e r r o r .
The in te r a c tiv e approach did not r e c o v e r the true order as
w e ll and perform ed m o r e poorly as the e r r o r in c r e a se d . S m aller
fo c u se s yielded c o n siste n tly lo w er r .. The d ifferen ce w as not
S u
sign ifican t in the a n a ly sis of v a ria n ce, though. The in tra n sitiv ity
in the solu tions for K = 3 w as quite s im ila r to the com p lete judg
m en ts solutions e x c ep t at E = . 60. The r e la tiv e ly fa s te r r is e in
38
✓
J
l -
39
in tra n sitiv ity was re flec te d in a r e la tiv e ly fa ste r d e c r e a s e in t
S t
for K = 3. The m o st puzzling finding was that fo cu se s of K = 6
yielded slig h tly better solu tions from m o re in tran sitive ad jacen cy
m a tr ic e s .
C o m p arison of Interactive to N on -Interactive
Incom plete judgm ent d esig n s a r e needed when the num ber of
judgm ents is far in e x c e s s of what one subject can be exp ected to
provide. F or N = 2 5 the com p lete pair co m p a riso n m ethod req u ires
300 judgm en ts. Since Ss m ight be expected to provide up to this
m any judgm en ts, only the in te r a c tiv e r e su lts for N = 49 w ere c o m
pared to a n o n -in tera ctiv e approach.
The a n a ly sis of varian ce com p ared the in tera ctiv e resu lts
with in com p lete judgm ents data generated for c y c lic s e ts of n r /2
p a ir s. The per cen ts of judgm ents (J) w ere se le c te d to be as c lo s e
a s p o ssib le a c r o s s m ethods (M). The 25% and 33^% le v e ls c o r r e s
ponded to K = 3 and K = 6 in the a n a ly sis of the in tera ctiv e m ethod.
The sa m e three e r r o r le v e ls and judgm ent sim u lation m od el w ere
u se d to c r ea te the n o n -in tera ctiv e adjacen cy m a trix , V. Only one
2 3
c y c le of pow ering w as p erform ed to obtain V and V . T h ese three
ad jacen cy m a tr ic e s corresp on d ed to the three pow ers of V o c c u r
ring on the c y c le s e le c te d accord in g to SR -T 5 for the in tera ctiv e
data. The com p lete d e sig n w as: J(2) X E(3) X M(2) x P(3) in which
P w as treated a s a repeated m e a su r e w ithin r e p lic a tio n s. The
40
TABLE 8
C om paring the Two A pproaches:
Interactive v s. N on-Interactive
Source d. f. F
B etw een Reps
% Judgs. (J) 1 8. 76
E r r o r (E)
(2)
L in ear 1 31. 00
Q uadratic 1 1. 38
Method (M) 1 .0 1
J x E 2 3. 94
J x M 1 .4 2
E x M 2 . 25
J x E x M 2 1. 76
Reps within JEM 24 -
Within Reps
P o w er (P) 2 3 8 2 .1 6
J X P 2 2. 61
E x P 4 40. 9 8 :
M x P 2 5 0 .2 6 1
J x E x P 4 1. 06
J x M x P 2 17. 80:
E x M x P 4 . 30
J x Ex M x P 4 1 0 .8 0 :
R X P within JEM 48
-
*p < . 01
^p < . 05
41
d ep en d en t m e a s u r e w as t
The sign ifican t effects for E and J w e r e in the e x p ected d i
r e c tio n s . M ore judgm ents im proved t j w hile higher e r r o r r e
duced the corresp on d en ce with the tru e order. B etw een rep lica tio n s
th e r e w e r e no sign ificant e ffects involving m eth od . Within r e p li
ca tio n s th ere was a very la rg e effect for pow ering ir r e s p e c tiv e of
m ethod. P ow ering V appeared to im p rove t t w hether it w as ap
p lied in te r a c tiv e ly or n o n -in tera ctiv ely . The E x P in teraction w as
sig n ifica n t. As E in c re a se d , pow erings of V m ad e s m a lle r im
p ro v em en ts in t .. At E = . 60 t . d eclined fr o m P = 2 to P = 3.
SC sc
T hree in teraction s involving m ethod d iffe r en ce s w e r e s ig n ifi
can t. The in teraction betw een m ethod and p ow ering illu str a te d in
F ig u r e 4 r e v e a le d T . w as the h ig h e s t for the in te r a c tiv e ap p ro a ch
s c
w h en P = 3 but Tgt w as h igh est for the n on -in teractive m ethod when
P = 2. In both c a s e s pow ering the judgm ent m a tr ix V, im p roved
T . co n sid erab ly. Inspection of the J X M x P c e l l m eans pointed to
s c
a drop fro m the secon d to third pow er for a ll 33^% judgm ent con
d itio n s. The 25% judgm ent condition for the n on -in teractive m ethod
a l s o declin ed on t from P = 2 to P = 3. The four way in te r a c tio n
s c
w a s sign ifican t but w as not in terp reted . In v ie w of these M x P in
te r a c tio n s a second a n a ly sis of varian ce was p erform ed to m atch
on ly the b e s t pow erings of each m ethod. The pow ering variab le
w as dropped leaving: J(2) X E(3) xM (2). In this an alysis on ly two
e ffe c ts w e r e significant. The e r ro r le v e ls a ffe c te d Tg^ . a s in p r e
v iou s a n a ly s e s . The sign ificant J X M in tera ctio n depicted in F ig -
42
43
44
u re 5 in d icates the 25% judgm ents condition for the in tera ctiv e
TABLE 9
B e st P o w e r s of Interactive v s . N on -Interactive
Source d. f. F
% Judgs. (J) 1 1. 26
E r r o r (E) 2 6 2 .4 2 *
M ethod (M) 1 2. 34
Jx E 2 2 .4 7
J x M 1 5. 70*
E x M 2 2. 33
Jx E X M 2 0. 39
Reps 24
*p < . 01
*p < .0 5
m ethod w as su p e r io r . The ste p w ise N ew m an-K eu ls p rocedure
(W iner, 1962) w as follow ed in testin g the d iffe r en ce s am ong the c e ll
m e a n s. The four c e ll m ea n s w ere placed in d escen din g order:
J j M j, the m ean for the in tera ctiv e m ethod with the focus s iz e of
th ree (m ean JE = 25. 1%), the n o n -in teractive m ethod e m p lo y
ing o n e-th ird of a ll judgm ents; j £ M j , the in te r a c tiv e m ethod for
which K = 6 and the m ean JE = 34.7%, and J jM£» the n o n -in teractive
m ethod for 25% of a ll judgm en ts. P a ir w ise d ifferen ces w ere tested
for p < . 0 5 . Only the f ir s t te s t yielded sign ificant r e su lts, indicating
that the in te r a c tiv e m ethod p erform ed better than the n o n -in teractive
with few er ju d gem en ts. The other three t e s t s which w ere conducted
did not indicate th ere w ere sign ifican t d iffe r e n c e s betw een the other
m ea n s.
CH APTER V — DISCUSSION
Pow ering in com p lete data m a tr ic e s was a ssu m e d to be helpful
in recoverin g com p lete o rd ers from in com p lete data. The s ig n ifi
cant m ain e ffe c t for P in the J xExMx P a n a ly sis of varian ce
d em onstrated that one or m o re pow erin gs im p r o v e s Tg^. One p o
wering im proved r e c o v e r y of the true o rd er for both the in teractive
and n on -in teractive ap p roach es. The M x P in tera ctio n graphically
dem onstrated that the n o n -in tera ctiv e approach yield ed p o o rer r e -
3
suits from a second pow ering to obtain V . The J x M x P in te r
action narrow ed the u s e fu ln e ss of the secon d pow ering to only the
low er le v e l of judgm ents for the in tera ctiv e approach. This le v e l
of in co m p leten ess of the data w as obtained by gathering judgm ents
to fill fo c u se s of th ree stim u li.
The pow ering variab le w as in c r e a s e d to three p ow erin gs, P = 4 >
in the K x P a n a ly sis of varian ce. If the s m a lle r focus s iz e w ere to
benefit from further pow ering, a sign ifican t in tera ctio n would be
likely. None occu rred and an in sp ectio n of the c e ll m ean s indicated
T was a d v e r s e ly affected by the additional pow ering. A s som e
lev e l of e r r o r is exp ected to o ccu r in the judgm en ts, extending the
judgm ents accord in g to the tra n sitiv ity property even u tally resu lts
in i Pj and jPi» When the extent of th ese sy m m e tr ic connections
hinders r e c o v e r y of the true o r d e r , the in te r a c tiv e approach should
sw itch from pow ering to seek ing additional judgm en ts. T h ese r e -
46
47
suits £• c V >e K x P d esig n and the co m p a riso n s with the non-
int a c tiv e . pproach indicated two pow erings a r e optim al for filling
s m a ll fo c u se s w hen N = 49. F or larger fo c u se s one pow ering m ight
be su fficien t.
Two c r ite r ia w ere u sed to evalu ate the in tera ctiv e approach.
The le v e l of r e co v e r y of the true o rd er attained by each solution
was b ased on t The le v e l of in c o m p le te n e ss of the data w as d e
term in ed by the extent to which judgm ents w e r e required to obtain
a solution. The fir st c r ite r io n c h a r a c te r iz e d the e ffe c tiv e n e ss of
IRIS w hile the secon d depicted the e fficie n c y of obtaining solu tion s.
The in tera ctiv e approach w as divided into four m ethods accord ing
to the c e lls of the sign ifican t N x K in tera ctio n s on t and JE.
F o r 49 stim u li the focus s iz e of s ix stim u li, requiring 15
judgm ents per c y c le , w as c le a r ly w astefu l of the data gathered. A
sligh t drop in t ^ o ccu rred in the N x K in tera ctio n . Up to 555
judgm ents w ere required to obtain what proved to be solu tions with
s lig h tly low er c o rresp o n d en ce to the true o r d e r than for K = 3. The
n o n -in tera ctiv e approach p erfo rm ed a s w e ll a s K = 6 for one-th ird
of a ll judgm ents and n early as w e ll for on e-fou rth .
L arge fo c u se s yielded h igh er T gt for N = 25 than s m a ll fo
c u s e s . H ow ever, e fficie n c y w as poor a s about tw o-th ird s of all
judgm ents w ere required. While no a n a ly se s betw een the in tera ctiv e
and n o n -in tera ctiv e approaches w e r e p erform ed for N = 25, it
s e e m e d lik e ly that com p arab le r e su lts would be obtained u sin g the
c y c lic d e sig n proced u re for e ith e r le v e l of N. Three runs w ere
48
exam ined for N = 25 at E = . 40 with d e sig n v e c to r s con stru cted as
before to s e le c t e v e r y third s e t s iz e beginning with gj = 1. F o r one-
third of a ll judgm ents the m ea n Tgt was . 81 based on one pow ering
of V. The m ean Tgt for a ll runs of K = 6 w as . 84. Obtaining half
a s m any judgm ents with a n o n -in tera ctiv e approach should be e x
pected to yield low er t t but this co m p a riso n indicated no great
d ifferen ce. The higher le v e l of Tgj. for K = 6 w as s till not nearly
a s high a s for all judgm en ts, t = . 91.
s t
It se em e d the la rg e focu s m ethod did not p erfo rm w e ll enough
to recom m en d retaining it. The in c r e a s e in judgm ents w ere not
com p en sated for with notably higher t . One a ltern ative to filling
fo c u se s with judgm ents would be to pow er focus m a tr ic e s . In this
way few er judgm ents would be required and the c o st of pow ering
would s till be reduced. O ther a lte r n a tiv e s that m ight be con sid ered
are using c y c lic d e sig n s for focus m a tr ic e s along with other m eth
ods of com pleting the connections in focus m a tr ic e s b e sid e s pow er
ing. A m ethod that is equivalent to infinite pow ering of adjacen cy
m a tr ic e s can be im p lem en ted v ery in e x p e n siv e ly (Kehoe, p erson al
com m unication).
F o r N = 25 the sm a ll focus m ethod w as not p a rticu la rly s u c
c e s s fu l, eith er. A p p roxim ately 42% of a ll judgm ents w ere needed
to obtain solu tions averagin g T t = .7 8 . U sing the th ree runs of the
n o n -in tera ctiv e approach at E = . 40 as a co m p a riso n , few er judg
m en ts (33^%) yielded better solu tion s with T .8 1 . The c e ll
m ea n s a c r o s s e r r o r le v e ls rev ea led K = 3 w as p a rticu la rly v u ln e r
49
able to high e r r o r . R e su lts for the higher e r r o r le v e ls w ere s ig n i
ficantly im proved when p rio r inform ation about the ordering w as
u se d to form sta r t-u p p a ir s . Som e a s s is ta n c e such as this would
be needed before this m ethod would be recom m en ded for gathering
r ea l data.
The m o st s u c c e s s fu l in teractive m ethod w as filling sm a ll f o
c u s e s to r e c o v e r o r d e r s for 49 stim u li. This approach was s ig n i
ficantly better than the n o n -in tera ctiv e approach which a lso required
25% of a ll judgm ents. It w as at le a s t a s good a s gathering 33§% of
a ll judgm ents, then pow ering V once. Some im p rovem en ts would
be n e c e s s a r y to r e c o v e r the true ord er a lm o st a s w ell a s a c o m
p lete judgm ents p ro ced u re. The a v era g e t . for a ll le v e ls of e r -
S t
ror w as .8 3 w hile co m p lete data yielded solu tions averagin g t ^ =
.9 1 .
F urther in v estig a tio n of the p erform an ce of the in tera ctiv e
approach is needed to d eterm in e optim al solu tions for real data.
F our c a te g o r ie s of in d ice s w ere evaluated in this study and c o m
pared to T No one index of p erform an ce proved su p erior in in
dicating w hen r e c o v e r y of the true ord er w as h ig h est. PV behaved
m o s t s im ila r ly to t t for m anipulations of E, K, and N. The c o r
relation of th e se two in d ic e s was . 73 over a ll 48 runs. PV w as not
sig n ifica n tly affected by the N x K in teraction a s w as t . and w as
OI
not m o st s im ila r ly a ffected in the other a n a ly ses of v a r ia n ce . GOF^
2
w as m o re s im ila r ly a ffe c te d in th ese other a n a ly s e s . A \ te s t
d em on strated that the peaks of GOF^ tended to ov erla p the peaks of
50
T .. G O F , w a s not a s highly related to T as PV , though, and
St j St^ • —
did no better in pinpointing peaks of r in te r a c tiv e ly , than the
stopping rule, S R -T 5 .
Throughout the a n a ly se s of the in tera ctiv e data it w as c le a r
that in te r a c tiv e solu tion s a r e not required to d eterm in e a ll the c o n
n ection s am ong the stim u li. The le v e l of fill, P F R , did not appear
to lo ca te the m o s t peaks of t .. The m ean t . for P F R = 100 w as
SC St
.8 3 5 , slig h tly lo w e r than for S R -P . S R -P located a s m any peaks
as SR -T 5 but did not indicate a stop until a fter m any peaks had
ended. Since P F R = 100 o ccu rred at the sa m e tim e or a fte r S R -P
was a ctivated , pow ering to fill V can not be exp ected to yield higher
C H A P T E R V I - C O N C L U S I O N S
P ow ering ad jacen cy m a tr ic e s formed eith er in te r a c tiv e ly or
n o n -in te r a c tiv e ly adds con n ection s which im p r o v e s the c o r r e s p o n
d en ce of the o rd er of the s c o r e vector to that of the true s c a le
v a lu e s. N o n -in ter a c tiv e so lu tio n s yielded h ig h er tau 's for one
pow ering w h ile two p ow erin gs benefited the in te r a c tiv e approach
u sin g sm a ll focus s iz e s .
F illin g la r g e r fo c u se s with judgments proved in efficien t and
in e ffe ctiv e , e s p e c ia lly w hen 49 stim uli w ere ord ered . A lte r n a
tiv e s to obtaining a ll p o s s ib le focus judgm ents w ere con sid ered
such as pow ering focus m a t r ic e s .
The in te r a c tiv e a p p r o a c h w a s m ost s u c c e s s f u l w hen 49 s tim
u li w ere s c a le d by fillin g s m a ll fo c u se s. T h is m eth od of o r d e r in g
25 stim u li w a s not a s e ff e c tiv e o r efficien t in r e c o v e r in g th e tru e
o r d e r as p ilo t ru n s in d ic a te d th e n o n -in te r a c tiv e a p p ro a ch w ou ld b e .
F urther stu d ies w ill be needed to find in d icators of good r e
c o v ery of the true o rd er for r e a l data. The lev el of a sy m m e tr y in
the pow ered a d ja cen cy m a tr ix and a m easure of p red ictive validity
b ased on additional ju d gm en ts w e r e the m o s t p rom isin g of the in
d ic e s in v estig a ted . O ptim al interactive solu tion s need not be based
on fully d e ter m in ed a d ja c en cy m atrices.
51
R E FER EN C ES
B u sa c k e r , R. G. and Saaty, T. L. F inite graphs and n etw ork s.
New York: M cG raw -H ill, 1965.
C liff, N. C om plete o r d e r s from in com p lete data, P sy c h o lo g ic a l
B u lletin , in p r e s s .
C liff, N. and R eynolds, T. A c o m p u te r -in te r a ctiv e ordering p r o
gram for p a irw ise p r e fe r e n c e judgm en ts. P ap er p resen ted
at the m eetin g of the P sy c h o m e tr ic S ociety, P alo A lto, C alif. ,
M arch 1974.
H arary, F . , N orm an, R. , and C artw right, D. Structural m o d e ls :
an introduction to the th eory of d irected grap h s. New York:
W iley, 1963.
O 'N eil, P . E. and O 'N eil, E. J. A fa st expected tim e algorith m
for B oolean m a trix m u ltip lication and tra n sitiv e c lo su r e ,
Inform ation and C on trol, 1973, ZZ, 132-138.
Spence, I. and D om oney, D. W. Single subject in com p lete d esign s
for n onm etric m u ltid im en sio n a l sc a lin g , P sy c h o m etr ik a , in
p r e s s .
T o rg erso n , W. S. T h eory and m ethods of sc a lin g . New York:
W iley, 1958.
W iner, B . J. S ta tistica l p rin cip les in ex p erim en ta l d e sig n . New
York, M cG raw -H ill, 1962.
52
A PPE N D IX
53
ACRONYMS
IRIS Interactive R an k-O rders for Individual Subjects
SR -M Stopping Rule for M onotonicity
SR -T 3 Stopping Rule for Trends of T hree
SR -T 5 Stopping Rule for T rends of F ive
S R -P Stopping Rule for Pow ering
PI P e r Cent of Intransitivity
GOF G oodness - o f - F ill
JE Judgm ent E fficien cy
P F R P e r Cent of F ille d R elations
RJI R e v e r s a l of Judgm ents Index
PV P r e d ic tiv e Validity
Rnd
Kwn
Note:
TABLE A
Sum m ary of R esults
I. V .'s Interactive
P E K(J) N
Tst3
P V f PI GOI
3 . 20 3 (i) 25 88 96 . 7 89
49
88 89 . 1 87
6(£)
25 91
93 .6 95
49 89
96 1. 5 91
.4 0
3(i)
25 80 90 2. 2 90
49
84 81 2 .4 88
6 (i) 25 86 91 8 .4 80
49
80 87 6 .5 83
. 60
3(i)
25 66 74 7 .0 83
49 76 80 7. 5 80
6(i)
25 76 83 16. 9 72
49 77 81 9 .8 80
4 . 20 3 49 86 97 . 0 81
6 49 89
97 .4 94
3 .4 0 3 25 87 91 1. 8 92
.6 0 3 25 74 73 3 .6 88
M ean of 3 runs for each com bination of I. V .'s.
Com plete
Jdgs
Non-
Int.
JE
t
Tall
p I
all
T
St.
46 95 . 7
25 94 .6 85
77 93 .4
41 94 . 7 89
42
89 3. 0
28 91 2. 5
79
65 93 2. 3
32
91 3 .0 80
37 87 4. 7
22 89 5 .4 76
60 86 4. 5
31 89 5. 3 79
23 95 . 7
43 95 .7
45 91 1 . 8
39
83
4 .9
^D ecim al points w ere om itted.

Linked assets

University of Southern California Dissertations and Theses

Conceptually similar

PDF

A Monte Carlo Evaluation Of Interactive Multidimensional Scaling

PDF

A Factor Analysis Of The Semantic-Evaluation Abilities

PDF

A Factor Analysis Of The Symbolic-Evaluation Abilities

PDF

A Multidimensional Scaling Of Mood Expressions

PDF

Non-Specific Treatment Factors And Deconditioning In Fear Reduction

PDF

Intellectual And Cognitive Factors In The Production Of Psychological Stress Reactions

PDF

Factorial Stability As A Function Of Analytical Rotational Method, Type Of Simple Structure, And Size Of Sample

PDF

A behavioral -analytic model for assessing social competence in male batterers

PDF

Personality Variables And Intellectual Abilities As Determinants Of Concept Learning

PDF

A Rotational Approach To Psychological Invariance

PDF

The Subjective Utility Function As An Estimator Of Optimal Test Weights For Personnel Selection

PDF

The Relationship Of Multidimensional Scaling Spaces Of Trait Adjectives For Different Reference Persons

PDF

Decision Making: An Individually Parameterized Deterministic Model

PDF

Experimenter Expectancy Effect Examined As A Function Of Task Ambiguity And Internal Versus External Control Of Reinforcement

PDF

A Temporal Approach-Avoidance Conflict In An Academic Test Situation

PDF

Age, Sex, And Task Difficulty As Predictors Of Social Conformity: A Search For General Tendencies Of Conformity Behavior

PDF

The Effects Of Justice, Balance, And Hostility On Mirth

PDF

The Relationship Of Teacher Empathy And Student Personality To Academic Achievement And Course Evaluation

PDF

Dissonance and Self-Perception Analysis of "Forced Compliance": When Two Theories Make Competing Predictions

PDF

The Role Of Intellectual Abilities In Concept Learning

Asset Metadata

Creator Green, Rex Stanley (author)

Core Title A Monte Carlo Evaluation Of A Computer-Interactive Extended Transitivity Dominance Scaling Model

Degree Doctor of Philosophy

Degree Program Psychology

Publisher University of Southern California (original), University of Southern California. Libraries (digital)

Tag OAI-PMH Harvest,psychology, general

Language English

Contributor Digitized by ProQuest (provenance)

Advisor Cliff, Norman (committee chair), Gutman, Jonathan (committee member), Jellison, Jerald M. (committee member)

Permanent Link (DOI) https://doi.org/10.25549/usctheses-c18-747558

Unique identifier UC11356794

Identifier 7506417.pdf (filename),usctheses-c18-747558 (legacy record id)

Legacy Identifier 7506417.pdf

Dmrecord 747558

Document Type Dissertation

Rights Green, Rex Stanley

Type texts

Source University of Southern California (contributing entity), University of Southern California Dissertations and Theses (collection)

Access Conditions The author retains rights to his/her dissertation, thesis or other graduate work according to U.S. copyright law. Electronic access is being provided by the USC Libraries in agreement with the au...

Repository Name University of Southern California Digital Library

Repository Location USC Digital Library, University of Southern California, University Park Campus, Los Angeles, California 90089, USA