University of Southern California Dissertations and Theses

Validity Concomitants Of Various Scoring Procedures Which Attenuate The Effects Of Response Sets And Chance

(USC Thesis Other)

Validity Concomitants Of Various Scoring Procedures Which Attenuate The Effects Of Response Sets And Chance

PDF

Download a page range

Download transcript

Copy asset link

Request this asset

Transcript (if available)

Content T his d isser ta tio n has been Mic 61— 2532 m icr o film ed exa ctly as r ec eiv ed HOPKINS, Kenneth Dean. VALIDITY CON COMITANTS OF VARIOUS SCORING PRO CEDURES WHICH ATTENUATE THE E FFEC TS OF RESPONSE SETS AND CHANCE. U n iversity of Southern C alifornia, P h.D ., 1961 Education, psychology University Microfilms, Inc., Ann Arbor, Michigan VALIDITY CONCOMITANTS OF VARIOUS SCORING PROCEDURES WHICH ATTENUATE THE EFFECTS OF RESPONSE SETS AND CHANCE by K e n n e th Dean H o p k in s A D i s s e r t a t i o n P r e s e n t e d t o t h e FACULTY OF THE GRADUATE SCHOOL UNIVERSITY OF SOUTHERN CALIFORNIA In P a r t i a l F u l f i l l m e n t o f t h e R e q u i r e m e n t s f o r th e D egree DOCTOR OF PHILOSOPHY ( E d u c a t i o n ) J a n u a r y 1961 UNIVERSITY O F SOUTHERN CALIFORNIA G RA D U A TE SC H O O L U N IV ER SIT Y PA RK L O S A N G E L E S 7 . C A L IFO R N IA jP /h f R il + * # f/stt AtJ under the direction of hd£. .Dissertation Com mittee, and approved by all its members, has been presented to and accepted by the Dean of the Graduate School, tn partial fulfillm ent of requirements fo r the detjree of D O C T O R OF P H I L O S O P H Y D tan Date. January, 1961 ACKN 0 WLE DGME NTS The w r i t e r would l i k e to e x p r e s s a p p r e c i a t i o n to t h e S c i e n c e R e s e a r c h A s s o c i a t e s , I n c . and t o t h e C a l i f o r n i a T e s t B u re a u f o r t h e i r k i n d p e r m i s s i o n to r e p r o d u c e t h e n e c e s s a r y r e c a l l fo rm s of t h e t e s t s r e l a t i n g to t h i s s t u d y ; to t h e H udson S c h o o l D i s t r i c t and t h e r e s p e c t i v e p r i n c i p a l s , P ra n k Thompson, W i l l i a m L o n g e n e c k e r , M alcolm A c r e e , an d Howard D a v is f o r p r o v i d i n g t h e n e c e s s a r y s u b j e c t s ; amJ t o th e W e s t e r n D ata P r o c e s s i n g C e n t e r f o r th e p r o c e s s i n g o f th e d a t a . The w r i t e r i s much in d eb te d to P h i l l i p S u th er la n d , Robert I h in g e r , and 0. A. D ld r id g e , and to ray b r o t h e r s , Bob and Dale, for 1 h eir encouragement, a d v i c e , and a s s i s t ance in the v a r i o u s phases of t ! is s tu d y . The w r i t e r i s e s p e c i a l l y g r a t e f u l to h is v i f e , C o lle e n , fo r her p a t ie n c e , u n d e r sta n d in g , and s u p p o r t . TABLE OF CONTENTS Page ACKNOWLEDGMENTS.................................................................................................. i i LIST OF TABLES .................................................................................................. V C h a p t e r I . INTRODUCTION ....................................................................................... I The P r o b le m H y p o t h e s e s D e l i m i t a t i o n s O r g a n i z a t i o n o f t h e R e m a in in g C h a p t e r s I I . REVIEW OF TNE L I T E R A T U R E .................................................... 8 The n a t u r e o f r e s p o n s e s e t s G u e s s i n g The s p e e d v e r s u s a c c u r a c y s e t P o s i t i o n a l p r e f e r e n c e s The e f f e c t s o f d i r e c t i o n s S c o r i n g f o r m u l a s Summary I I I . METHODOLOGY............................................................................................ 29 The s a m p le The e v a l u a t i v e i n s t r u m e n t s The c r i t e r i o n A d m i n i s t r a t i o n p r o c e d u r e s S c o r i n g S i g n i f i c a n c e l e v e l IV. F I N D I N G S .................................................................................................. 40 H y p o t h e s i s I H y p o t h e s i s I I H y p o t h e s i s I I I H y p o t h e s i s IV H y p o t h e s i s V H y p o t h e s i s VI H y p o t h e s i s V II Sex d i f f e r e n c e s Summary C h a p t e r P ag e V. SUMMARY AND CONCLUSIONS, AND RECOMMEND ATIONS ............................................................................................ 80 Summary and C o n c l u s i o n s R e c o m m e n d a tio n s BIBLIOGRAPHY ........................................................................................................ 89 iv LIST OF TABLES T a b le Page 1. The e f f e c t s o f g u e s s v e r s u s d o - n o t - g u e s s d i r e c t i o n s ............................................................................................. 23 2 . Summary o f r e s e a r c h w i t h t h e c o n v e n t i o n a l c o r r e c t i o n f o r m u l a ........................................................................ 25 3 . P r e v i o u s a c h i e v e m e n t and a p t i t u d e t e s t r e s u l t s a t t h e f o u r e x p e r i m e n t a l s c h o o l s . . 30 4 . A c o m p a r i s o n of t h e a r i t h m e t i c s e c t i o n s o f t h e CAT and SRA, e l e m e n t a r y b a t t e r i e s . . 32 5 . I n t e r c o r r e l a t t o n s among t h e CAT and SRA A r i t h m e t i c T e s t s .................................................................... 33 6 . An e x a m i n a t i o n o f t h e i n f l u e n c e o f s p e e d on t h e CAT and SRA A r i t h m e t i c T e s t s . . . 41 7 . R i s k - a c c e p t a n c e d a t a f o r t h e CAT and SRA A r i t h m e t i c T e s t s ................................................................... 43 8 . The number o f r e s p o n s e s keyed c o r r e c t a t e a c h o p t i o n p o s i t i o n f o r t h e C a l i f o r n i a A c h ie v e m e n t T e s t s and t h e SRA A c h ie v e m e n t S e r i e s .............................................................. 47 9 . O b s e rv e d f r e q u e n c y o f i n t r a i n d i v i d u a l p o s i t i o n a l p r e f e r e n c e r e s p o n s e s e t s a t v a r i o u s l e v e l ? o f c o n f i d e n c e .......................... 51 10. O b s e rv e d and e x p e c t e d f r e q u e n c i e s w i t h w h ich v a r i o u s r e s p o n s e p o s i t i o n s w ere c h o s e n 51 T a b l e Page 1 1 . C o r r e l a t i o n m a t r i x o f s c o r e s y i e l d e d by v a r i o u s s c o r i n g m e th o d s on th e CAT A r i t h m e t i c T e s t s , w i t h means and s t a n d a r d d e v i a t i o n s .......................................................... 53 1 2 . V a l i d i t y and r e l i a b i l i t y c o e f f i c i e n t s and i n t e r c o r r e l a t i o n s y i e l d e d by v a r i o u s s c o r in g , p r o c e d u r e s o n t h e C A T .......... 54 1 3 . C o r r e l a t i o n m a t r i x o f s c o r e s y i e l d e d by v a r i o u s s c o r i n g p r o c e d u r e s on t h e SRA A c h ie v e m e n t S e r i e s , a r i t h m e t i c c o m p u ta t i o n , w i t h means and s t a n d a r d d e v i a t i o n s . 59 14. V a l i d i t y and r e l i a b i l i t y c o e f f i c i e n t s y i e l d e d by v a r i o u s s c o r i n g p r o c e d u r e s on t h e SRA A r i t h m e t i c C o m m u tatio n T e s t . . 60 15. V a l i d i t y , r e l i a b i l i t y , and i n t e r c o r r e l a t i o n s y i e l d e d by v a r i o u s s c o r i n g procedures on the SUA A r it h m e t ic R easoning Test . . . . 62 16. k e c a l ! and r e c o g n i t i o n mean s c o r e s on a l t e r n a t e forms o f the CAT and S1L A A r i t h m etic T e s t s .............................................................. 65 17. V a l i d i t y o f c h a n c e - l e v e l s c o r e s on the SRA at the _.0 5 l e v e l , wit ' m d w ith o u t c o r r e c t i o n l o r r a n g e r e s t r i c t i o n ..... 69 v i T a b le Page 18. V a l i d i t y o f c h a n c e - l e v e l s c o r e s a t t h e .01 l e v e l , w i t h a n d w i t h o u t c o r r e c t i o n f o r r a n g e r e s t r i c t i o n ................................................... 71 19. Sex d i f f e r e n c e s in a c h i e v e m e n t , t i m e , v a l i d i t y , and i n t h e g a m b lin g and p o s i t i o n a l p r e f e r e n c e r e s p o n s e s e t s . . . 77 v i i CHAPTER I INTRODUCTION A c c u r a t e m e a s u re m e n t o f e d u c a t i o n a l a c h i e v e m e n t has l o n g b e e n a g o a l i n e d u c a t i o n a n d p s y c h o l o g y . A w a re n e s s o f some o f t h e p r o b le m s i n t e s t s w h ic h a r e p r e s e n t e d i n m u l t i p l e c h o i c e fo rm h a s e x i s t e d s i n c e t h e i r d e b u t e a r l y i n t h e c e n t u r y . One o f I h e s e p r o b le m s h a s r e s u l t e d from t h e f a c t t h a t t h e t e s t i t e m s c a n be a n s w e re d c o r r e c t l y by a p e r s o n w i t h no k n o w le d g e i n t h e f i e l d b e i n g t e s t e d . T e s t s p e c i a l i s t s , a f t e r more t h a n t h r e e d e c a d e s o f e x p e r i e n c e w i t h o b j e c t i v e t e s t s , s t i l l h a v e d i s a g r e e d c o n c e r n i n g w h a t to do a b o u t t h e c h a n c e f a c t o r and g u e s s i n g ( 1 , 4 8 ) . The i d e n t i f i c a t i o n o f c e r t a i n r e s p o n s e s e t s h a s made t h e I n c r e a s e d i m p o r t a n c e o f d e a l i n g w i t h t h e p r o b le m s p r e s e n t e d by c h a n c e s u c c e s s on m u l t i p l e r e s p o n s e t e s t s a p p a r e n t . The P ro b le m I f t e s t s a r e to h a v e t h e i r maximum v a l i d i t y , e l e m e n ts w h ic h c a u s e d i s c r e p a n c i e s b e tw e e n o b t a i n e d and t r u e s c o r e s s h o u l d be i d e n t i f i e d and c o r r e c t i v e m e a s u r e s s o u g h t . The p u r p o s e o f t h i s s t u d y was t o d e t e r m i n e w h e t h e r c e r t a i n r e s p o n s e s e t s w ere p r e s e n t i n e l e m e n t a r y s c h o o l p u p i l s and t o e x a m in e t h e e f f e c t s on t e s t v a l i d i t y when v a r i o u s s c o r in g m e th o d s w ere em p lo y ed w h ic h a t t e m p t e d to c o r r e c t f o r i r r e l e v a n t v a r i a n c e i n t r o d u c e d by s u c h s e t s and by c h a n c e . 2 The i m p o r t a n c e o f c o n s i d e r i n g t h e e f f e c t s o f c h a n c e and r e s p o n s e s e t s on s t a n d a r d i z e d t e s t s i s g r a p h i c a l l y d e m o n s t r a t e d w hen i t i s r e a l i z e d t h a t on t h e a r i t h m e t i c f u n d a m e n t a l s s e c t i o n s o f t h e c u r r e n t e l e m e n t a r y b a t t e r i e s ( 4 - b ) o f t h e C a l i f o r n i a A c h ie v e m e n t T e s t s and SRA A c h i e v e m e n t S e r i e s , c h a n c e g r a d e p l a c e m e n t s c o r e s a r e 4 . 4 and 4 . 3 r e s p e c t i v e l y . A l t h o u g h f o u r t h g r a d e u s a g e i s n o t recom m ended by t h e SRA, e x c e p t f o r s u p e r i o r c l a s s e s , u n t i l t h e s e c o n d s e m e s t e r , c h a n c e s c o r e s t i l l p l a c e s a p u p i l a t t h e t h i r t i e t h c e n t i l e a c c o r d i n g t o t h e s t a n d a r d i z a t i o n n o r m s . C o n c e r n i n g t h e s e c h a n c e - l e v e l s c o r e s , G u l l i k s e n s t a t e d : A s c o r e t h a t i s w i t h i n one o r two s t a n d a r d d e v i a t i o n s o f a c h a n c e s c o r e s h o u l d n o t be i n t e r p r e t e d a s s i g n i f y i n g a n y k n o w le d g e o f t h e s u b j e c t m a t t e r o f t h e e x a m i n a t i o n . ( 3 8 : 2 6 3 ) I f t h i s s u g g e s t i o n was f o l l o w e d , t h e s e two t e s t s w o u ld be i n a p p r o p r i a t e f o r t h e f o l l o w i n g p e r c e n t a g e s o f c a s e s a t e a c h g r a d e l e v e l a c c o r d i n g to t h e n o r m a t i v e d a t a ( 7 7 , 7 8 ) : 4 . 0 - 4 . 4 4 . 5 - 4 .9 5 . 0 - 5 . 4 6 . 0 - 6 . 4 CAT 9 0 $ 8 0 $ 6 0$ 20$ SRA 8 5 $ 6 0 $ 3 5 $ 18$ T h e se f i g u r e s show t h e p e r c e n t a g e o f e x a m i n e e s f o r whom t h e n u l l h y p o t h e s i s o f no k n o w le d g e o f c o n t e n t w o u ld n o t be r e j e c t e d a t t h e .01 l e v e l , a s s u m i n g o n l y c h a n c e i s o p e r a t i n g on e x a m i n e e s ' i n c o r r e c t r e s p o n s e s . T h e re w ould a p p e a r t o be o n l y t h r e e p l a u s i b l e e x p l a n a t i o n s : ( l ) t h e t e s t s a r e much t o o d i f f i c u l t , (2 ) t h e d i s t r a c t o r s a t t r a c t w i t h g r e a t e r t h a n c h a n c e p r o b a b i l i t y , ( 3 ) many e x a m in e e s do n o t a t t e m p t a l a r g e r n u m ber o f t h e i t e m s . A l l t h r e e o f t h e s e f a c t o r s a r e v u l n e r a b l e t o t h e i n f l u e n c e o f r e s p o n s e s e t s , w h i c h d e m o n s t r a t e s t h e n e e d f o r a n i n v e s t i g a t i o n o f p o s s i b l e c o r r e c t i v e m e a s u r e s . H y p o t h e s e s The f o l l o w i n g h y p o t h e s e s w e re e x a m in e d i n t h i s s t u d y . 1 . The s t a n d a r d i z e d t e s t s u s e d i n t h i s s t u d y a r e e s s e n t i a l l y pow er t e s t s . 2 . The s e t t o g am ble I s p r e s e n t i n e l e m e n t a r y s c h o o l p u p i l s . 3 . T h e re i s a p o s i t i v e r e l a t i o n s h i p b e tw e e n a b i l i t y and t e n d e n c y to f o l l o w d i r e c t i o n s . 4 . S y s t e m a t i c p o s i t i o n a l b i a s e s a r e p r e s e n t in t h e k e y i n g o f th e c o r r e c t r e s p o n s e s on I he s t a n d a r d i z e d a c h i e v e m e n t t e s t s u s e d in t h i s s t u d y . 5 . I n t r a i n d i v i d u a l p o s i t i o n a l p r e f e r e n c e r e s p o n s e s e t s a r e p r e s e n t i n e l e m e n t a r y s c h o o l p u p i l s . 6 . I n t e r i n d i v i d u a l p o s i t i o n a l p r e f e r e n c e r e s p o n s e s e t s e x i s t i n e l e m e n t a r y s c h o o l p u p i l s . Ta t h r o u g h 7 g . The v a l i d i t y c o e f f i c i e n t s o f s t a n d a r d i z e d a c h i e v e m e n t t e s t s y i e l d e d by t h e f o l l o w i n g s c o r i n g m e th o d s a r e g r e a t e r t h a n t h o s e o b t a i n e d u s i n g t h e c o n v e n t i o n a l num ber r i g h t s c o r e : a ) A t h e o r e t i c a l c h a n c e c o r r e c t i o n a p p l i e d * b ) An e m p i r i c a l ( m u l t i p l e r e g r e s s i o n ) s c o r i n g f o r m u l a e m p lo y e d . c ) S c o r e s o b t a i n e d u s i n g t h e f o r c e d - c h o i c e t y p e o f d i r e c t i o n s . d) The " 3 - 4 c u t o f f " s c o r e s . e ) The " 3 -3 c u t o f f " s c o r e s . f ) The " 4 - 4 c u t o f f " s c o r e s . g) The " c h a n c e t h r e s h o l d " s c o r e s . D e f i n i t i o n o f Terms F o r p u r p o s e s o f t h i s s t u d y , th e f o l l o w i n g d e n o t a t i o n s h av e a p p l i e d . G u e s s i n g o r g a m b l i n g . — The t e n d e n c y t o r e s p o n d to an i t e m when an a n s w e r c a n n o t be d e d u c e d w i t h c e r t a i n t y from t h e i n f o r m a t i o n p o s s e s s e d by t h e e x a m in e e i s r e f e r r e d to a s g u e s s i n g o r g a m b lin g i n t h i s s t u d y ( 3 3 ) . I n d e x o f r i s k - a c c e p t a n c e . — T h is i n d e x r e p r e s e n t s a p e r s o n ' s w i l l i n g n e s s t o g a m b le . T h e o r e t i c a l l y t h e i n d e x i n d i c a t e s t h e r a t i o o f t h e num ber o f i t e m s on w h ic h t h e s u b j e c t g u e s s e d to t h e t o t a l num ber o f i t e m s he d i d n o t know b u t upon w h ic h he c o u l d h av e g u e s s e d . R e s p o n s e s e t , r < s p o n s > s t y l g , o r r > s p o m « b i a s , — T hese terms are used i n t e r c h a n g e a b l y to i n d i c a t e a h a b i t c a u s i n g a s u b j e c t to e a r n a d i f f e r e n t s c o r e from what be would ea r n i f the same it em s were p r e s e n t e d in a d i f f e r e n t form ( 1 6 : 3 7 2 ) . P o s i t i o n a l p r e f e r e n c e . — The tendency to choose a l t e r n a t i v e s with a frequency s i g n i f i c a n t l y d i f f e r e n t from chance p r o b a b i l i t v is r e f e r r e d to as a p o s i t i o n a l p r e f e r ence re s p o n se s e t . Chance t h r e s h o l d . — This term s i g n i f i e s the p o i n t in a t e s t helow which an e x a m i n e e r e s p o n d s at no g r e a t e r than c'-ance s u c c e s s . Thus, a s u b j e c t ' s p I; nnc e - t ! r e s ho 1 d s c o r e is the number of o o r r e r t . l v answered items b e f o r e r e a c h i n g h i s cha n o e - 1hre s bo I d . 3 — 4 c u t o f f s c o r e . — This s c o r e is the pip-her of i i ems c o r r e c t l y answered b e f o r e a s u b j e c t i n c o r r e c t l y answered I h r e e in a s e o u e n c e of f o u r i t e m s . 3 —3 c u t o f f s c o r e . — T h is s c o r e is the number o f items c o r r e c t l y answered b e f o r e an exa m ine e re s p on d ed i n c o r r e c t l y to t h r e e c o n s e c u t i v e p r o b l e m s . t -1 c u t o f f s c o r e . — T h is s c o r e is the number o f c o r r e c t r e s p o n s e s (riven by an exam inee b e f o r e a n s w e r i n g i n c o r r e c t l y four c o ns ecu t i v e i t e m s . 6 C o i n c i d e n t v a l i d i t y . - - T h e te rm " c o i n c i d e n t v a l i d i t y " r e f e r s t o t h e d e g r e e t o w h i c h a t e s t i n c o n v e n t i o n a l m u l t i p l e r e s p o n s e f o rm c o r r e l a t e s w i t h a n a l t e r n a t e fo rm o f t h e sam e t e s t w i t h c h a n c e a n d r e s p o n s e s e t e f f e c t s m i n i m i z e d , i . e . , i n r e c a l l f o r m a t . C ro n b a c h s t a t e s : "A m a j o r i s s u e i n e d u c a t i o n a l t e s t i n g i s w h e t h e r r e c o g n i t i o n and r e c a l l t e s t s on t h e same c o n t e n t m e a s u r e t h e same a b i l i t y . " / H a v e c o i n c i d e n t v a l i d i t y ^ ( 1 6 : 3 7 3 ) . The ty p es o f t e s t s f o r w h ic h i t i s p o s s i b l e t o o b t a i n an i n d e x o f c o i n c i d e n t v a l i d i t y i s l i m i t e d p r i m a r i l y to a c h i e v e m e n t t e s t s , w h e r e t h e c o n t e n t i s a m e n a b le t o b o t h r e c o g n i t i o n and r e c a l l f o r m a t s . In t h i s s t u d y , t h e te r m " v a l i d i t y " r e f e r s t o c o i n c i d e n t v a l i d i t y u n l e s s o t h e r w i s e s p e c i f i e d . A l t e r n a t i v e " d o n * t know" o p t i o n . — T h i s t e r m s i g n i f i e s a s p e c i f i c r e s p o n s e o p t i o n t h a t an e x a m in e e i s d i r e c t e d to t a k e i f he c a n n o t w ork a p r o b le m , o r i f he t h i n k s n one o f t h e a n s w e r s g i v e n i s c o r r e c t . T h is t y p e o f o p t i o n i s fo u n d on t h e C a l i f o r n i a A c h ie v e m e n t T e s t s u s e d i n t h i s s t u d y . D e l i m i t a t i o n s The s c o p e o f t h i s s t u d y was l i m i t e d t o f i f t h g r a d e p u p i l s fro m f o u r e l e m e n t a r y s c h o o l s i n t h e Hudson S c h o o l D i s t r i c t . T h e se p a r t i c u l a r s c h o o l s w ere s e l e c t e d b e c a u s e t h e y r e p r e s e n t e d t h e n a t i o n a l means i n IQ and a c h i e v e m e n t 7 on s t a n d a r d i z e d t e s t s g i v e n a t t h e b e g i n n i n g o f t h e s c h o o l y e a r . The s o h o o l d i s t r i c t s e r v e s a p p r o x i m a t e l y 1 2 ,0 0 0 c h i l d r e n i n g r a d e s k i n d e r g a r t e n th r o u g h e i g h t i n La P u e n te , C a l i f o r n i a . The co m m u n ity w o u ld a p p e a r t o c o n s i s t l a r g e l y o f p e r s o n s o f u p p e r - l o w e r and l o w e r - m i d d l e s o c i o e c o n o m i c s t a t u s . The s t a n d a r d i z e d t e s t s u s e d i n t h i s i n v e s t i g a t i o n c o n s i s t e d o f t h e a r i t h m e t i c s e c t i o n s o f tw o w i d e l y u s e d s t a n d a r d i z e d a c h i e v e m e n t s e r i e s , t h e CAT a n d t h e SICA. O r g a n i z a t i o n o f th e R e m a in in g C h a p t e r s C h a p t e r I I p r e s e n t s a r e v i e w o f t h e l i t e r a t u r e p e r t a i n i n g t o r e s p o n s e s e t s , c h a n c e e f f e c t s , and s c o r i n g m e th o d s as t h e y r e l a t e to t h e p r e s e n t s t u d y . C h a p t e r I I I d e s c r i b e s t h e s a m p l e , t h e m e a s u r i n g i n s t r u m e n t s , t h e r e s e a r c h m e t h o d o l o g y , and t h e t r e a t m e n t o f t h e d a t a . C h a p t e r IV d i s c u s s e s t h e f i n d i n g s o f th e p r e s e n t i n v e s t i g a t i o n w ith an e v a l u a t i o n i n te r m s o f t h e r e s e a r c h h y p o t h e s e s . C h a p te r V p r e s e n t s a summary o f t h e s t u d y and r e l a t e s g e n e r a l c o n c l u s i o n s a n d r e c o m m e n d a tio n s w h ic h c a n be d raw n from t h e f i n d i n g s . CHAPTER I I REVIEW OF THE LITERATURE I t was t h e p u r p o s e o f t h i s c h a p t e r to p r e s e n t a r e v i e w o f t h e l i t e r a t u r e r e l a t i n g to t h e f o l l o w i n g : ( l ) t h e n a t u r e o f r e s p o n s e s e t s , (2 ) t h e s e t t o g am b le, (3) t h e s p e e d v e r s u s a c c u r a c y s e t , (4) t h e p o s i t i o n a l p r e f e r e n c e s e t , (fS) t h e i n f l u e n c e o f g u e s s i n g and c h a n c e s u c c e s s , and (6 ) t h e c o r r e c t i v e m e a s u r e s w h ich h a v e b e e n a t t e m p t e d i n e f f o r t s to c o m p e n s a te f o r t h e s e f a c t o r s . The n a t u r e o f r e s p o n s e s e t s C ro n b a o h ( 1 3 , 1 4 ) p r e s e n t e d t h e f i r s t s y s t e m a t i c i n v e s t i g a t i o n s i n t o t h e e x i s t e n c e and n a t u r e o f r e s p o n s e s e t s . T h e se s e t s a l t e r a r e s p o n s e t o a t e s t it e m I n s u c h a way t h a t t h e r e s p o n s e i n d i c a t e s s o m e t h i n g o t h e r t h a n w hat was i n t e n d e d ( 3 6 : 4 5 1 ) . S e v e r a l s u c h b i a s e s h a v e b e e n i d e n t i f i e d . Those r e l a t i n g t o t h e p r e s e n t s t u d y a r e ( l ) t h e t e n d e n c y t o g a m b le , (2 ) t h e sp e e d v e r s u s a c c u r a c y s e t , and (3 ) t h e p o s i t i o n a l p r e f e r e n c e s e t . When e v a l u a t i n g t h e r e s u l t s from an o b j e c t i v e t e s t , t h e e x a m i n e r u s u a l l y a ssu m ed t h e o b t a i n e d s c o r e s i n d i c a t e d t h e e x a m i n e e s ' k n o w le d g e o f t h a t p a r t i c u l a r c o n t e n t w i t h o u t c o n s i d e r i n g t h e a l l o y s o f t e s t - t a k i n g h a b i t s and p e r s o n a l i t y f a c t o r s w hich may be i n t e r s p e r s e d i n t h e t e s t s c o r e s . V a r i a n c e from r e s p o n s e s e t s h a s b een shown t o bo r e l a t i v e l y I n d e p e n d e n t o f a b i l i t y ( 6 6 ) , and c o n s e q u e n t l y t o r e p r e s e n t b i a s v a r i a n c e i n a p t i t u d e a n d a c h i e v e - n e n t t e s t s . P e r s o n s o f t h e s a n e a b i l i t y may e a r n s i g n i f i c a n t l y d i f f e r e n t s c o r e s s o l e l y b e c a u s e o f r e s p o n s e s t y l e s . I n some s i t u a t i o n s , h o w e v e r , i t i s p o s s i b l e f o r r e s p o n s e s e t s t o a c t u a l l y I n c r e a s e e m p i r i c a l v a l i d i t y . T h i s o c c u r s when t h e s e t I t s e l f c o r r e l a t e s w i t h t h e c r i t e r i o n . R e s p o n s e s e t s p r e s e n t a p a r a d o x . Some o f t h e r e s p o n s e s e t v a r i a n c e i s p o t e n t i a l l y u s e f u l , some o f i t i s a n I n t e r f e r e n c e w i t h m e a s u r e m e n t . The p r o b le m o f t h e t e s t e r i s t o c a p i t a l i z e on t h e e f f e c t s o f r e s p o n s e s e t s w h e re t h e y a r e h e l p f u l t o v a l i d i t y , and t o e l i m i n a t e t h e i r I n f l u e n c e w h e re i t i s u n d e s i r a b l e . . . . t h e p r o b a b l e e f f e c t o f r e s p o n s e s e t v a r i a n c e i s h arm f u l . (1 4 ) The i u f l u e n c e o f r e s p o n s e s e t s i s p o s i t i v e l y c o r r e l a t e d w i t h t h e d i f f i c u l t y o f a t e s t . As t e s t d i f f i c u l t y i n c r e a s e s , f a c t o r s o t h e r t h a n k n o w le d g e o f c o n t e n t a r e r e s p o n s i b l e f o r i n d i v i d u a l d i f f e r e n c e s i n s c o r e ( 1 3 ) . When t h e d i f f i c u l t y o f an i t e m a p p r o a c h e s . 0 0 , t h a t i s , when no e x a m in e e knows t h e c o r r e c t a n s w e r , r e s p o n s e s a r e d e t e r m i n ed by c a u t i o n , a c q u i e s c e n c e , o r o t h e r s e t s ( 4 1 ) . F o r t h i s r e a s o n C r o n b a c h ( 1 4 ) , G u i l f o r d ( 3 6 : 4 4 8 ) , and G u l l i k s e n ( 3 8 : 2 6 3 ) h a v e recom m ended n o t u s i n g a t e s t on s u b j e c t s f o r whom i t i s q u i t e d i f f i c u l t . R e s p o n s e s e t s h a v e b e e n shown t o be r e l i a b l e and t o p e r s i s t from one t e s t t o a n o t h e r (2 9 , 1 4 ) . The gamb l i n g a n d t h e s p e e d v e r s u s a c c u r a c y r e s p o n s e s e t s h a v e , 10 f o r e x a m p le , I n c r e a s e d t e s t r e l i a b i l i t y i n s i t u a t i o n s w here t h e i r i n f l u e n c e e x p a n d e d t h e s p r e a d o f s c o r e s ( 8 4 ) . At t i n e s , when t h e b i a s e s h ave lo w e r e d t h e c o r r e l a t i o n b e tw e e n i t e m s , r e l i a b i l i t y h as b e e n fo u n d t o d i m i n i s h ( 1 3 ) . T e s t s c o r e s a r e more d i f f i c u l t t o i n t e r p r e t i f r e s p o n s e s e t s a r e p r e s e n t . Ite m s h a v i n t h e same o s t e n s i b l e c o n t e n t m e a s u re more t h a n one t r a i t i f r e s p o n s e b i a s e s a r e o p e r a t i n g in t h e t e s t s . C ro n b a c h (1 4 ) fo u n d e v i d e n c e f o r s e t v a r i a n c e i n t h e s p a c e f a c t o r t e s t o f th e T e s t s o f P r im a r y M e n ta l A b i l i t i e s . Some o f t h i s v a r i a n c e was u n c o r r e l a t e d w i t h t h e s p a c e f a c t o r , c o n s e q u e n t l y t h e f a c t o r i a l p u r i t y o f th e t e s t was r e d u c e d , and t h e m eaning o f t h e s c o r e s made more a m b ig u o u s . G u i l f o r d (3(1:453) i n d i c a t e d t h a t a t e s t w i l l be a b e t t e r d i a g n o s t i c i n s t r u m e n t ev en i f t h e b i a s v a r i a n c e i s r e p l a c e d by e r r o r v a r i a n c e . In c o n t r a d i s t i n c t i o n t o th e b i a s v a r i a n c e , e r r o r v a r i a n c e i s n o t f o c u s e d s y s t e m a t i c a l l y i n any d i r e c t i o n . G u e s s in g I f a l l p u p i l s o f e u u a l a b i l i t y g u e s s e d w i t h e q u a l f r e q u e n c y , no s e r i o u s p ro b le m would r e s u l t . I t h a s been w e l l e s t a b l i s h e d , h o w e v e r, t h a t t h e r e a r e g r e a t i n d i v i d u a l d i f f e r e n c e s in t h e t e n d e n c y to g u e s s on t e s t i t e m s , t h a t t h e s e d i f f e r e n c e s a r e r e l i a b l e w i t h i n a t e s t and g e n e r a l l y a r e c o n s i s t e n t fro m one t e s t to a n o t h e r ( 4 8 : 3 4 8 , 3 3 , 37, 71, 7 2 ) . P s y c h o m e t r i c i a n s a r e s t i l l n o t i n c o m p l e t e a g r e e m e n t IX a s t o how t o m ost e f f e c t i v e l y d e a l w i t h t h e p ro b le m s o f c h a n o e s u c c e s s and g u e s s i n g ( 4 8 : 3 4 7 , 1 , 4 6 ) . The t e n d e n o y to g u e s s v a r i e s f r o a t h e p e r s o n who w i l l n o t g u e s s e v e n when t o l d he a u s t a n s w e r e v e r y q u e s t i o n t o t h e "gam bler*1 who a t t e m p t s e v e r y I te m r e g a r d l e s s o f c o n t e n t o r d i f f i c u l t y l e v e l ( 4 8 : 2 7 4 ) . "The f a c t t h a t n o t a l l e x a m in e e s a t t e m p t e v e r y i t e m im p o se s t r o u b l e s o m e s t a t i s t i c a l p r o b lem s a l l a l o n g t h e l i n e " ( 3 6 : 3 4 3 ) . I t i s a p p a r e n t t h a t when no p e n a l t y f o r g u e s s i n g i s em p lo y ed and when many p e r s o n s do n o t a t t e m p t a l l t h e i t e m s , t h e " g a m b l e r " w i l l be g i v e n an a d v a n t a g e o v e r t h e more d e l i b e r a t e s t u d e n t . F. B. D av is ( 4 8 :2 7 7 ) made a s t u d y w i t h a p p r o x i m a t e l y 400 h i g h s c h o o l p u p i l s who w ere g i v e n t h e N e ls o n -D e n n y R e a d in g T e s t . The e x a m in e e s who o b t a i n e d t h e f o r t y - s e v e n l o w e s t s c o r e s w ere t o l d t h e y m ig h t h a v e done b e t t e r had t h e y m arked an a n s w e r f o r e v e r y it e m w h e t h e r t h e y knew t h e a n s w e r o r n o t . They w e re t h e n g i v e n an a l t e r n a t e form o f t h e same t e s t . The mean s c o r e i n c r e a s e d from 2 5 .5 3 on t h e f i r s t t e s t t o 4 6 .3 2 on t h e s e c o n d t e s t w i t h an a v e r a g e g r a d e p la c e m e n t g a i n o f 2 . 7 g r a d e s a c c o r d i n g t o t h e p u b l i s h e d noruis. When b o th t e s t s w ere c o r r e c t e d f o r c h a n c e , an a v e r a g e c a i n o f o n l y two r a w - s c o r e p o i n t s was f o u n d . J a c k s o n (4 6 ) p r e s e n t e d a s t u d y w h ere t h e s u b j e c t s i n d i c a t e d t h e i t e m s on w hich t h e y had g u e s s e d . I n s t e a d o f a n s w e r i n g o n e - f i f t h c o r r e c t l y ( c h a n c e p r o b a b i l i t y ) , 12 s l i g h t l y more t h a n o n e - t h i r d o f t h e I t e m s c l a s s i f i e d a s g u e s s e s w e re o o r r e o t . O t h e r s t u d i e s ( 8 0 , 1 3 ) h a v e a l s o i n d i o a t e d t h a t on o b j e c t i v e t e s t s , t h e s t u d e n t u s u a l l y h a s b e t t e r t h a n c h a n c e p r o b a b i l i t y o f c h o o s i n g t h e c o r r e c t a n s w e r . T h i s i s p r o b a b l y a c c o u n t e d f o r by t h e r e l a t i v e u n a t t r a c t i v e n e s s o f some d i s t r a c t o r s and by t h e p a r t i a l k n o w le d g e p o s s e s s e d by many e x a m i n e e s . The t e n d e n c y t o g am ble h a s b e e n found t o i n c r e a s e t e s t r e l i a b i l i t y i n many c a s e s by e n l a r g i n g t h e r a n g e o f i n d i v i d u a l d i f f e r e n c e s . H ow ever, Wood (8 4 ) fo u n d t h a t t h i s s e t t e n d e d t o r e d u c e v a l i d i t y s i n c e t h e r e l a t i o n s h i p b e tw e e n k n o w le d g e and t h e c r i t e r i o n was a t t e n u a t e d . S t u d i e s by S w i n e f o r d ( 7 2 , 7 3 ) and J a c k s o n (4 6 ) fo u n d t h a t a b i l i t y i n t h e f i e l d c o v e r e d by a t e s t was i n d e p e n d e n t o f t h e s e t t o g a m b le . M o re o v e r , t h e r e s u l t s s u g g e s t e d t h a t f a m i l i a r i t y w i t h t h e m a t e r i a l had some e f f e c t on 1 h i s s e t ; new t y p e s o f m a t e r i a ] seem ed to e n c o u r a g e g u e s s i n g . S e v e r a l i n v e s t i g a t o r s hove r e p o r t e d a r e l a t i o n s h i p b e tw e e n g u e s s i n g and p e r s o n a l i t y c l a r a c t e r i s t i c s . Votaw (8 1 ) f o u n d t h a t s u b m i s s i v e s t u d e n t s ( a s d e f i n e d by t h e A l l p o r t A-S R e a c t i o n S t u d y ) o m i t t e d more i t e m s t h a n a s c e n d a n t s t u d e n t s . When a s k e d to a t t e m p t t h e o m i t t e d i t e m s , t h e " s u b m i s s i v e ” s t u d e n t s im p ro v e d t h e i r s c o r e s by a much g r e a t e r m a r g in t h a n d i d t h e " d o m i n a n t " g r o u p . An i n v e s t i g a t i o n w i t h A i r F o r c e p e r s o n n e l by T o r r e n c e and Z i l l e r 13 (8 0 ) showed t h a t g u e s s i n g b e h a v i o r on a n a c h i e v e m e n t e x a m i n a t i o n was c o r r e l a t e d w i t h p e r s o n a l i t y v a r i a b l e s , a s d e r i v e d from a b i o g r a p h i c a l i n v e n t o r y . They fo u n d a p o s i t i v e r e l a t i o n s h i p b e tw e e n g u e s s i n g a n d s e l f - c o n f i d e n c e , c o m p e t l v e n e s s , a n d s t r o n g i d e n t i f i c a t i o n w i t h t h e m asc u l i n e r o l e . S h e r r i f f ' s s t u d y (6 8 ) i n d i c a t e d t h a t g u e s s i n g b e h a v i o r c o r r e l a t e d n e g a t i v e l y w i t h an i n d e x o f m a l a d j u s t ment d e r i v e d from s e l e c t e d MMPI i t e m s . The more " m a la d j u s t e d ” s u b j e c t s o m i t t e d more ite m s t h a t t h e y w ere c a p a b l e o f a n s w e r i n g c o r r e c t l y t h a n d i d t h e l e s s " m a l a d j u s t e d . " Commenting on t h e p e r s o n a l i t y - r e s p o n s e s e t r e l a t i o n s h i p , C ro n b a c h s t a t e d : O nly u n d e r e x c e p t i o n a l c i r c u m s t a n c e s when a t e s t i s d e s i g n e d t o s t u d y t h e v a r i e d p e r s o n a l i t y c h a r a c t e r i s t i c s w h ic h a r e r e f l e c t e d i n t h e r e s p o n s e s e t , d o e s t h e r e s p o n s e s e t a p p e a r t o be a p o t e n t i a l l y h e l p f u l s o u r c e o f v a r i a n c e . (1 4 ) q u a n t i f y t h e s e t to gam ble was made by S w i n e f o r d (71) i n 1938. Through s p e c i a l d i r e c t i o n s on an a c h i e v e m e n t t e s t , s t u d e n t s w ere d i r e c t e d t o i n d i c a t e t h e i r c o n f i d e n c e i n t h e i r r e s p o n s e to e a c h it e m by s e l e c t i n g t h e amount o f c r e d i t d e s i r e d (2, 3, o r 4 u n i t s ) . D ouble t h e c r e d i t c l a i m e d was s u b t r a c t e d i f t h e r e s p o n s e was i n c o r r e c t . The e x a m i n e e ' s " g a m b lin g t e n d e n c y " was g i v e n by t h e f o l l o w i n g f o r m u l a : The m e a s u re m e n t o f g u e s s i n g . — The f i r s t a t t e m p t t o G am bling t e n d e n c y = 100 ( e r r o r s m arked " 4 " ) t o t a l e r r o r s + | o m l s s l 14 I n h e r e n t In t h i s f o r m u l a i s t h e a s s u m p t i o n t h a t a l l e r r o r s a r e g u e s s e s . The r e p o r t e d r e l i a b i l i t y o f t h i s i n d e x was . 7 9 6 . The i n d e x was n o t d e v e l o p e d fro m a c l e a r l y d e f i n e d t h e o r e t i c a l fra m e w o rk , h o w ev er, and a p p l i e s o n l y t o t e s t s w i t h s p e c i a l d i r e c t i o n s , d i r e c t i o n s to o u n w i e l d y f o r many e l e m e n t a r y s c h o o l p u p i l s . Z i l l e r (65) d e v e l o p e d an i n d e x o f r i s k - a c c e p t a n o e ( w i l l i n g n e s s to gam b le) i n 1 9 5 7 . T h e o r e t i c a l l y , t h i s i n d e x r e p r e s e n t s t h e r a t i o o f t h e number o f it e m s on w h ich t h e s u b j e c t g u e s s e d , to t h e t o t a l num ber o f ite m s he d i d n o t know b u t upon w h ic h he c o u l d h ave g u e s s e d . S t a t e d d i f f e r e n t l y , th e i n d e x e q u a l s t h e p r o p o r t i o n o f g u e s s e d it e m s com pared w i t h t h e t o t a l unknown i t e m s . The f o r m u l a f o r t h i s i n d e x r e n d s : R = g a m b lin g o r r i s k - R = ( n / n - 1 ) W a c c e p t a n c e i n d e x ( n / n - 1 ) W + 0 n = num ber o f o p t i o n s / i t e m W = number o f wrong re s p o n s e s 0 = num ber o f o m i t t e d i t e m s From t h e f o r m u l a i t would a p p e a r t h a t t h e g a m b lin g i n d e x i s a f u n c t i o n o f th e num ber o f ite m s marked i n c o r r e c t l y and t h e number o f ite m s o m i t t e d . B o th o f t h e s e q u a l i t i e s a r e f u n c t i o n s o f t h e d i f f i c u l t y o f a t e s t . I t i s e v i d e n t t h e n t h a t as a t e s t becomes more d i f f i c u l t , i t becomes i n c r e a s i n g l y I m p o r t a n t to c o n s i d e r t h e t e n d e n c y to g u e s s i n t e s t p e r f o r m a n c e . O th e r m ethods o f a s s e s s i n g g u e s s i n g have b e e n p r o p o s e d by ” o r s t (4 5 ) and B ro w n le s s and K e a t s ( 5 ) , 15 n e i t h e r o f w hich w ould a p p e a r a p p r o p r i a t e f o r t h i a s t u d y . H o r s t ' s m ethod r e q u i r e s a d m i n i s t r a t i v e p r o c e d u r e s w h ic h w ould d e p a r t from t h e s t a n d a r d i z e d d i r e c t i o n s , t h e r e b y i n t r o d u c i n g a n o t h e r v a r i a b l e . The method p r o p o s e d by B r o w n le s s and K e a t s r e q u i r e s two a d m i n i s t r a t i o n s w i t h t h e same t e s t u s i n g t h e m u l t i p l e r e s p o n s e fo rm . T h is p r o c e d u r e d i d n o t a p p e a r t o be a p p l i c a b l e to t h e p r e s e n t s t u d y . Z i l l e r ' s m ethod (85) was d e t e r m i n e d to be t h e m ost a p p r o p r i a t e f o r p u r p o s e s o f t h i s i n v e s t i g a t i o n and was c o n s e q u e n t l y e m p lo y e d . The s p e e d v e r s u s a c c u r a c y s e t In num erous t e s t s , r e s p o n s e s p e e d i s an i m p o r t a n t e l e m e n t . One a u t h o r i t y on t h e t e a c h i m : o f a r i t h m e t i c , t h e a c h i e v e m e n t a r e a u s e d i n t h i s s t u d y , b e l i e v e s to o much e m p h a sis i s p l a c e d upon s p e e d in s t a n d a r d i z e d t e s t s (5 8 ) . To t h e e x t e n t t h i s i s t r u e , t h e a r i t h m e t i c t e s t s a r e p r o b a b l y m e a s u re s o f c l e r i c a l s p e e d and a c c u r a c y r a t h e r t h a n c u r r i c u l a r o b j e c t i v e s o f a r i t h m e t i c . Many s t u d i e s h av e shown t h a t s p e e d o f r e s p o n s e h as l i t t l e c o r r e l a t i o n w i t h m ost t y p e s o f a b i l i t i e s . T a te (73) found no b e t t e r t h a n c h a n c e c o r r e l a t i o n s b e tw e e n s p e e d and a b i l i t y w i t h t e s t s of a r i t h m e t i c r e a s o n i n g , s e n t e n c e c o m p l e t i o n , number s e r i e s , and s p a t i a l r e l a t i o n s , i-ivcn a t t h r e e d i f f i c u l t y l e v e l s . F a c t o r a n a l y s e s have a l s o shown s p e e d to be r e l a t i v e l y i n d e p e n d e n t o f a b i l i t y l e v e l ( 1 7 , 6 1 ) . S i n c e 16 s p e e d I s u s u a l l y o n l y a m in o r o r i t e r i o n c o m p o n e n t, m e a s u r e s w h ic h p e r m i t v a r i a n c e f ro m t h i s s o u r o e to s u b s t a n t i a l l y l n f l u e n o e t h e t o t a l v a r i a n c e s h o u l d n o t be u s e d . U n f o r t u n a t e l y , few s t a n d a r d i z e d t e s t a g i v e a com p l e t e d e s c r i p t i o n o f t h e d e g r e e t o w h ich t h e t e s t s a r e s p e e d e d ( 6 1 ) . T a t e (7 4 ) fo u n d t h a t s u b j e c t s who r e s p o n d e d q u i c k l y a t one d i f f i c u l t y l e v e l , g e n e r a l l y r e s p o n d e d q u i c k l y a t o t h e r d i f f i c u l t y l e v e l s . B e n n e t t (2 ) fo u n d t h i s r e l a t i o n s h i p f o r t h e f a s t e r r e s p o n d i n g e x a m i n e e s , b u t n o t f o r t h e s l o w e r r e s p o n d i n g e x a m i n e e s . I t i s e v i d e n t t h a t some f a c t o r o t h e r t h a n ite m d i f f i c u l t y i s a f f e c t i n g th e r e s p o n s e s p e e d o f t h e s l o w e r r e s p o n d i n g g r o u p . T h i s h a s b e e n shown n o t t o be a b i l i t y ( 1 7 , 7 4 ) w h ic h e v i n c e s t h e n e e d f o r c o r r e c t i v e m e a s u r e s on t i m e - l i m i t t e s t s . A t i m e - l i ;nit t e s t i s n o t n e c e s s a r i l y s p e e d e d . I f a d d i t i o n a l tim e would a f f e c t l i t t l e c h a n g e i n t h e r e l a t i v e p l a c e m e n t o f t h e s u b j e c t s , s p e e d w ould be c o n s i d e r e d t o be an i n s i g n i f i c a n t f a c t o r . Some s t a n d a r d i z e d t e s t s , a l th o u g h p u r p o r t i n g t o be e s s e n t i a l l y pow er t e s t s , h a v e b e e n fo u n d t o d e p e n d on s p e e d t o a s u b s t a n t i a l e x t e n t ( 1 : 1 1 4 , 1 4 :2 :1 2 ). A l t h o u g h a c o r r e l a t i o n b e tw e e n s c o r e s o b t a i n e d on t h e K u h lm a n n - A n d e rs o n I n t e l l i g e n c e T e s t u n d e r t im e d and u n t i m e d c o n d i t i o n s a t g r a d e f i v e was f o u n d t o be . 8 3 , C r o n b a c h e s t i m a t e d t h a t 30% o f t h e t e s t v a r i a n c e w o u ld b e due t o s p e e d i f a l l o w a n c e was made f o r t h e l a c k o f 17 I n d e p e n d e n c e o f t h e two s c o r e s ( 1 4 : 2 2 2 ) . I n s u c h c a s e s w here s p e e d i s a s i g n i f i c a n t f a c t o r , r e s p o n s e s e t v a r i a n c e w i l l c o n t a m i n a t e t h e t e s t r e s u l t s . P o s i t i o n a l p r e f e r e n c e s In r e l a t i v e l y u n s t r u c t u r e d s i t u a t i o n s , human r e s p o n s e s o f t e n show m arked d e p a r t u r e s from c h a n c e d i s t r i b u t i o n s ( 3 ) . . . . when e x a m in e e s a r e i g n o r a n t o f t h e a n s w e r t o an i t e m , t h e i r h a b i t s o f t a k i n g t e s t s a r e s u c h t h a t t h e y do n o t c h o o s e among t h e a l t e r n a t i v e s e n t i r e l y a t ra n d o m . C e r t a i n p o s i t i o n s in a l i s t o f f i v e r e s p o n s e s may be f a v o r e d by h a b i t s o f r e a d i n g o r o f a t t e n t i o n . ( 3 6 :3 8 0 ) The e v i d e n c e on th e e x i s t e n c e o f a f a v o r i n g o f c e r t a i n r e s p o n s e p o s i t i o n s s e t i s e q u i v o c a l . C ro n b a c h ( 1 3 , 1 4 , 1 5 ) h a s r e p e a t e d l y c l a i m e d t h a t t h e m u l t i p l e - c h o i c e t e s t i s v i r t u a l l y immune to r e s p o n s e b i a s e s o t h e r t h a n t h e g a m b lin g and s p e e d v e r s u s a c c u r a c y s e t s . C l a r k ( 8 ) , u s i n g s t a n d a r d i z e d t e s t s w i t h c o l l e g e s t u d e n t s , found t h a t g e n e r a l p o s i t i o n a l p r e f e r e n c e s , a s t h e y e x i s t w i t h f i v e - c h o i c e i t e m s , to be w eak. W orking u n d e r p r e s s u r e o f ti m e , h o w e v e r, s u b j e c t s u s e d t h e p o s i t i o n s i n a s l i e h t l y d e c l i n in g f r e q u e n c y , b u t , he c o n c l u d e d : When t h e r e l a t i v e a t t r a c t i v e n e s s o f o p t i o n s i s e x p e r i m e n t a l l y c o n t r o l l e d and when t h e e x a m in e e s a r e u n d e r no tim e p r e s s u r e t o f i n i s h , p o s i t i o n a l p r e f e r e n c e s a r e in d e e d i n f i n i t e s i m a l . M e t f e s s e l (6 4 ) c o n t r o l l e d t h e r e l a t i v e a t t r a c t i v e n e s s o f t h e o p t i o n s by u s i n g e x p e r i m e n t a l l y d e t e r m i n e d , e q u a l l y 18 a t t r a c t i v e a l t e r n a t i v e s ( n o n s e n s e s y l l a b l e s ) , and fo u n d a s i g n i f i c a n t e x a m in e e p r e f e r e n c e f o r t h e m id d le ( t h i r d ) r e s p o n s e p o s i t i o n . R a p a p o r t and Berg (6 4 ) r e p o r t e d no g e n e r a l t e n d e n c y f o r p o s i t i o n a l b i a s e s u s i n g a g e n e r a l i n f o r m a t i o n t e s t w i t h c o l l e g e s t u d e n t s . O nly 8 . 8 ^ o f t h e g r o u p r e v e a l e d a s i g n i f i c a n t p o s i t i o n a l b i a s and c h a n c e w ould h a v e d i c t a t e d 5 # . A s i g n i f i c a n t s e x d i f f e r e n c e was f o u n d — t h e m a le s composed a g r e a t e r t h a n c h a n c e p r o p o r t i o n o f t h e " p o s i t i o n a l b i a s e d " g r o u p . F o r t h o s e f o r t y - s i x e x a m in e e s i n t h e " p o s i t i o n a l b i a s e d " g r o u p , f i f t e e n p r e f e r r e d t h e t h i r d o p t i o n p o s i t i o n . McNamara and Weitzman (5 2 ) fo u n d t h e p o s i t i o n f a c t o r t o be i m p o r t a n t o n l y when d e a l i n g w i t h many q u e s t i o n s and a l a r g e number o f s u b j e c t s , a c o n d i t i o n met i n t h e p r e s e n t s t u d y . The e f f e c t s o f d i r e c t i o n s T r a x l e r ( 4 8 : 3 5 1 ) has p o i n t e d o u t t h a t t h e p ro b le m o f optimum d i r e c t i o n s n e e d s o b j e c t i v e i n v e s t i g a t i o n . F e d e r (2 6 ) u r g e d t e s t p u b l i s h e r s to d e t e r m i n e by e x p e r i m e n ta l p r o c e d u r e s t h e b e s t d i r e c t i o n s on t h e i r p a r t i c u l a r t e s t s b e f o r e m a r k e t i n g t h e i r p r o d u c t s . Many t e s t a u t h o r i t i e s c o n s i d e r d i r e c t i o n s w hich l e a v e t h e s t u d e n t to j u d g e w h at t o do a b o u t g u e s s i n g to be i n a d e q u a t e ( 4 8 : 3 3 1 , 1 6 , 4 9 ) ; a l t h o u g h t h i s f a c t w ould a p p e a r to have l o g i c a l v a l i d i t y , l i t t l e e m p i r i c a l e v i d e n c e i s a v a i l a b l e on t h i s p o i n t . The f a c t t h a t a m b l g u l t v i s r e d u c e d by s u c h d i r e c t i o n s may be 19 o f f s e t b y t h e i n c r e a s e i n r e s p o n s e s e t v a r i a n c e i n t r o d u c e d by c e r t a i n t y p e s o f d i r e c t i o n s ( 1 6 ) . F o r c e d - c h o i c e d i r e c t i o n s . — The f o r c e d - c h o i c e t y p e o f d i r e c t i o n s h a v e b e e n recom m ended by some a u t h o r i t i e s to m i n i m i z e t h e e f f e c t s o f r e s p o n s e s e t s ( 1 5 , 3 5 ) . I n 1950, C r o n b a c h s t a t e d : " I n m ost t e s t s s u b j e c t s s h o u l d be d i r e c t e d t o a n s w e r a l l i t e m s , e v e n th o u g h t h i s t e n d s t o i n c r e a s e t h e random e r r o r v a r i a n c e " ( 1 5 ) . He f e l t t h i s l i a b i l i t y w ould be more t h a n c o m p e n s a t e d f o r by t h e c a n c e l i n g o u t o f t h e r e s p o n s e s e t v a r i a n c e . I t would seem t h a t h i s p o s i t i o n h a s been m o d i f i e d a s , i n 1960, he i n d i c a t e d a p r e f e r e n c e f o r d i r e c t i o n s t h a t d i s c o u r a g e w i l d g u e s s i n g b u t e n c o u r a g e g u e s s i n g b a s e d on p a r t i a l k n o w l e d g e ( 1 4 : 5 1 ) . HThen d i r e c t e d t o r e s p o n d t o e v e r y i t e m , D avis ( 4 6 : 2 7 4 ) fo u n d t h a t many s t u d e n t s w ould n o t c o m p ly . Only 421 i n t h e s a m p le o f 541 c o l l e g e f r e s h m e n a c t u a l l y m arked an a n s w e r t o e v e r y i t e m . He s t a t e d t h a t f o r c i n g s t u d e n t s t o mark a n s w e r s to i t e m s on w h ic h t h e y h a v e no k n o w le d g e i s f r u s t r a t i n g to t h e s t u d e n t s , c o n t r a r y t o good t e a c h i n g , and b r e a k s h a b i t s o f c a r e f u l n e s s t h a t t h e s c h o o l s t r y t o i n c u l c a t e ( 4 6 : 2 7 4 - 2 7 5 ) . R e s e a r c h seem s t o i n d i c a t e t h e a d v a n t a g e s , i f an y , o f t h e f o r c e d - c h o i c e d i r e c t i o n s t o l i e i n a r e a o f p e r s o n a l i t y a s s e s s m e n t ( 3 1 , 6 2 ) . 20 "Do n o t g u e s s " i n s t r u c t i o n s . — S w i n e f o r d and M i l l e r ( 7 3 ) , a f t e r e x p e r i m e n t i n g w i t h i n s t r u c t ! m s t h a t e n c o u r ag ed s t u d e n t s t o g u e s s a s c o n t r a s t e d w i t h i n s t r u c t i o n s n o t to g u e s s , c o n c l u d e d t h a t t h e d e g r e e o f g u e s s i n g v a r i e d w i t h t h e i n s t r u c t i o n s . J a c k s o n (4 6 ) r e p o r t e d a c o r r e l a t i o n o f .87 on a v o c a b u l a r y t e s t g i v e n w i t h and w i t h o u t d i r e c t i o n s t h a t wrong a n s w e r s w ould c o u n t a g a i n s t t h e e x a m i n e e s . T h is c o r r e l a t i o n was n o t s i g n i f i c a n t l y d i f f e r e n t from t h e p a r a l l e l form r e l i a b i l i t y c o e f f i c i e n t o f .9 1 , g i v e n w i t h i d e n t i c a l d i r e c t i o n s . A lth o u g h i t was c o n c l u d e d t h a t s t u d e n t p e r f o r m a n c e d i d n o t v a r y w i t h t h e p r e s e n c e o r a b s e n c e o f d i r e c t i o n s r e g a r d i n g p e n a l t i e s f o r g u e s s i n g , t h i s d o e s n o t seem t o be a l e g i t i m a t e c o n c l u s i o n s i n c e c o n t e n t was a l l o w e d t o v a r y . One d o e s n o t know w h a t t h e t e s t - r e t e s t r e l i a b i l i t y would h av e b e e n ; had i t b een v e r y h i g h , s i g n i f i c a n c e would h av e been r e a c h e d . In s e v e r a l i n v e s t i g a t i o n s ( 1 5 , 1 6 , 3 4 ) i t h as b een fo u n d t h a t d i f f e r e n c e s in t e n d e n c y to gam ble a r e n o t e l i m i n a t e d oy any c h a n g e o f d i r e c t i o n s o f p e n a l t y . T r a x - l e r ( 4 8 : 3 5 0 ) p o i n t e d o u t t h a t p e o p l e c a n n o t f o l l o w i n s t r u c t i o n s u n i f o r m l y s i n c e t h e m ean in g o f t h e te rm " g u e s s " d i f f e r s m a r k e d ly from one i n d i v i d u a l to a n o t h e r . The work o f S o d e r o u i s t (b9) and S w in e f o r d (7 2 ) h a s s u p p o r t e d t h e h y p o t h e s i s t h a t " d o - n o t - g u e s s " i n s t r u c t i o n s r e s u l t s in t h e m e a s u re m e n t o f p e r s o n a l i t y v a r i a b l e s . D unlap (2 3 ) p r o p o s e d t h e u s e o f " d o - n o t - g u e s s " 21 I n s t r u c t i o n s , b u t t o l n s t r u o t t h e e x a m in e e , a f t e r o o u n t i n g t h e num ber o f r e s p o n s e s o f w h ic h he was s u r e , t o mark a l l r e m a i n i n g i t e m s w i t h t h e l e a s t u s e d o p t i o n . H is r e s u l t s showed t h e s e d i r e c t i o n s to be s l i g h t l y s u p e r i o r t o i n s t r u c t i o n s t o g u e s s , b u t som ew hat I n f e r i o r t o " d o - n o t - g u e s s " I n s t r u c t i o n s . E x p e r i m e n t a l a t t e m p t s to s t r u c t u r e t h e t e s t by i n f o r m i n g t h e e x a m in e e t h a t a c e r t a i n p e r c e n t a g e o f t h e i t e m s a r e k ey ed c o r r e c t i n a s p e c i f i e d way h ave n o t b e e n s u c c e s s f u l ( 1 5 ) . Coombs (1 2 ) d i s c o v e r e d an I n c r e a s e i n r e l i a b i l i t y by r e q u i r i n g t h e e x a m in e e s to c r o s s o u t i n c o r r e c t a l t e r n a t i v e s w i t h th e u n d e r s t a n d i n g t h a t one p o i n t c r e d i t was a l l o w e d f o r e ach s u c h d i s t r a c t o r c r o s s e d - o u t . S o d e r q u i s t ( 6 9 ) , by h a v i n g t h e s u b j e c t a c c o r d t h e d e g r e e o f c o n f i d e n c e w i t h which he h a s made h i s r e s p o n s e , fo u n d a s l i g h t i n c r e a s e i n r e l i a b i l i t y when s c o r e d w i t h a p e n a l t y f o r e r r o r . S t u d i e s have g e n e r a l l y a g r e e d t h a t " d o - n o t - g u e s s " i n s t r u c t i o n s y i e l d c o e f f i c i e n t s somewhat h i g h e r t h a n " g u e s s " i n s t r u c t i o n s ( 1 1 , 1 3 , 3 4 , 6 6 ) . T h is p r o b a b l y d o e s n o t r e p r e s e n t a g a i n in r e l e v a n c e , h o w e v e r, w h ich w ould be e x p e c t e d fro m a t r u e r e l i a b i l i t y i n c r e a s e . V a l i d i t y s t u d i e s h av e shown i n c o n c l u s i v e r e s u l t s . Votaw (81) i n f e r r e d t h a t v a l i d i t y was d e c r e a s e d when s t u d e n t s w ere d i r e c t e d n o t to g u e s s . C ro n b a c h ( 1 4 : 5 1 ) s t a t e d t h a t " d o - n o t - g u e s s " i n s t r u c t i o n s have u s u a l l y shown s l i g h t l y 22 g r e a t e r p r e d i c t i v e p o w e r. I t i s p r o b a b l e t h a t t h e l a c k o f c o n t r o l o f i t e m d i f f i c u l t y , a l o n g w i t h t h e v a r i e d d i r e c t i o n s , a c c o u n t f o r many o f t h e a p p a r e n t l y c o n f l i c t i n g r e s u l t s . A s t u d y o f Ruch and S t o d d a r d ( 6 5 ) i s p r e s e n t e d i n some d e t a i l b e c a u s e o f i t s p e r t i n e n c e t o t h e p r o b le m . They c o m p a re d t h e r e s u l t s from s i x t y p e s o f t e s t f o r m a t s u s i n g t h e i d e n t i c a l c o n t e n t u n d e r c o n d i t i o n s o f " g u e s s " and " d o - n o t - g u e s s " i n s t r u c t i o n s . A r e c a l l fo rm o f t h e t e s t was a lw a y s g i v e n f i r s t , f o l l o w e d by a m u l t i p l e c h o i c e form o f 7, 5, 3, o r 2 a l t e r n a t i v e s ; o r by a t r u e - f a l s e fo rm o f t h e same t e s t . The s a m p le c o n s i s t e d o f 2 ,4 5 3 p u p i l s i n g r a d e s s e v e n , e i g h t , e l e v e n , an d t w e l v e . The r e s u l t s a r e g i v e n i n T a b l e 1 . The g e n e r a l c o n c l u s i o n s draw n fro m t h i s s t u d y showed t h a t : ( l ) t h e v a r i o u s t y p e s o f i t e m s m e a s u re d d i f f e r e n t i n f o r m a t i o n and s k i l l s o n l y to a s l i g h t e x t e n t ; (2 ) l i t t l e d i f f e r e n c e i n r e l i a b i l i t y r e s u l t e d from t h e a d d i t i o n o f more t h a n t h r e e r e s p o n s e op t i o n s ; (3 ) a s m a l l , b u t s t a t i s t i c a l l y s i g n i f i c a n t t r e n d f o r t h e a l t e r n a t e fo rm r e l i a b i l i t y t o be i n c r e a s e d e x i s t e d when t h e c o r r e c t i o n s f o r c h a n c e w ere a p p l i e d to t h e d a t a f o r s t u d e n t s i n s t r u c t e d t o g u e s s , b u t n o t f o r t h o s e t o l d t o r e f r a i n fro m g u e s s i n g ; ( 4 ) m e a g e r v a l i d i t y i n c r e m e n t s r e s u l t e d from t h e u s e o f c o r r e c t e d s c o r e s ; and (5 ) d i r e c t i o n s n o t t o g u e s s r e d u c e d mean raw s c o r e s 10 t o 2 0 co and r e d u c e d mean c o r r e c t e d s c o r e s two t o t h r e e p e r c e n t . T 13L E 1 .—T he effects of guess versus do-not-guess directions (66) R e lia b ility : E n u iv alen t Forira V a lid ity ! C o rre la tio n w ith R ecall Guess D o Not Guess Guess D o Not Guess Means Guess Do Not Guess Unc. C orr. Unc. C o rr. nc. C orr. Unc. C orr. Unc. C orr. Unc. C orr. R ecall .95 27 27 7-Response .80 .81* .69 .91 .87 .87 .93 .93 50 la 1 * 5 1 * 0 5-Response .86 .90 .86 .68 .91 .91 .89 .92 5 1 * 1 * 3 1 * 8 1 * 2 3-Response .81* .86 .89 .89 .81* .85 .85 .92 62 1 * 3 5U la 2-Response .75 .86 .86 .81* .86 .86 .71* .77 72 1 * 3 65 1 * 5 T ru e -fa lse .61 * .78 .89 .81* .80 .4* .75 .86 66 32 51 31 juean .78 .85 .88 .87 .85 .87 .85 .89 ro 24 P r e s e n t l y the most w i d e l y a c c e p t e d p r a c t i c e i s to inform the examine# t h a t w il d g u e s s i n g i s t o h i s d i s a d v a n tag e but to en cou rag # him to respond when an informed judgment as to the most r e a s o n a b l e answer can be made, even though he may he u n c e r t a i n ( 1 4 : 5 1 , 4 8 : 3 5 1 ) . S c o r i n g formu las As p r e v i o u s l y i n d i c a t e d , c o n s i d e r a b l e d i f f e r e n c e of o p i n i o n e x i s t s r e g a r d i n g 1 he most d e s i r a b l e p ro ced ur e to be used in s c o r i n g t e s t s (45, 48, 7 0 ) . Mjtnv p u b l i s h e d t e s t s r e q u i r e the use o f a c o r r e c t i o n formula and manv do not (22 :1 5'. A summary o f the f i ndi ngs by v a r i o u s i n v e s t i g a t o r s with t lie c o nve n t i ona 1 scuri ■ • n formula is "iven in Tab 1e 2. From Table 2 i t at'pears that a l l the s t u d i e s and a u t h o r i t i e s are not in a c c o r d . It seems prob ab le I 1at the d i f f e r e n t find tugs r e s u l t l a r g e Iv from s i t u a t i o n a l f a c t o r s in the t e s t i n g s i t u a t i o n and in t 1 i n s I ru me n 1 s u s e d . alien an *xte run I cr i t *r i on is a v a i l a b l e , and a ben t lie " r i g h t s ” and "wrongs" do not c o r r e l a t e h i g h l y , ad va n t ag eou s have so metimes been found bv t r e a t i rig 1 he 1! and V s c o r e s as i f t h e y had come from two d i f f e r e n t t e s t s and combine them i n t o a m u l t i p i e - r e g r e s s ion e u u n t i o n ( 3 7 , 4 0 , 7 7 ) . The optimum p e n a l t y f or t h e "wrongs" i s thus d e t e r mined e x p e r i m e n t a l l y to y i e l d maximum v a l i d i t y . F a c t o r a n a l y s i s has d e m o n s t r a t e d that ri ^h t and 25 TABLE 2 . — Summary o f r e s e a r c h w i t h t h e c o n v e n t i o n a l c o r r e c t i o n f o r m u l a E f f e c t s o n : R e s p o n s e s e t s I n t r o d u c e s v a r i a n c e i n t h e fo rm o f a p e r s o n a l i t y t r a i t I f exam i n e e s a r e aw are of t h e p e n a l t y f o r g u e s s i n g ( 4 , 1 6 , 4 8 ) . R e d u c e s v a r i a n c e from s p e e d v e r s u s a c c u r a c y o r g a m b lin g s e t s (1 4 , 4 2 ) . I te m s t a t i s t i c s R e s u l t s i n more u s e f u l i t e m s t a t i s t i c s ( 4 8 : 2 7 7 , 4 ) . Y i e l d s a m a x i m u m - l i k e l i h o o d e s t i m a t e o f an i n d i v i d u a l ' s t r u e k n o w le d g e o f t h e f i e l d m e a s u re d u n d e r c e r t a i n c o n d i t i o n s ( 3 7 : 4 4 9 , 1 5 ) . O v e r c o r r e c t s to t h e e x t e n t t h a t e x a m in e e s c h o o s e i n c o r r e c t a n s w e r s on t h e b a s i s o f m i s i n f o r m a t i o n r a t h e r t h a n by b l i n d g u e s s i n g ( 4 8 : 2 6 8 , 3 4 9 ) . R e l i a b i l i t y R e s u l t s i n s l i g h t l y h i g h e r r e l i a b i l i t i e s ( l l , 1 3 , 3 4 , 6 5 , 8 0 ) . No s i g n i f i c a n t d i f f e r e n c e r e s u l t s fro m u s e o f c h a n c e c o r r e c t i o n ( 4 8 : 3 6 6 ) . V a l i d i t y I n c r e a s e s v a l i d i t y to a s l i g h t e x t e n t ( 4 8 : 3 6 6 , 6 6 , 8 8 ) . D e c r e a s e s v a l i d i t y t o a s l i g h t e x t e n t ( 8 0 ) . V a l i d i t y n o t s i g n i f i c a n t l y e f f e c t e d ( 3 4 : 7 3 ) . 26 wrong r e s p o n s e s m e a s u re d i f f e r e n t f a c t o r s to some e x t e n t ( 2 8 ) . To t h e d e g r e e t h a t t h i s o c c u r s , and t h a t t h e " r i g h t " and "w rong" s c o r e s i n d e p e n d e n t l y c o r r e l a t e w i t h t h e c r i t e r i o n , g r e a t e r p r e d i c t i o n w i l l r e s u l t from u s i n g t h e o p t i m a l w e i g h t i n g p r i n c i p l e I n c o r p o r a t e d i n t o t h e m u l t l p l e - r e g r e s s l o n e m p i r i c a l s c o r i n g f o r m u l a . G u i l f o r d ( 3 6 : 4 8 3 ) h a s s u g g e s t e d t h a t t h e r e may be r i c h p o s s i b i l i t i e s in t h e f u l l e r u s e o f t h e w e i g h t i n g o f wrong r e s p o n s e s . Summary T h i s c h a p t e r sum m arized t h e l i t e r a t u r e r e l a t i n g to t h e p r o b le m s o f c e r t a i n r e s p o n s e s e t s , t h e i n f l u e n c e o f d i r e c t i o n s on e x a m in e e p e r f o r m a n c e , and t h e e f f e c t s from t h e u s e o f v a r i o u s s c o r i n g p r o c e d u r e s . T h is r e v i e w has le d t o t h e f o l l o w i n g summary s t a t e m e n t s . 1 . R e s p o n s e s e t s a r e g e n e r a l l y r e l i a b l e and o f t e n i n c r e a s e t h e o b s e r v e d r e l i a b i l i t y o f a t e s t by e n l a r g i n g t h e s p r e a d o f s c o r e s . 2 . The v a r i a n c e from r e s p o n s e s e t s i n c r e a s e s p r o p o r t i o n a t e l y w i t h t e s t d i f f i c u l t y , o t h e r f a c t o r s c o n s t a n t . 3 . T e s t s c o r e s a r e more d i f f i c u l t to i n t e r p r e t i f r e s p o n s e s e t e f f e c t s a r e p r e s e n t . 4 . The most e f f e c t i v e way o f d e a l i n g w i t h t h e p r o b le m s o f g u e s s i n g i s s t i l l u n s e t t l e d . Some a u t h o r i t i e s s t r o n g l y f a v o r t h e u s e o f f o r m u l a s w h ich a t t e m p t to c o r r e c t f o r c h a n c e , o t h e r s f e e l t h e s e f o r m u l a s i n t r o d u c e more 27 • r r o r t h a n t h e y c o r r e c t . 5 . The r e l a t i o n s h i p b e tw e e n a b i l i t y and t e n d e n c y t o g u e s s i s n o t s i g n i f i c a n t . 6 . P e r s o n a l i t y c h a r a c t e r i s t i c s a r e c l o s e l y r e l a t e d t o t h e s e t t o g a m b le . F o r e x a m p le , some e x a m in e e s w i l l g u e s s r e g a r d l e s s o f c o n t e n t o r d i r e c t i o n s ; o t h e r s w i l l n o t g u e s s and a t t e m p t a l l i t e m s e v e n w i t h f o r c e d - c h o i c e d i r e c t i o n s . 7 . Most s t u d i e s show l i t t l e r e l a t i o n s h i p b e tw e e n s p e e d o f r e s p o n s e and a b i l i t y . 8 . L i t t l e a g r e e m e n t I s fo u n d among i n v e s t i g a t o r s r e g a r d i n g t h e e x i s t e n c e o f a p o s i t i o n a l p r e f e r e n c e r e s p o n s e s t y l e . 9 . The d i f f e r e n c e i n t e n d e n c y to gam ble i s n o t e l i m i n a t e d by any c h a n g e o f d i r e c t i o n s o r p e n a l t y . 10. D i s a g r e e m e n t e x i s t s among t e s t a u t h o r i t i e s r e g a r d i n g t h e u s e o f t h e f o r c e d - c h o i c e ty p e o f d i r e c t i o n s . 11. Most s t u d i e s show t h a t " d o - n o t - g u e s s " i n s t r u c t i o n s y i e l d r e l i a b i l i t y c o e f f i c i e n t s somewhat h i g h e r t h a n " g u e s s " i n s t r u c t i o n s . The c o n c o m i t a n t i n f l u e n c e on v a l i d i t y a p p e a r s u n c l e a r . 12. C o n s i d e r a b l e d i f f e r e n c e o f o p i n i o n e x i s t s r e g a r d i n g t h e most d e s i r a b l e p r o c e d u r e t o be u s e d i n t e s t s c o r i n g . Most s t u d i e s h a v e shown t h e t h e o r e t i c a l c h a n c e c o r r e c t i o n f o r m u l a t o y i e l d s l i g h t i n c r e c s e s in t e s t r e l i a b i l i t y and, to some e x t e n t , v a l i d i t y when u s e d i n c o n j u n c t i o n w i t h t h e " d o - n o t - g u e s s " i n s t r u c t i o n s . CHAPTER I I I METHODOLOGY The p u r p o s e o f t h i s c h a p t e r was t o p r e s e n t a d e s c r i p t i o n o f t h e s a m p le o f t h e e v a l u a t i v e I n s t r u m e n t s and t o o u t l i n e t h e r e s e a r c h d e s i g n and p r o c e d u r e s o f t h i s s t u d y . The sa m p le The s u b j e c t s o f t h i s s t u d y c o n s i s t e d o f 246 f i f t h g r a d e s t u d e n t s from f o u r e l e m e n t a r y s c h o o l s in t h e Hudson S c h o o l D i s t r i c t (K -8) l o c a t e d in t h e s o u t h e a s t e r n s e c t i o n o f Los A n g e le s C o u n ty . T h is d i s t r i c t i s composed o f n i n e t e e n s c h o o l s w h ic h s e r v e a p p r o x i m a t e l y 1 2 ,0 0 0 p u p i l s . This sa m p le c o n s i s t e d o f t h e e n t i r e f i f t h g r a d e p o p u l a t i o n s o f th e f o u r s c h o o l s (N = 2 8 6 ) . The 16f$ who w ere a b s e n t a t one o r b o th o f t h e t e s t i n g s i t u a t i o n s w e re not i n c l u d e d . These p a r t i c u l a r s c h o o l s were s e l e c t e d b e c a u s e t h e y a p p r o a c h e d t h e n a t i o n a l means i n b o t h a c h i e v e m e n t and IQ i n p r e v i o u s s t a n d a r d i z e d t e s t i n g . T a b le 3 p r e s e n t s t h e r e s u l t s o f t e s t s g i v e n in O c t o b e r , 1959, a t t h e s e s c h o o l s . The s c h o o l s w ere m a tc h e d as e v e n l y a s p o s s i b l e on th e b a s i s o f num ber o f s t u d e n t s , r e s u l t s o f p r e v i o u s t e s t in g , and s o c io e c o n o m i c c l a s s s e r v e d by t h e s c h o o l . C o n se q u e n t l y , s c h o o l s H and K w ere a d m i n i s t e r e d t h e a r i t h m e t i c s e c t i o n s o f t h e CAT, and S c h o o l s C and T th e a r i t h m e t i c 30 TABLE 3 . — P r e v io u s ® a c h i e v e m e n t and a p t i t u d e t e s t r e s u l t s a t t h e f o u r e x p e r i m e n t a l s c h o o l s S c h o o l N CTMM Mean IQ CAT Grade A r i t h . R e a s . P la c e m e n t A r i t h . F u n d . V a lu e s T o t a l A r i t h . C 94 99 .88 5 . 3 1 5 .0 9 5 .2 3 H 47 1 0 4 .4 2 5 .3 2 5 .0 7 5 .2 3 K 81 9 8 . 0 5 5 . 2 6 5 .0 3 5 . 1 8 T 45 9 6 .5 9 4 . 8 7 4 .74 4 .83 Sample I (H & K) : 128 1 0 0 .5 8 5 .2 8 5 .0 4 5 .1 9 Sam ple I I (C & T) : 139 9 8 .8 3 5 .1 4 4 .09 5 .1 2 T o t a l : 267 9 9 .67 5 .2 0 5 .0 1 T . .15 a 0 c t o b c r f 1959; N a t i o n a l mean a t tim e o f t e s t i n g : 5 .1 31 s e c t i o n s o f t h e SRA. F i f t h g r a d e c l a s s e s w e r e s e l e c t e d f o r t h i s s t u d y a s t h i s I s t h e m e d ia n g r a d e f o r w h ic h t h e s e t e s t s w e re d e s i g n e d . The c o m m u n ity w h ic h t h e s e s c h o o l s s e r v e w o u ld a p p e a r t o c o n s i s t p r i m a r i l y o f p e r s o n s o f u p p e r - l o w e r and l o w e r - m i d d l e s o c i o e c o n o m i c s t a t u s , as s u b j e c t i v e l y e v a l u a t e d by t h e w r i t e r . The e v a l u a t i v e I n s t r u m e n t s The a r i t h m e t i c t e s t s o f two w i d e l y u s e d s t a n d a r d i z e d a c h i e v e m e n t b a t t e r i e s w e re s e l e c t e d f o r t h i s s t u d y , t h e C a l i f o r n i a A c h ie v e m e n t T e s t s (CAT), fo rm s W and X, and t h e STtA A c h ie v e m e n t S e r i e s (SRA) , fo rm s A a n d B. The a r i t h m e t i c s e c t i o n s w e re s e l e c t e d f o r t h i s s t u d y f o r th e f o l l o w i n g r e a s o n s : ( l ) t h e r e i s g e n e r a l a g r e e m e n t r e g a r d in g t h e i r c o n t e n t v a l i d i t y , (2 ) a h i g h d e g r e e o f o b j e c t i v i t y i s fo u n d i n t h e t e s t s i n b o t h m u l t i p l e c h o i c e an d r e c a l l f o r m a t s , an d ( 3 ) th e " f a c e " a t t r a c t i v e n e s s o f t h e r e s p o n s e o p t i o n s w ould a p p e a r to be more n e a r l y e q u a l t h a n t h o s e o f o t h e r a r e a s o f a c h i e v e m e n t . Some c o m p a r i s o n s o f t h e s e two t e s t s fro m v a r i o u s s o u r c e s a r e p r e s e n t e d i n T a b l e s 4 and 5 . The v a l i d i t y o f b o th a c h i e v e m e n t s e r i e s d e p e n d s p r i m a r i l y on c o n t e n t v a l i d i t y . The CAT p u r p o r t s c o n s t r u c t v a l i d i t y i n t h e form o f c o r r e l a t i o n s w i t h two o t h e r a c h i e v e m e n t s e r i e s and w i t h T A B L E lu —A comparison of the arithm etic sectio n s of the C A T and SRA,* elem entary b a tte r ie s (6 ,5 1 ,7 6 ) Test C h a r a c te r is tic Source CAT Reas. Fund. Reas. SR A Concodts Cone. R e lia b ility : (CATb : K R 21; SRA: K R U; N * 200 (6,76) .79 .70 .877 .819 .813 K R 21—CAT, N s 1U3; SRA,® ii = 108; grade 5, (51) .81 .73 .85 .55 .78 K R 21—CAT, SRA: N r 161:, same s u b je c ts (51) .78 .8 0 .8 8 .72 .81* Number of item s: 1 * 5 80 1 * 9 20 1 * 1 * Number of ra tio n s per item : 5C 5C 1 * 1 * 1 * Time lim it (m in .): 16 1 * 2 60 60 Range: CAT, N t lU3> SRA, N ■ 108; grade 5; (51) la-39 6-U 1 1-4*6 3-18 2-35 i.:ean/number of item s (above s a n p le ): .52 .33 . 1 * 6 .52 .32 Grade 6 , N = 161*, same s u b je c ts (51) .59 .1*3 .55 .59 . 1 * 6 ^ean G.P. v alu e of above sam ple: (51) 5.8 5.9 5.6 5 .5 5.1* a Form A of the SR A s e r ie s , and form W of the C A T are in p lie d unless otherw ise n otated. bForm not s p e c ifie d . > -4 U cFive item s have only four o p tio n s. 1 0 33 TABLE 5 - I n t e r - c o r r e l a t i o n s among t h e CAT and SRA A r i t h m e t i c T e s t s s R A S o u r c e R e a s . *4000 C o n e . rw<» Comp. CAT R e a s.® ( 5 1 ) b .70 .86 .70 . to o .58 .72 CAT Fund.® (51 ) b .68 .9 6 .58 1 . 0 1 C .73 1 . 0 6 ° SRA R e a s . (76) • « e • # • .69 .81 .6 8 .75 SRA Cone. (76) • • • # • • • • • • • • .65 o C D a I n t e r c o r r e l f i t i o n b e tw e e n CAT K e a s . and CAT F und, n o t r e p o r t e d ( 6 , 7 6 ) . b N » 164 p u p i l s in g r a d e s i x . c One o r b o t h r e l i a b i l i t y e s t i m a t e s w e re s p u r i o u s l y low . 34 t h e CTMM ( 7 8 : 1 2 ) . C o n s t r u c t v a l i d i t y i s r e p o r t e d t o r t h e SRA a s a r e s u l t o f a f a c t o r a n a l y s i s w h ic h w as made a t e a c h g r a d e l e v e l ( 7 6 : 2 0 ) . A l l t e s t s h a v e h e a v y l o a d i n g s i n t h e g e n e r a l a c h i e v e m e n t f a c t o r . A l l a r i t h m e t i c t e s t s a l s o h a v e s u b s t a n t i a l l o a d i n g s i n t h e q u a n t i t a t i v e a c c u r a c y an d p r i n c i p l e s f a c t o r , a f a c t o r n o t p o s s e s s e d by t h e o t h e r t e s t s . The a r i t h m e t i c c o m p u t a t i o n t e s t a l s o h a s a s l i g h t l o a d i n g in t h e s t r u c t u r a l l a n g u a g e f a c t o r . P r e d i c t i v e v a l i d i t y i s r e p o r t e d f o r t h e SRA s e r i e s i n t h e fo rm o f c o r r e l a t i o n s o f t e s t s c o r e s i n g r a d e e i g h t w i t h s c h o o l m arks i n g r a d e n i n e . The v a l i d i t y c o e f f i c i e n t s w e re h ig h f o r t h i s t y p e o f p r e d i c t i o n . The f i r s t s e m e s t e r g r a d e p o i n t a v e r a g e iti g r a d e n i n e c o r r e l a t e d .77 w i t h t h e c o m p o s i t e s c o r e fro m t h e SRA b a t t e r y ( 7 6 : 3 3 ) . Of s e p a r a t e a c h i e v e m e n t a r e a s , t h e a r i t h m e t i c s e c t i o n s c o r r e l a t e d m o st h i g h l y w i t h g r a d e p o i n t a v e r a g e . From T a b le 4, page 3 1, i t c a n be s e e n t h a t t h e r e l i a b i l i t y c o e f f i c i e n t s o f t h e SRA s e r i e s w e re g e n e r a l l y s l i g h t l y h i g h e r t h a n t h o s e r e p o r t e d f o r t h e CAT, w i t h t h e e x c e p t i o n o f t h e SRA c o n c e p t s t e s t . The SRA s e r i e s , how e v e r , r e q u i r e s more t h a n t w i c e t h e a d m i n i s t r a t i o n t i m e , y e t h a s o n l y 90.41$ as many i t e m s . The r e l i a b i l i t i e s w o u ld a p p e a r to be c o m p a r a b l e in te r m s o f r e l a t i v e a d m i n i s t r a t i o n t i m e . The r e a s o n i n g s e c t i o n s of b o th t e s t s seem ed to h a v e som ew hat h i g h e r r e l i a b i l i t i e s t h a n t h e c o m p u t a t i o n a l 35 ( f u n d a m e n t a l s ) s e c t i o n s . T h is p r o b a b l y r e s u l t e d f ro m t h e f a c t t h a t t h e s e s e c t i o n s on b o t h t e s t s w e re t o o d i f f i c u l t t o s e c u r e maximum r e l i a b i l i t y . Lord (4 9 ) d e m o n s t r a t e d t h a t t h e r e l i a b i l i t y o f a t e s t w i l l b e g r e a t e s t when I te m d i f f i c u l t y was som ew hat e a s i e r t h a n t h e m i d p o i n t b e t w e e n 1 . 0 and c h a n c e . The optimum d i f f i c u l t y l e v e l f o r t h e s e t e s t s w ould th e n be i n t h e n e i g h b o r h o o d o f .6 2 ; b u t , as shown in T a b le 4, p ag e 31, t h e mean d i f f i c u l t y l e v e l s on t h e c o m p u t a t i o n t e s t s w ere o n l y a p p r o x i m a t e l y o n e - h a l f t h i s v a l u e a t g r a d e f i v e . T h e se t e s t s w o u ld a p p e a r to be p a r t i c u l a r l y v u l n e r a b l e t o e f f e c t s from c h a n c e and response s e t s . Many ite m s on th e CAT f u n d a m e n t a l s t e s t s w e re found to c o n t r i b u t e l i t t l e , as i n d i c a t e d by th e t e s t c e i l i n g o f t h i r t y - s i x it e m s w i t h a s i x t h g r a d e s a m p le ( 5 1 ) . The l e v e l o f p e r f o r m a n c e r e q u i r e d to o b t a i n a p a r t i c u l a r g r a d e p l a c e m e n t v a l u e on t h e two a c h i e v e m e n t s e r i e s a p p e a r e d t o d i f f e r , w i t h th e SRA r e q u i r i n g a h i g h e r a b i l i t y l e v e l t o a c h i e v e th e same g r a d e p l a c e m e n t s t a t u s . T h is norm d i f f e r e n c e , a s i t r e l a t e s t o th e p r e s e n t s t u d y , w i l l be d i s c u s s e d i n g r e a t e r d e t a i l in C h a p t e r IV, t o g e t h e r w ith a d d i t i o n a l c o m p a r a t i v e e v a l u a t i o n s . By u s i n g t h e s e p a r t i c u l a r a c h i e v e m e n t s e r i e s i t was a l s o p o s s i b l e to I n v e s t i g a t e t h e e f f e c t s from t h e u s e o f t h e a l t e r n a t i v e " d o n ' t know" o p t i o n . Such an o p t i o n i s i n c o r p o r a t e d i n t o t h e CAT t e s t s b u t n o t on t h e SRA e l e m e n t a r y s e r i e s ( a l t h o u g h u sed on o t h e r b a t t e r y l e v e l s o f t h e 36 SRA). The q u e s t i o n s o f w h e t h e r c h a n c e e f f e c t s a r e r e d u c e d and i f a c o r r e s p o n d i n g i n c r e a s e I n v a l i d i t y i s e v i d e n c e d w i l l he ex am in ed i n C h a p t e r IV. The c r i t e r i o n ' A b i l i t y t o do t h e k i n d s o f a r i t h m e t i c one i s l i k e l y t o h av e t o do i n l a t e r l i f e ' i s one . . . im m e d i a t e o b j e c t i v e w hich i s much more e a s i l y d e r i v e d from u l t i m a t e o b j e c t i v e s th a n a r e m ost o t h e r s . (46:654) The c r i t e r i o n m e a s u re u s e d i n t h i s i n v e s t i g a t i o n was a t e a t from t h e same c o n t e n t p o p u l a t i o n ( a l t e r n a t i v e form ) i n a r e c a l l f o r m a t . C ro n b a c h s t a t e d : "The b e s t c r i t e r i o n f o r a t e s t w i l l o f t e n be some o t h e r t e s t m e a s u r in g t h e same a t t r i b u t e , p a r t i c u l a r l y In t h e c a s e o f p r o f i c i e n c y o r a c h i e v e m e n t t e s t s " ( 1 7 ) . G u i l f o r d ( 4 0 :3 8 6 ) p o i n t e d o u t t h a t a c h i e v e m e n t e x a m i n a t i o n s a r c u s u a l l y r e g a r d e d as s e l f - v a l i d a t i n g and s e r v e a s t h e i r own c r i t e r i a . T hese f a c t s make i n f o r m a t i o n r e g a r d i n g t h e c o r r e l a t i o n b e tw een t e s t s from th e sa;ie c o n t e n t p o p u l a t i o n i n r e c a l l and r e c o g n i t i o n f o r m a t s to be h i g h l y d e s i r a b l e . By c o m p a rin g a l t e r n a t e form r e l i a b i l i t y w i t h c o i n c i d e n t v a l i d i t y , th e v a r i a n c e from c h a n c e and r e s p o n s e s e t s c o u ld be a s s e s s e d . A d m i n i s t r a t i o n p r o c e d u r e s A ll t e s t s i n t h i s s t u d y w ere a d m i n i s t e r e d by t h e w r i t e r t o c o n t r o l p o s s i b l e " p a c i n g " e f f e c t s , m e n tio n e d by T r a x l e r ( 4 8 : 3 5 1 ) , o r o t h e r v a r i a b l e s i n t r o d u c e d by th e r e g u l a r c l a s s r o o m t e a c h e r a s a t e s t a d m i n i s t r a t o r . The 37 t e a c h e r s o f t h e e x p e r i m e n t a l c l a s s e s w e re c o n t a c t e d t h e week p r i o r t o t e s t i n g and I n f o r m e d t h a t t h e i r c l a s s e s w ould be n e e d e d f o r some e x p e r i m e n t a l t e s t i n g by t h e d i s t r i c t ; a l l t e a c h e r s c o o p e r a t e d . A l l t h e t e s t s w e re g i v e n in t h e a u d i t o r i u m o f t h e s t u d e n t s ' r e s p e c t i v e s c h o o l s . No more t h a n two c l a s s e s w ere t e s t e d s i m u l t a n e o u s l y . I m m e d i a t e l y b e f o r e t h e t e s t i n g p e r i o d , t h e s t a n d a r d i z e d a d m i n i s t r a t i o n p r o c e d u r e s w ere d i s c u s s e d w i t h t h e c l a s s r o o m t e a c h e r s who t h e n s e r v e d a s p r o c t o r s . The t e s t i n g r e q u i r e d t w e l v e c o n s e c u t i v e s c h o o l d a y s i n A p r i l , 1 9 6 0 . The c r i t e r i o n m e a s u r e , an a l t e r n a t e form o f t h e same t e s t i n r e c a l l f o r m a t , was a d m i n i s t e r e d one week f o l l o w i n g t h e i n i t i a l t e s t , w i t h d a y - o f - w e e k and t i m e - o f - d a y c o n t r o l l e d as n e a r l y as p o s s i b l e . The t e a c h e r s w ere i n f o r m e d a s to t h e t r u e p u r p o s e o f t h e t e s t i n g o n l y a f t e r a l l d a t a had been c o l l e c t e d . P r i o r t o t h e tim e t h e s t u d e n t s came t o t h e a u d i t o r i u m f o r t e s t i n g , t h e n e c e s s a r y m a t e r i a l s w e re d i s t r i b u t e d , w i t h a l t e r n a t e form s o f t h e same t e s t p l a c e d on e a c h s i d e o f t h e c e n t e r a i s l e . As t h e c l a s s e s e n t e r e d , th e b o y s and g i r l s l i n e d - u p in s e p a r a t e l i n e s and were a s s i g n ed a l t e r n a t i n g l y t o e a c h a i d e o f t h e a i s l e . In t h i s way a random a s s i g n m e n t to t h e d i f f e r e n t t e s t fo rm s was made w i t h c l a s s and s e x c o n t r o l l e d . A l l t e s t s f o r a p a r t i c u l a r s c h o o l w e re c o m p l e t e d in one d a y . W ith t h e SRA t e s t s , t h e r e a s o n i n g s e c t i o n was 38 a lw a y s g i v e n i n t h e m o rn in g s e s s i o n and t h e c o n c e p t s and c o m p u t a t i o n s s e c t i o n s lri t h e a f t e r n o o n . S t a n d a r d i z e d a d m i n i s t r a t i o n p r o c e d u r e s w ere f o l lowed u n t i l t h e t i m e l i m i t had e x p i r e d . The s t u d e n t s who s t i l l h ad n o t c o m p l e t e d t h e t e s t s w ere t h e n g i v e n r e d pen c i l s and a l l o w e d a s much tim e a s n e c e s s a r y to c o m p l e t e t h e e n t i r e t e s t s . S c o r i n g The f o l l o w i n g t y p e s o f s c o r e s w e re o b t a i n e d f o r e a c h i n d i v i d u a l : ( l ) num ber r i g h t ( a t : \ n d a r d i z e d p r o c e d u r e ) , (2 ) num ber r i g h t , c o r r e c t e d f o r c h a n c e , (3) c h a n c e - t h r e s h o l d ( s e e d e f i n i t i o n o f t e r m s ) , (4) 3 -4 c u t o f f , (5) 3 - 3 c u t o f f , (6 ) 4 - 4 c u t o f f , and (7 ) number r i g h t , f o r c e d c o m p l e t i o n . In a d d i t i o n , a s c o r e on t h e CAT was o b t a i n e d , c o r r e c t e d f o r c h a n c e o n l y on t h o s e i t e m s n o t m arked i n t h e " d o n ' t know" o p t i o n . A r i s k - a c c e p t a n c e i n d e x , p r o p o s e d by Z i l l e r ( 8 5 ) , was o b t a i n e d f o r e;tch ex a m in e e t o a s c e r t a i n g a m b lin g t e n d e n c i e s . The p o s i t i o n a l p r e f e r e n c e s e t was e v a l u a t e d by t a k i n g t h e e n t i r e s a m p le front one s c h o o l (N = 93) and d e t e r m i n i n g w h e t h e r t h e i r r e s p o n s e s were d i s t r i b u t e d a o c o r d i n g t o c h a n c e p r o b a b i l i t y on t h e i t e m s on w h ic h t h e y a n s w e re d i n c o r r e c t l y . By c o m p a r in g t h e e x p e c t e d f r e q u e n c i e s f o r e a c h opt io n and c o m p a r in g t h i s w i t h t h e o b s e r v e d f r e q u e n c i e s , d i f f e r e n c e s w e re e v a l u a t e d f o r s i g n i f i c a n c e . I t was a ssu m ed t h a t t h e r e l a t i v e a t t r a c t i v e n e s s o f t h e d i s t r a c t o r s was d i s t r i b u t e d ra n d o m ly among t h e f o u r o p t i o n p o s i t i o n s t h r o u g h o u t t h e t e s t s . The SRA t e s t s w e re u s e d t o t e s t t h e p o s i t i o n a l p r e f e r e n c e h y p o t h e s e s s i n c e t h e a d d i t i o n a l v a r i a b l e o f t h e " d o n ' t know" o p t i o n w o u ld h a v e b e e n o p e r a t i n g on t h e CAT. L o w er-b o u n d e s t i m a t e s o f r e l i a b i l i t y w ere fo u n d by u s i n g t h e f o r m u l a p r o p o s e d by G u l l i k s e n ( 3 9 ) . The v a r i o u s v a l i d i t y c o e f f i c i e n t s w ere c o r r e c t e d f o r a t t e n u a t i o n i n o r d e r t o o b t a i n r e l e v a n c e e s t i m a t e s . S i g n i f i c a n c e l e v e l The w r i t e r a c c e p t e d a 5^ r i s k o f m aking a Type I e r r o r ; c o n s e q u e n t l y , t h i s s i g n i f i c a n c e l e v e l was s e t as t h e maximum p r o b a b i l i t y l e v e l a t w hich t h e n u l l h y p o t h e s e s w ould be r e j e c t e d . CHAPTER IV FINDINGS The p u r p o s e o f t h i s c h a p t e r was t o p r e s e n t t h e f i n d i n g s o f t h e s t u d y a n d to make a n e v a l u a t i o n o f t h e h y p o t h e s e s r e l a t i v e t o t h e s e f i n d i n g s . H y p o t h e s i s I B e f o r e an i n v e s t i g a t i o n i s made i n t o t h e p r o b le m o f r e s p o n s e s e t s , I t s h o u l d be e s t a b l i s h e d t h a t t h e t e s t s u s e d a r e e s s e n t i a l l y power t e s t s . I f s p e e d v a r i a n c e i s p r e s e n t i n t h e s c o r e s , t h e p ro b le m o f s e p a r a t i n g s p e e d v a r i a n c e fro m r e s p o n s e s e t v a r i a n c e w ould be c o m p le x ; t h e v a r i a b l e s o f t h e s e t to g u e s s , s p e e d v e r s u s a c c u r a c y s e t , and t r u e i n d i v i d u a l d i f f e r e n c e s in r e s p o n s e s p e e d w ould a l l be o p e r a t i n g s i m u l t a n e o u s l y . I f s p e e d v a r i a n c e i s n e g l i g i b l e , t h e n n e i t h e r o f t h e l a t t e r two f a c t o r s becom es an i m p o r t a n t c o n s i d e r a t i o n . The i n f l u e n c e o f s p e e d was e x a m in e d i n t h r e e w a y s: 1 . C hance s u c c e s s on t h e mean num ber o f u n a t t e m p t e d i t e m s a t t h e end o f t h e t i m e - l i m i t s was a d d e d t o t h e mean n u m ber o f c o r r e c t r e s p o n s e s and t h e n c o m p a re d w i t h t h e mean s c o r e o f t h e f o r c e d c o m p l e t i o n m e th o d . T a b le 6 shows t h a t no g r e a t e r t h a n c h a n c e s u c c e s s was o b t a i n e d when a l l e x a m i n e e s w ere a l l o w e d to r e s p o n d to e v e r y i t e m . The e f f e c t s from s p e e d , a s e v a l u a t e d by t h i s 41 TABLE 6 . — An e x a m i n a t i o n o f t h e i n f l u e n c e o f s p e e d on t h e CAT and SRA A r i t h m e t i c T e s t s CAT T o t a l SRA R e a s . SRA Comp. Mean n u m ber o f i t e m s a t t e m p t e d : 1 1 3 .1 9 8 4 6 .5 2 0 4 1 .9 9 2 Mean s c o r e , s t a n d a r d tim e l i m i t s : 5 8 .2 0 7 2 4 . 0 2 4 1 8 .2 2 4 Mean s c o r e , c h a n c e c r e d i t g i v e n f o r u n a t t e m p t e d i t e m s : 6 0 .5 6 7 2 4 .6 4 4 1 8 .7 2 6 Mean s c o r e , f o r c e d c o m p l e t i o n : 6 0 .5 7 9 2 4 .6 6 4 1 8 .6 2 4 z - r a t i o f o r d i f f e r e n c e b e tw e e n m e a n s : .112 C o r r e l a t i o n b e t w e e n t i m e - l i m i t s c o r e s , c o r r e c t e d f o r c h a n c e , and f o r c e d c o m p l e t i o n s c o r e s ® : .987 .987 .968 R e l i a b i l i t y , KH 21: .841 .839 .7 9 4 R e l i a b i l i t y , G u l l i k s e n ' s l o w e r - b o u n d : .8 5 5 .869 .7 9 5 a N o n i n d e p e n d e n t c o e f f i c i e n t s . 42 ■ • t h o d , w ould seem t o be r e l a t i v e l y I n s i g n i f i c a n t , 2 , A c o m p a r i s o n o f c o r r e c t e d s c o r e s and s c o r e s u s i n g t h e f o r c e d c o m p l e t i o n m ethod was m ade. S i n c e t h e s e c o r r e l a t i o n s a r e e x t r e m e l y h i g h , as shown by T a b le 6, t h e two t e s t s would be c o n s i d e r e d t o be m e a s u r i n g e s s e n t i a l l y t h e same q u a l i t i e s . 3 . R e l i a b i l i t y c o e f f i c i e n t s o f t h e c o n v e n t i o n number r i g h t s c o r e , u s i n g G u l l i k s e n ' s (3 9 ) lo w e r - b o u n d e s t i m a t e , w ere co m p ared w i t h c o e f f i c i e n t s y i e l d e d by t h e K u d e r - R i c h a r d s o n f o r m u l a 2 1 . S i n c e t h e K u d e r - R i c h a r d s o n f o r m u l a i s i n f l a t e d t o t h e e x t e n t t h a t a t e s t i s s p e e d e d , i f t h e two c o e f f i c i e n t s sho'wed s m a ll d i f f e r e n c e s , t h e t e s t s w ould be assum ed to be e s s e n t i a l l y power t e s t s . T a b le 6 shows t h a t t h e K u d e r - R i c h a r d s o n c o e f f i c i e n t s a r e n o t l a r g e r , in any t e s t , t h a t t h e c o r r e s p o n d i n g l o w e r - bound e s t i m a t e . On t h e b a s i s o f t h e s e r e s u l t s , h y p o t h e s i s I i s a c c e p t e d and i t i s c o n c l u d e d t h a t t h e t e s t s a r e e s s e n t i a l l y power t e s t s . H y p o t h e s i s I I To d e t e r m i n e w h e t h e r t h e s e t to gam ble was p r e s e n t i n e l e m e n t a r y s c h o o l p u p i l s , Z i l l e r * s g a m b lin g i n d e x was co m puted f o r e a c h s t u d e n t . The r e s u l t s a r e g i v e n i n T a b le 7 . This- i n d e x r e p r e s e n t s th e r a t i o o f t h e num ber o f i t e m s on w h ic h t h e s u b j e c t g u e s s e d t o t h e t o t a l num ber o f i t e m s n o t known, b u t upon w h ic h he c o u l d h a v e s u e s s e d . 43 TABLE 7 R i s k - a c c e p t a n c e d a t a f o r t h e CAT and SllA A r i t h m e t i c T e s t a CAT® SRA® P Mean i n d e x o f r i s k - a c c e p t a n c e : .6 1 4 .9 3 0 < .o ib M edian i n d e x o f r i s k - a c c e p t a n c e : .89 1 .0 0 0 Range o f r i s k - a c c e p t a n c e i n d e x e s : . 0 4 - 1 . 0 0 . 12-1 o c • P e r c e n t o f e x a m i n e e s g u e s s i n g on a l l i t e m s n o t known ( r i s k - a c c e p t - a n c e s o f 1 . 0 0 ) : 2 8 . 1 ^ 5 3 . 6 £ R e a s o n i n g and f u n d a m e n t a l s ( c o m p u t a t i o n ) s e c t i o n s c o m b i n e d . bChi s q u a r e e q u a l e d 3 3 . 7 9 . 44 An I n d e x o f . 9 0 , f o r e x a m p le , w o u ld i n d i c a t e t h a t t h e s t u d e n t g u e s s e d on 9 0 # o f t h e i t e m s he d i d n o t know . S i n c e t h e CAT I n s t r u c t s e x a m in e e s t o mark t h e a l t e r n a t i v e " d o n ' t know” o p t i o n when th e y c a n n o t w o rk a p r o b le m , r e s p o n s e s i n t h a t colum n w e re n o t s c o r e d a s g u e s s e s . T h is no d o u b t a c c o u n t e d f o r t h e l o w e r r i s k - a c c e p t a n c e i n d e x e s o b t a i n e d on th e CAT. The SRA t e s t s g i v e no i n s t r u c t i o n s r e g a r d i n g g u e s s i n g , b u t i t a p p e a r s t h a t m o st e x a m in e e s u s u a l l y g u e s s e d when t h e y c o u l d n o t w ork a p r o b l e m . T a b l e 7 show s t h a t h y p o t h e s i s I I i s s u p p o r t e d and t h a t t h e r e w as, i n e l e m e n t a r y s c h o o l p u p i l s , a s t r o n g s e t t o g am b le when u n s u r e o f an a n s w e r . T h i s t e n d e n c y was s i g n i f i c a n t l y d e c r e a s e d by u s i n g t h e " d o n ' t know" a l t e r n a t i v e . Even when t h i s o p t i o n was p r e s e n t , h o w e v e r, t h e meat] e x a m in e e g a m b le d on 6 1 . 4 # o f it e m s n o t known. I n h e r e n t i n t h e g a m b l i n g i n d e x and i n t h e s e c o n c l u s i o n s i s t h e a s s u m p t i o n t h a t a l l i n c o r r e c t r e s p o n s e s a r e g u e s s e s and t h a t c h a n c e s u c c e s s i s o b t a i n e d on a l l ite m s on w h ic h t h e s u b j e c t g u e s s e d . H y p o t h e s i s I I I The CAT p r o v i d e d o p p o r t u n i t y t o e x a m in e t h e r e l a t i o n s h i p b e tw e e n a b i l i t y and t e n d e n c y t o f o l l o w d i r e c t i o n s by o b s e r v i n g how s u b j e c t s r e s p o n d e d a f t e r r e a c h i n g t h e i r c h a n c e t h r e s h o l d . The c r i t e r i a f o r " f o l l o w i n g 45 d i r e c t i o n s " and f o r " n o t f o l l o w i n g d i r e c t i o n s " w e re a r b i t r a r i l y e s t a b l i s h e d a s f o l l o w s : 1 . A minimum o f s i x i t e m s m u st h a v e r e m a in e d a f t e r t h e c h a n c e t h r e s h o l d h ad b e e n r e a c h e d i n o r d e r f o r a p u p i l t o be i n c l u d e d i n e i t h e r c a t e g o r y . 2 . I f an e x a m in e e had a t l e a s t s i x , and g r e a t e r t h a n 5 0 ^ , o f h i s i n c o r r e c t r e s p o n s e s i n t h e " d o n ' t know" o p t i o n , h e was i n c l u d e d i n t h e " f o l l o w i n g d i r e c t i o n s " , g r o u p . S i n c e t h e r e l a t i o n s h i p b e tw e e n a b i l i t y and f o l l o w i n g d i r e c t i o n s was b e i n g e x a m in e d , t h e s a m p le was d i v i d e d i n t o two g r o u p s a b o v e and below t h e m edian and t h e f r e q u e n c i e s o f f o l l o w i n g d i r e c t i o n s w ere t a b u l a t e d . The CAT c o n t a i n s s e v e n s u b t e s t s ; t h e r e f o r e , i t was p o s s i b l e to s u b d i v i d e t h e a b o v e g r o u p s f u r t h e r i n t o t h e f o l l o w i n g : (1 ) t h o s e who c o n s i s t e n t l y f o l l o w e d d i r e c t i o n s (N • 2 9 ) , (2 ) t h o s e who f o l l o w e d d i r e c t i o n s p a r t o f th e t i m e ( f o l lowed d i r e c t i o n s 011 a l l e x c e p t one o r two o f t h e s u b t e s t s on w h ic h t h e r e was o p p o r t u n i t y t o o b s e r v e s u c h b e h a v i o r , N = 5 1 ) , and ( 3 ) t h o s e who d i d n o t f o l l o w d i r e c t i o n s ( o b s e r v e d o p p o r t u n i t i e s to f o l l o w d i r e c t i o n s e x c e e d e d o b s e r v e d d i r e c t i o n s f o l l o w i n g by t h r e e o r m ore, N = 4 1 ) . The c h i s q u a r e v a l u e f o r t h e 2 x 3 c o n t i n g e n c y t a b l e was 1 4 . 1 1 . A v a l u e o f 9 . 2 1 0 was r e q u i r e d f o r s i g n i f i c a n c e a t t h e .01 l e v e l o f c o n f i d e n c e . The c o e f f i c i e n t o f c o n t i n g e n c y was fo u n d to be . 3 2 3 . H y p o t h e s i s I I I was 46 a c c e p t e d and i t was c o n c l u d e d t h a t t h e r e was a d e f i n i t e r e l a t i o n s h i p b e tw e e n a b i l i t y a n d t e n d e n c y t o f o l l o w d i r e c t i o n s . A lth o u g h t h i s i n d i c a t e s a d i f f e r e n t i a l s e t t o n e g a t e t h e " d o n ’t know" o p t i o n , t h e o v e r a l l e f f e c t s f r o n t h e u s e o f t h i s o p t i o n w e re b e n e f i c i a l to v a l i d i t y ( s e e T a b le 12, p age 5 4 ) . n y p o t h e s i s IV I t has been stiown t h a t c e r t a i n s t a n d a r d i z e d t e s t s have s y s t e m a t i c b i a s e s i n t h e k e y i n g o f c o r r e c t r e s p o n s e s ( 5 5 ) . I t was n e c e s s a r y t o e s t a b l i s h th e n r e s e n c e o r a b s e n c e o f s u c h b i a s e s on th e p r e s e n t t e s t s so t h a t , i f n e c e s s a r y , c o r r e c t i v e m e a s u re s c o u l d be ta k e n when e x a m in ing t h e p o s i t i o n a l p r e f e r e n c e h y p o t h e s e s . The r e s u l t s o f t h i s e x a m i n a t i o n a r c r i v e n in T a b le 8 . T h r e e o f t h e t h i r t e e n t e s t s were shown to h ave p o s i t i o n b i a s e s in t h e k e y i n g of th e c o r r e c t o p t i o n . The s i g n i f i c a n t ( .0 1 l e v e l ) k e y i n g p a t t e r n on t h e CAT A r i th m e t i c F u n d a m e n ta ls T e s t p r o b a b l y r e s u l t e d from an a t t e m p t to i n c r e a s e t e s t v a l i d i t y by u s i n g t h e a l t e r n a t i v e " d o n ' t know" o p t i o n . When t h i s o p t i o n was i g n o r e d , s i g n i f i c a n c e was no l o n g e r r e a c h e d . The o t h e r two t e s t s w i t h s i g n i f i c a n t ( . 0 5 l e v e l ) b i a s e s were th e SRA R e a d in g V o c a b u l a r y T e a t . w h ic h had a d i s p r o p o r t i o n a t e l y l a r g e number o f c o r r e c t a n s w e r s in o p t i o n one, and t h e SRA L anguage A r t s T e s t , w h ic h had f e w e r c o r r e c t a n s w e r s i n t h e l a s t ( t h i r d ) 47 TABLE 8 . — The n u m b er o f r e s p o n s e s k e y e d c o r r e c t a t e a c h o p t i o n p o s i t i o n f o r t h e C a l i f o r n i a A c h ie v e m e n t T e s t s and t h e SKA A c h i e v e m e n t S e r i e s T e s t R e s p o n s e O p t i o n 1 2 3 4 5 C hi S q u a r e C a l i f o r n i a A c h ie v e m e n t T e s ts ® R e a d i n g V o c a b u l a r y 12 12 13 13 • • .8 0 0 R e n d i n g C o m p r e h e n s io n 17 18 16 19 • # .2 8 5 A r i t h m e t i c R e a s o n i n g o m i t t i n g "DK" o p t i o n 10 9 13 9 4 4 . 6 6 7 1 .0 4 2 A r i t h m e t i c f u n d a m e n t a l s o m i t t i n g "UK" o p t i o n 18 21 21 18 o 1 5 . 8 7 5 ° .3 1 5 M e c h a n ic s o f E n g l i s h o m i t t i n g "DK" o p t i o n 3 2 d 37d 12 13 12 1 .391 1 .4 2 7 S p e 1 1 i n g o m i t t i n g "DK" o p t i o n 5 9 7 7 2 4 .9 9 9 1 .1 1 1 SRA A c h i e v e m e n t S e r i e s 0 lY o rk -S tu d y S k i l l s 21 19 19 23 m • .5 3 7 R e a d in g V o c a b u l a r y 22 8 7 13 m • 11 , 2 8 0 e T i e n d i n r C o m p r e h e n s io n 13 16 11 10 • • 1 .5 2 0 L a n g u a g e A r t s 48 53 2 9 • • • * 7 . 4 5 3 ° A r i t h m e t i c R e a s o n i n g 14 10 13 12 • • 1 .123 A r i t h m e t i c C o n c e p t s 4 7 6 3 • • 2 .0 0 0 A r i t h m e t i c C o m p u t a t i o n 13 14 9 8 • « 2 .3 6 4 E l e m e n t a r y b a t t e r y , a l l f o r m s . A l t o r n a t i v e " d o n ’ t know" o p t i o n . c S i g n i f i c a n t a t t h e .0 1 l e v e l . ^ E x p e c t e d f r e q u e n c y 1 7 .5 g r e a t e r th a n f o r o t h e r o p t i o n s . S i g n i f i c a n t a t t h e .05 l e v e l . 48 r e s p o n s e p o s i t i o n . I t ia c o n c lu d e d t h a t h y p o t h e s i s IV was g e n e r a l l y n o t su p p o r te d and t h a t k e y in g b i a s e s were found o n l y i n a m i n o r it y o f t h e t e s t s com p osing t h e s e b a t t e r i e s . H y p o t h e s is V The SRA t e s t s w ere u s e d to d e t e r m i n e w h e t h e r I n t r a i n d i v i d u a l p o s i t i o n a l p r e f e r e n c e r e s p o n s e s e t s w e r e p r e s e n t i n t h e e l e m e n t a r y s c h o o l s a m p l e . The CAT was n o t u s e d s i n c e t h e a d d i t i o n a l v a r i a b l e o f t h e a l t e r n a t i v e " d o n ' t know" o p t i o n w ould a l s o be o p e r a t i n g on t h e s e t e s t s , i n t r o d u c i n g a m b i g u i t y i n t o t h e r e s u l t s . I t was n e c e s s a r y t o e x a m in e t h e p o s s i b i l i t y o f i n t r a i n d i v i d u a l p o s i t i o n a l p r e f e r e n c e s e t s , f o r i f s u c h s e t s e x i s t e d and w e re n o t t a k e n i n t o a c c o u n t , e r r o n e o u s c o n c l u s i o n s m ig h t he d raw n r e g a r d i n g t h e e x i s t e n c e o f a n i n t e r i n d i v i d u a l p o s i t i o n a l s e t . I f some e x a m in e e s had an a v e r s i o n to r e s p o n d i n g t o a c e r t a i n r e s p o n s e p o s i t i o n and o t h e r s had a s e t to a c q u i e s c e to t h a t p o s i t i o n , t h e o v e r a l l e v a l u a t i o n r e g a r d i n g t h e e x i s t e n c e o f a p o s i t i o n a l s e t w ould be n e g a t i v e u n l e s s t h e s e i n t r a i n d i v i d u a l s e t s w ere c o n s i d e r e d , The f o l l o w i n g p r o c e d u r e s w ere u s e d t o e v a l u a t e t h e i n t r a i n d i v i d u a l p o s i t i o n a l p r e f e r e n c e h y p o t h e s i s . 1 . I t was assum ed t h a t th e r e l a t i v e a t t r a c t i v e n e s s o f t h e d i s t r a c t o r s was d i s t r i b u t e d r a n d o m ly among t h e 49 ▼ a rio u s r e s p o n s e p o s i t i o n s t h r o u g h o u t t h e t e s t s . T h i s w o u ld seem t o be a r e a s o n a b l e a s s u m p t i o n s i n c e no k e y i n g b i a s e s w e re f o u n d on t h e s e p a r t i c u l a r t e s t s . 2 . The e n t i r e f i f t h g r a d e p o p u l a t i o n fro m one s c h o o l (N = 93) was u s e d t o e x a m in e t h i s h y p o t h e s i s . 3 . The s t u d e n t s ' i n c o r r e c t r e s p o n s e s w e re t a b u l a t e d w i t h r e g a r d t o t h e r e l a t i v e f r e q u e n c y p e r e a c h r e s p o n s e p o s i t i o n , n o t i n g t h e p o s i t i o n o f t h e c o r r e c t a n s w e r i n e a c h i t e m . E ach d i s t r a c t o r t h u s o b t a i n e d an e x p e c t e d f r e q u e n c y o f .33 f o r e a c h i n c o r r e c t r e s p o n s e . 4 . The s u m s o f e x p e c t e d and o b s e r v e d f r e q u e n c i e s w e re t a b u l a t e d f o r e a c h s t u d e n t and t e s t e d f o r s i g n i f i c a n c e u s i n g t h e K o lm o g o ro v - S m irn o v o n e - s a m p l e t e s t ( 6 7 : 4 7 ) . The f i n d i n g s a r e g i v e n i n T a b l e 9 . The r e s u l t s i n d i c a t e t h a t g e n e r a l i n t r a i n d i v i d u a l p o s i t i o n a l p r e f e r e n c e s w ere n o t p r e s e n t ; h y p o t h e s i s V was t h e r e f o r e r e j e c t e d . H y p o t h e s i s VI The K o lm o g o ro v - S m irn o v o n e - s a m p l e t e s t ( 6 7 : 4 7 ) was u s e d t o d e t e r m i n e w h e t h e r i n t e r i n d i v i d u a l p o s i t i o n a l p r e f e r e n c e s e x i s t e d i n e l e m e n t a r y s c h o o l p u p i l s . A l t h o u g h i t h a s b e e n e s t a b l i s h e d from h y p o t h e s i s V, t h a t i n t r a i n d i v i d u a l p o s i t i o n a l p r e f e r e n c e s w ere n o t p r e s e n t , i t c o u l d s t i l l be p o s s i b l e t h a t a g e n e r a l p o s i t i o n a l s e t e x i s t e d i n t h e s a m p l e , t o o s l i g h t t o be d e l e c t e d i n t h e i n d i v i d u a l c a s e . so To t e s t t h i s h y p o t h e s i s , t h e e x p e c t e d and o b s e r v e d f r e q u e n c i e s f o r e a c h r e s p o n s e p o s i t i o n w ere c o m b in e d . The r e s u l t s a r e g i v e n i n T a b le 1 0 . S i n c e no s i g n i f i c a n t d i f f e r e n c e was p r e s e n t , t h e h y p o t h e s i s o f i n t e r i n d l v l d u a l p o s i t i o n a l p r e f e r e n c e r e s p o n s e s e t s was r e j e c t e d . The c h i s q u a r e v a l u e I n d i c a t e d t h a t d i s c r e p a n c i e s b e tw e e n o b s e r v e d and e x p e c t e d f r e q u e n c i e s a s l a r g e o r l a r g e r t h a n t h a t found i n t h i s i n v e s t i g a t i o n w o u ld be p r e s e n t a p p r o x i m a t e l y 22% o f t h e tim e on t h e b a s i s o f c h a n c e a l o n e . One v a r i a b l e w hich a p p e a r s t o h av e b een u n i q u e w i t h r e g a r d t o o t h e r p o s i t i o n a l p r e f e r e n c e s t u d i e s was t h e s p a t i a l a r r a n < rem ent o f t h e r e s p o n s e o p t i o n s on t h e a n s w e r s h e e t u s e d . I n s t e a d o f th e u s u a l a r r a n g e m e n t of t h e a l t e r n a t i v e s b e in g i n a s i n g l e row, t h e o p t i o n s on t h e SRA a n s w e r s h e e t a r e a r r a n g e d i n a two row by two colum n c o n f i g u r a t i o n . C o n s e q u e n t l y any " c e n t r a l t e n d e n c y " o r i n c l i n a t i o n to w a rd a " n o rm a l d i s t r i b u t i o n " in t h e s t u d e n t s * r e s p o n s e p a t t e r n s would have been e l i m i n a t e d . H y p o t h e s i s V II The m a jo r a s r i e e t o f t h i s s t u d y d e a l t w i t h a com p a r i s o n o f v a r i o u s m ethods o f s c o r i n g t e s t s to d e t e r m i n e w hich m ost a d e q u a t e l y m in im iz e d t h e e f f e c t s from r e s p o n s e s e t s and c h a n c e . The C a l i f o r n i a A c h ie v e m e n t T e s t s . — The r e s u l t s from t h e u s e o f th e v a r i o u s s c o r i n g m ethods on t h e CAT a r e 51 TABLE 9 . — O b s e rv e d f r e q u e n c y o f i n t r n i n d i v i d u a l p o s i t i o n a l p r e f e r e n c e r e s p o n s e s e t s a t v a r i o u s l e v e l s o f c o n f i d e n c e L e v e l o f C o n f i d e n c e .3 0 o CM • .1 0 .05 .01 O b s e rv e d f r e q u e n c i e s 26 16 9 5 4 P e r c e n t o f s a m p le 28 .0 1 7 .2 9 . 7 5 .4 4 . 3 Da .121 <D.0S ■ .2 6 4 ) ^ o l r a o g o r o v - S m i r n o v o n e - s a m p l e t e s t . TAULV 10 . — O b s e rv e d and e x p e c t ed r r e q u o n c i c s wi t h w!i i ch v a r i o u s r e s pons e pos i t io n s w ere ch o s e n R e s p o n s e P o s i t i o n O b s e rv e d f r e q u e n c y 1, 183 1 f 086 1 , 199 1, 267 P e r c e n t o f r e s p o n s e s 24 .09 22 .9 4 2 5 .32 2 6 . 7 6 E x p e c t e d p e r c e n t o f r e s p o n s e s 24 .68 24 .21 25 .07 26 .0 4 Da .0096 ^ D -(15 = .1 4 1 ) n K o lm o g o ro v - S m irn o v o n e - s n m p l e t e s t . 62 g i v e n i n T a b l e s 11 and 1 2. The f i n d i n g s i n d i c a t e d t h a t t h e s c o r e s o b t a i n e d u s i n g t h e c o r r e c t i o n f o r c h a n c e , t h e i n c o r r e c t i t e m s m arked i n t h e " d o n ' t know" o p t i o n e x c e p t e d , y i e l d e d t h e h i g h e s t c o e f f i c i e n t o f c o i n c i d e n t v a l i d i t y ; s i g n i f i c a n t l y h i g h e r t h a n t h a t y i e l d e d by a n y o t h e r m ethod e x c e p t t h e 3 - 4 an d 4 - 4 c u t o f f p r o c e d u r e s . The r e l i a b i l i t y c o e f f i c i e n t y i e l d e d by t h i s m ethod was n o t e x c e e d e d by any o f t h e c o e f f i c i e n t s y i e l d e d by t h e o t h e r m e th o d s on w h ic h a c c u r a t e r e l i a b i l i t y d a t a w e re a v a i l a b l e . In i n t e r p r e t i n g t h e i n t e r c o r r e l a t i o n s b e tw e e n t h e v a r i o u s s c o r e s p r e s e n t e d i n T a b le 11, i t m ust b e re m e m b e r ed t h a t t h e s e c o e f f i c i e n t : w e re o b t a i n e d on t h e same s a m p le on t h e s a n e t e s t a t t h e same t i m e , and c o n s e q u e n t l y a r e n o t i n d e p e d n e n t . The v a l u e s a r e c o n s i d e r a b l y h i g h e r t h a n t h o s e w hich w o u ld be o b t a i n e d u s i n g two s e p a r a t e a d m i n i s t r a t i o n s . A l t h o u g h a c u r s o r y i n s p e c t i o n oi t h e v a r i o u s v a l i d i t y c o e i i i c i e n t s w ould l e a d one t o c o n c l u d e t h a t l i t t l e d i f f e r e n c e a c c r u e d r e g a r d l e s s or th e s c o r i n g m ethod u s e d , s i n c e t h e s e c o e f f i c i e n t s a r e c o r r e l a t e d t h e y c a n n o t be v ie w e d i n t h e s a n e r e l a t i v e te r m s o f c o m p a r i s o n as t h e more t y p i c a l l y r e p o r t e d i n d e p e n d e n t c o r r e l a t i o n s . I n o r d e r t o a p p r o x i m a t e t h e p r o b a b l e d i f f e r e n c e w h ic h would h a v e r e s u l t e d had t h e c o e f f i c i e n t s b e e n i n d e p e n d e n t , t h e f o l l o w i n ' ! e x a m p le i s g i v e n . An o b s e r v e d d i f f e r e n c e i n v a l i d i t y c o e f f i c i e n t s u s i n g t h e s t a n d a r d c o r r e c t e d s c o r e and t h e c o r r e c t e d s c o r e w i t h t h e 63 TABLE 1 1 , — C o r r e l a t i o n m a t r i x o f s c o r e s y i e l d e d by v a r i ous s c o r i n g m e th o d s on t h e CAT A r i t h m e t i c T e s t s , w i t h means and s t a n d a r d d e v i a t i o n s TT S c o r e XC Xe Xd A c5 Xf X3 - 4 Y 3— 3 X4 — 4 Xt C X 970 963 957 955 960 965 973 8528 X c 977 987 970 971 969 979 8578 Xc5 964 971 972 967 985 8962 Xf 958 959 958 969 8338 X3 -4 992 969 958 8839 X3 -3 977 961 8821 X4 -4 960 8617 Xt 8684 J 9961 Means 5 8 .2 44 .4 4 8 . 7 6 0 .6 5 0 .4 5 2 . 0 54 .5 5 1 .3 6 2 .3 o f Wrongs 5 5 . 0 2 8 . 7 S .D .s 1 3 .7 1 5 .3 1 5 .6 1 2.2 1 5 .5 15 .5 1 5 .7 1 5 .2 14 .6 o f Wrongs 1 5 .6 1 7 .6 D ecim al p o i n t s o m it t e d in c o r r e l a t i o n c o e f f i c i e n t s . X X. c5 3-4 l3-3 [4— 4 c o n v e n t i o n a l r ig h t s c o r e , sta n d a r d tim e l i m i t s c o r r e c t e d s c o r e u s i n g t h e o r e t i c a l chance c o r r e c t i o n . c o r r e c t e d s c o r e , i n c o r r e c t r e s p o n s e s in " d o n 't know" o p t i o n e x p e c t e d . f o r c e d c o m p le t i o n s c o r e . 3 -4 c u t o f f s c o r e . 3 -3 c u t o f f s c o r e . 4 -4 c u t o f f s c o r e , c h a n c e - t h r e s h o l d s c o r e , c r i t e r i o n . R c .x w C.XrW — : 5 5 .864 .898 T A H L E 12.—Validity and r e l i a b i l i t y c o e ffic ie n ts and in ter c o rr e la tio n s y ield ed by various scoring urocedures on the C A T V a lid ity : z-v alu es between sco rin g methods (above d iag o n al) and in te r c o r r e la tio n s (below d ia g o n a l) Score*: xc5 b-h X3.3 xt x c X Xf C b Xc5 N S N S 3.36c 2.91° 3.57° 3.3 5 c U.27c .8962 X 3-U N S 1 . 2 1 * 1.97d 2,09d 2.38d 3.1 3 c .8839 X3_3 N S 2 . 06^ 2.23d 2.03d 3.U0® .8821 Xt .9 h l .832 • N S 1.12 .75 2. 1 il*d . 868U X U .870 .877 .908 N S N S 2.27d .8617 Xc .910 .856 .889 .919 N S 2. 88c .8578 X .8-96 .833 .815 .708 N S .8528 Xf .883 .825 . 8 6 1 * .096 .895 .958 .818 V a lid ity c o e f fic ie n ts co rre .96U 1.077s cted fo r a tte n u a tio n : l.Q 21e 1 . 010® .969 .95U .920 .921 ?.eli a b ility * ": ‘.909 .9h6b .708h .8 2 li • 9li3E .lohh .f iie i • 932C .777h .780i .93UG .823h .**39 .903 .861 .952 a See Table 11 fo r meaning of score a b b re v ia tio n s. ■'Obtained v a lid ity c o e f f ic ie n ts . S i g n i f i c a n t a t th e .C l lev el* ^ S ig n ific a n t a t th e .05 le v e l. S p u r io u s ly high sin ce lower-bound r e l i a b i l i t y used. fFrcm Flanagan form ula (3 6s37). S o t in te r p r e ta b le , s e rio u s ly in f la te d . ty ju llik s e n 's low er-bound r e l i a b i l i t y estim a te (39). w ^■Luuroved low er-bound e stim a te (c f. c. 56). ► 05 i n c o r r e o t r e s p o n s e s i n t h e " d o n ' t know" o p t i o n i g n o r e d was o n l y .0 4 3 4 , a d i f f e r e n c e , h o w e v e r, s i g n i f i c a n t b e y o n d t h e .001 l e v e l s i n c e t h e c o e f f i c i e n t s w ere h i g h l y c o r r e l a t e d . I f t h e s c o r e s had b e e n i n d e p e n d e n t , an e s t i m a t e d c o e f f i c i e n t d i f f e r e n c e of ' roin .087 t o .136 w ould h a v e b e e n ob t a i n e d i f t h e same l e v e l o f s i g n i f i c a n c e was r e a c h e d . T h is a p p r o x i m a t i o n was d e t e r m i n e d by t r a n s f o r m i n g t h e o b s e r v e d c o e f f i c i e n t s to F i s h e r ' s z c o e f f i c i e n t s ; co m p u t i n g t h e s t a n d a r d e r r o r o f t h e d i f f e r e n c e ; m u l t i p l y i n g t h i s v a l u e by 3 . 6 7 , t h e o b t a i n e d c r i t i c a l r a t i o on t h e c o r r e l a t e d m e a s u r e s ; a d d i n g ( o r s u b t r a c t i n g ) t h i s v a l u e to th e z c o e f f i c i e n t from t h e o b s e r v e d c o r r e l a t i o n s , and c o n v e r t in g t h e s e v a l u e s b ack to p ro d u c t-m o m e n t c o e f f i c i e n t s . I t i s a p p a r e n t t h a t t h e d i f f e r e n c e s b e tw e e n t h e o b s e r v e d c o e f f i c i e n t s would have been o f t h e o r d e r o f two t o t h r e e t i m e s g r e a t e r th a n t h e o b t a i n e d d i f f e r e n c e s i f t h e r e had b e e n i n d e p e n d e n c e among t h e s c o r i n g m e th o d s . The f o r c e d c o m p l e t i o n method y i e l d e d t h e l o w e s t v a l i d i t y c o e f f i c i e n t , w i t h th e e x c e p t i o n o f t h e c o n v e n t i o n a l num ber r i g h t method .viiere s i g n i f i c a n c e was n o t r e a c h e d . T h re e f a c t o r s w ould a p p e a r t o have a c c o u n t e d f o r t h i s l o w e r i n g : ( l ) t h e r e s u l t a n t i n c r e a s e i n c h a n c e v a r i a n c e , (2 t h e low, y e t s i g n i f i c a n t , c o r r e l a t i o n ( r = .3 5 3 ) b e tw e e n t h e number o f ite m s a t t e m p t e d and t h e number o f c o r r e c t r e s p o n s e s , c o r r e c t e d f o r c h a n c e , and ( 3 ) t h e p o s i t i v e r e l a t i o n s h i p b etw een a b i l i t v and t e n d e n c y 66 to f o l l o w d i r e c t i o n s . G u i l f o r d ’s ( 3 7 :4 5 5 ) c o n t e n t i o n t h a t t h e r e w ere b e t t e r ways o f d e a l i n g w i t h t h e p ro b le m s o f r e s p o n s e s e t s t h a n t h e f o r c e d - c h o i c e m ethod was s u p p o r t e d by t h e s e r e s u l t s on t h e CAT. A l t h o u g h g r e a t e r r e l i a b i l i t y and v a l i d i t y h av e b e e n found u s i n g t h i s method i n p e r s o n a l i t y m e a su re m e n t ( 3 0 , 3 1 ) , t h e p r e s e n t e v a l u a t i o n found o t h e r m ethods c l e a r l y s u p e r i o r . The mean g r a d e p la c e m e n t v a l u e was .52 h i g h e r when t h e f o r c e d c o m p l e t i o n method was u s e d t h a n when t h e s t a n d a r d i z e d method was u s e d ; a d i f f e r e n c e s i g n i f i c a n t a t t h e .05 l e v e l o f c o n f i d e n c e , u s i n g t h e o n e - t a i l t e s t (z = 1 . 9 1 ) . The v a l i d i t y c o e f f i c i e n t s y i e l d e d by th e 3— 4 and 3 -3 c u t o f f m ethods w ere s i g n i f i c a n t l y h i g h e r Mian t h a t of t h e c o n v e n t i o n a l num ber r i g h t m eth o d ; c o n s e q u e n t l y , h y p o t h e s i s V II was s u p p o r t e d by t h e s e m e th o d s . I n t h e p r e s e n t s t u d y , an a c c u r a t e a p p r a i s a l o f t h e r e l i a b i l i t i e s was p o s s i b l e on a l l s c o r e s e x c e p t t h e c u t o f f m e th o d s . T a b le 12 shows t h a t o n ly t h e number r i g h t and c o r r e c t e d s c o r e , i g n o r i n g t h e " d o n ' t know" r e s p o n s e m e th o d s, y i e l d e d r e l i a b i l i t y c o e f f i c i e n t s a b o v e t h e .90 minimum recommended by s e v e r a l a u t h o r i t i e s f o r t h i s ty p e o f t e s t . The r e l i a b i l i t i e s o r t h e c u t o f f s c o r e s c o u l d n o t a c c u r a t e l y be a s s e s s e d in t h e p r e s e n t r e s e a r c h d e s i g n , as t h i s would have r e q u i r e d an a d d i t i o n a l p e r i o d o f t e s t i n g , 57 and w o u ld a l s o h a v e I n t r o d u c e d a n o t h e r v a r i a b l e . A l o w e r - bound c o e f f i c i e n t f o r t h e s e s c o r e s was com puted u s i n g t h e f o r m u l a p r o p o s e d by G u l l i k s e n ( 3 9 ) . I n t h r e e o f t h e f o u r c a s e s t h i s v a l u e was below t h e minimum v a l u e r e q u i r e d i n o r d e r f o r th e o b t a i n e d v a l i d i t y c o e f f i c i e n t t o be p o s s i b l e , as shown by c o e f f i c i e n t s g r e a t e r th a n u n i t y when c o r r e c t e d f o r a t t e n u a t i o n . A more a c c u r a t e lo w e r -b o u n d c o e f f i c i e n t f o r t h e s e p a r t i c u l a r s c o r e s was d e t e r m i n e d by u s i n g t h e f o l l o w i n g f o r m u l a , d e r i v e d from th e s t a n d a r d c o r r e c t i o n f o r a t t e n u a t i o n f o r m u l a . 2 x x ( mininnim) ~ r xy r yy The f a c t t h a t th o v a l i d i t y c o e f f i c i e n t c a n n e v e r ex c e e d t h e s q u a r e r o o t o f th e p r o d u c t s o f t h e r e l i a b i l i t y c o e f f i c i e n t s i s i l l u s t r a t e d by t h i s f o r m u l a . The mean number o f i ( ems a t t e m p t e d on th e CAT d u r in g t h e s t a n d a r d tim e l i m i t was 1 1 3 .2 o f t h e p o s s i b l e 125, w i t h a s t a n d a r d d e v L a t i o n o f 1 7 . G. T h i r t v - f o u r (28.1% ) e x a m in e e s a t t e m p t e d a l l i t e m s , th e r a n g e b e i n g from f o r t y - one to 125 i t e m s n tt.e m p le d . The mean amount o f a d d i t i o n a l tim e r e q u i r e d f o r a l l e x a m in e e s t o a t t e m p t a l l i t e m s was 5 .5 7 m i n u t e s w i t h a s t a n d a r d d e v i a t i o n o f 9 . 5 2 m i n u t e s . In th e e x t r e m e c a e f one e x a m in e e to o k an a d d i t i o n a l s e v e n t y m i n u t e s t o c o m p l e t e t h e t e s t . The SRA t e s t s . — S i n c e t h e i t e m s on t h e SRA 58 A r i t h m e t i c R e a s o n i n g T e s t a r e n o t a r r a n g e d on t h e same t y p e o f d i f f i c u l t y g r a d i e n t a s a r e t h e i t e m s on t h e SRA A r i t h m e t i c C o m p u t a t i o n T e s t , t h e s e two t e s t s w ere ex am in ed s e p a r a t e l y . The r e s u l t s on th e c o m p u t a t i o n e x a m i n a t i o n a r e g i v e n i n T a b le s 13 and 1 4 . The f o r c e d c o m p l e t i o n method was shown t o have s i g n i f i c a n t l y g r e a t e r v a l i d i t y t h a n o n l y t h e c h a n c e t h r e s h o l d and c o r r e c t e d s c o r e s . No s i g n i f i c a n t d i f f e r e n c e s in r e l i a b i l i t y w ere n o t e d among t h o s e s c o r e s on w hich a o c u r a t e r e l i a b i l i t y d a t a w ere a v a i l a b l e . The mean num ber o f i t e m s a t t e m p t e d was 4 1 .9 9 o f t h e p o s s i b l e f o r t y — f o u r . The c o r r e s p o n d i n g s t a n d a r d d e v i a t i o n was 5 . 0 1 ; t h e r a n g e b e i n g from t w e n t y - t w o to f o r t y - f o u r i t e m s a t t e m p t e d . N i i m t y - s e v o n ( 7 7 . O ') o f t h e exam i n e e s a t t e m p t e d a l l ite m s d u r i n g t h e s t a n d a r d tim e l i m i t ; t h e mean e x t r a tim e r e q u i r e d t o c o m p l e t e tho t e s t s f o r t h e r e m a i n i n g was 1 1 .9 m i n u t e s , w i t a s t a n d a r d d e v i a t i o n o f 9 .5 9 m i n u t e s . The u p p e r tim e s c o r e e x t e n d e d t o e i g h t y - e i g h t a d d i t i o n a l m i n u t e s . The mean num ber o f i t e m s a t t e m p t e d b e f o r e t h e p u p i l s r e a c h e d t h e i r mean a b i l i t y Ilm en wat d e t e r m i n e d by a d d i n g t h e mean r i g h t and wrong s c o r e s f o r e a c h s c o r i n g m e th o d . T h e se v a l u e s w ere 1 5 . 5 5 , 1 7 .5 5 , 2 1 , 9 6 , and 2 2 .3 7 m i n u t e s f o r t h e 3 - 4 , 3— 3, 4 -4 c u t o f f ; am' c h a n c e t h r e s h o l d s c o r i n g m ethods r e s p e c t i v e l y . T h is showed t h a t f o r 5 0 '’ o f t h e g r o u p , a l m o s t n i l r e l e v a n t i n f o r m a t i o n was 59 TABLE 13 . — C o r r e l a t i o n m a t r i x o f s c o r e s y i e l d e d by v a r i - o u r s c o r i n g p r o c e d u r e s on t h e SRA A c h ie v e m e n t S e r i e s , a r i t h m e t i c c o m p u t a t i o n , w i t h means and s t a n d a r d d e v i a t i o n s S c o r e 0 X X* c X X3— 3 X3 - 4 X4— 4 Xt C r c . rw X 956 984 905 860 934 948 7685 772 X 968 c 876 840 908 923 7309 x f 909 869 944 954 7724 X3 -3 939 941 830 7262 739 X3 -4 897 772 7179 719 X4— 4 869 7589 779 Xt 7228 731 J 9989 V a l i d i t y c o e f f i c i e n t s o f wrong s c o r e s : - 6 5 3 -7 7 2 355 429 291 086 M ean s: 1 8 .2 2 1 0 .4 7 1 8 .6 2 1 1 .3 8 1 0 .0 5 12.82 1 3 .1 9 1 5.21 1 5 .1 6 Means o f c o r r e s n o n d i n g wrong s c o r e s : 2 3 .8 1 2 5 . 3 8 6 .7 7 5 . 0 5 9 .1 4 9 . 1 8 2 8 .7 9 S t a n d a r d D e v i a t i o n s : 6 . 9 0 1 1 .7 4 6 .9 3 9 .4 3 8 .93 9 .2 6 9 .27 7 .2 2 7 .2 3 S t a n d a r d D e v i a t i o n s o f wron g s c o r e s : 8 . 1 5 6 .9 3 3 .83 2 .1 6 3 .9 9 1 6 .2 3 7 n 4 * £* im t Decim al p o i n t s o m i t t e d in c o r r e l a t i o n c o e f f i c i e n t s . b See T a b le 11, page 51, f o r m ean in g o f s c o r e a b b r e v i a t i o n s . C r i t e r i o n s c o r e a t end o f s t a n d a r d tim e l i m i t . 60 TABLE 1 4 . — V a l i d i t y and r e l i a b i l i t y c o e f f i c i e n t s y i e l d e d by v a r i o u s s c o r i n g p r o c e d u r e s on t h e SRA A r i t h m e t i c Com p u t a t i o n T e s t V a l i d i t y : z - v a l u e s (a b o v e d i a g o n a l an d c o r r e l a t i o n s b e t w e e n m e a s u r e s ( b e l o w d i a g o n a l ) . Score® Xf X X4 - 4 X c X CO 1 wO Xt X cb 3 -4 Xf .5 8 .71 2 . 4 6 ° 1 .5 6 2 . 6 7 d 1 .8 2 .7 7 2 4 .904 X .983 NS 2 . 0 9 ° 1 . 6 4 2 . 3 0 ° 1 .6 4 .7685 .895 X4 - 4 - 870 NS 1 .62 1 .21 1 .5 2 .7 5 8 9 .972® X .930 c .903 NS .34 .39 .7309 .8 6 7 X3 - 3 ' 772 .799 .8 79 NS .32 .7272 .946® Xt .903 .887 .7 3 4 .843 NS .7228 .930® X3 -4 - 713 .716 .7 9 0 .692 .8 7 6 .7179 .985® R e l i a b i l i t y f • * .8 3 0 .837 . 949® .80 8 . 9 G l g . 9 4 7 g . 9 6 6 g .8 8 0 . 6 9 2 h . 6 7 0 h . 0 8 9 h . 6 0 4 h a See T a b l e 11, p age 5 1, f o r m ean in g o f s c o r e a b b r e v i a t i o n s . v . O b t a i n e d v a l i d i t y c o e f f i c i e n t s . c S i g n i f l e a n t a t t h e .0 5 l e v e l . ^ S i g n i f i c a n t a t t h e .0 1 l e v e l . e S p u r i o u s l y h i g h s i n c e l o w e r - b o u n d r e l i a b i l i t y u s e d . *From F l a n a g a n f o r m u l a ( 3 7 : 3 7 9 ) . g Not i n t e r p r e t a b l e , s e r i o u s l y i n f l a t e d . b G u l l i k s e n ’ s l o w e r - b o u n d r e l i a b i l i t y e s t i m a t e ( 3 9 ) . 61 c o m p l e t e d by t h e t i m e t h e s t u d e n t had a t t e m p t e d o n e - h a l f o f t h e i t e m s . The r e s u l t s from t h e SRA A r i t h m e t i c R e a s o n i n g T e s t a r e g i v e n I n T a b l e 1 5 . S i n c e t h e i t e m s on t h i s t e s t a r e g r o u p e d a r o u n d s p e c i f i c p r o b l e m s , e a c h p r o b le m c o n t a i n i n g s i x t o e i g h t q u e s t i o n s , i t d o e s n o t h a v e a g r a d i e n t o f d i f f i c u l t y w i t h r e g a r d t o c o n s e c u t i v e i t e m s . As a r e s u l t t h e t e s t d e s i g n d i d n o t le n d i t s e l f to t h e u s e o f t h e c u t o f f and c h a n c e - t h r e s h o l d s c o r e s . T h e r e w e re no s i g n i f i c a n t d i f f e r e n c e s i n e i t h e r v a l i d i t y o r r e l i a b i l i t y among t h e t h r e e s c o r i n g m e th o d s u s e d . A l t h o u g h t h e SRA A r i t h m e t i c C o n c e p t s T e s t was n o t d i r e c t l y r e l a t e d t o t h e p u r p o s e s o f t h i s s t u d y , r e l i a b i l i t y and v a l i d i t y d a t a w ere c o m p u te d . T h i s t e s t c o n s i s t s o f t h e f i r s t t w e n t y i t e m s o f t h e s e c o n d d i v i s i o n o f t h e SRA a r i t h m e t i c t e s t s . A ll s t u d e n t s , t h e r e f o r e , a t t e m p t e d a l l ite m s e x c e p t f o r a s m a ll num ber o f o m i s s i o n s . The t e s t was shown t o h a v e a c o e f f i c i e n t o f c o i n c i d e n t v a l i d i t y o f .7 0 7 and a r e l i a b i l i t y , a c c o r d i n g t o t h e F l a n a g a n f o r m u l a , o f .7 7 9 . C o m p a r is o n o f th e SRA and CAT a r i t h m e t i c t e s t s . — I n o r d e r f o r a m e a n i n g f u l e v a l u a t i o n of t h e v a r i o u s s c o r i n g m e th o d s t o be made, i t was n e c e s s a r y to e x a m in e s e v e r a l a s p e c t s o f t h e t e s t s u s e d i n a d d i t i o n t o t h o s e 62 TABLE 15 . — V a l i d i t y , r e l i a b i l i t y , and i n t e r c o r r e l a t i o n s y i e l d e d by v a r i o u s s c o r i n g p r o c e d u r e s on t h e SRA A r i t h m e t i c R e a s o n i n g T e s t V a l i d i t y S c o r e a and I n t e r c o r r e l a t i o n s : X Xc Xf C *0000 r b XX X .981 .9 7 3 .6957 .7 8 0 .9 0 5 Xc .9 8 7 .7025 .768 .910 Xf z = 1 .3 9 .7167 .765 .918 J .9949 .928 C .920 M e a n s: J C 2 4 .0 2 1 6 .4 2 2 4 .6 6 23 .60 2 3 .4 3 S t a n d a r d D e v i a t i o n s : 8 .2 8 11 .02 8 . 4 0 9 .01 9 .1 5 X = c o n v e n t i o n a l r i g h t s c o r e , s t a n d a r d tim e l i m i t . X = c o r r e c t e d s c o r e u s i n g t h e o r e t i c a l c h a n c e c o r r e c t i o n , c Xj = f o r c e d c o m p l e t i o n s c o r e . C - c r i t e r i o n . b From F l a n a g a n ' s f o r m u l a ( 3 7 ) . Q C r i t e r i o n s c o r e a t end o f s t a n d a r d tim e l i m i t . 63 d i s c u s s e d i n C h ap ter I I I . T h ese c o m p a riso n s w i l l be made in term s w hich are a p p r o p r i a t e f o r , and r e l e v a n t t o , th e a p p r a i s a l o f t h e d i f f e r e n t s c o r i n g t e c h n i q u e s , n o t in term s o f th e r e l a t i v e m e r it s o f t h e t e s t s per s e . The s e c t i o n o f th e SRA on w hich i t was p a s s i b l e to u se a l l th e v a r i o u s s c o r i n g m ethods o f t h i s s tu d y ( t h e c o m p u ta tio n s e c t i o n ) had a mean ite m d i f f i c u l t y o f .3 7 2 ; th e v a l u e on th e CAT was . 4 6 6 . T his d i f f e r e n c e was s i g n i f i c a n t a t th e .01 l e v e l (z = 2 . 6 4 ) . The r e l i a b i l i t y o f t h e s c o r e s on t h e CAT w e r e a l l s i g n i f i c a n t l y h i g h e r t h a n t h e c o r r e s p o r n ! i n g s c o r e s on t h e SRA. T h i s was e x p e c t e d a s i n f o r m a t i o n i n C h a p t e r I I I p o i n t e d o u t t h a t t h e r e a s o n i n g s e c t i o n s o f b o t h t e s t s had h i g h e r r e l i a b i l i t i e s t h a n t h e c o m p u t a t i o n s s e c t i o n s . When t h e r e l i a b i l i t y o f t h e t o t a l CAT was c o m p a re d a g a i n s t t h e r e a s o n i n g s e c t i o n o f t h e SRA s e r i e s , no s i g n i f i c a n t d i f f e r e n c e s i n r e l i a b i l i t y e x i s t e d . The CAT had s i g n i f i c a n t l y h i g h e r c o i n c i d e n t v a l i d i t y t h a n d i d t h e SRA t e s t s . T h i s w ould a p p e a r t o h a v e r e s u l t e d fro m t h e CAT’ s u s e o f t h e a l t e r n a t i v e " d o n ' t know" o p t i o n an d , to some e x t e n t , t h e f a c t t h a t th e SRA t e s t s were v e r y d i f f i c u l t f o r a g r e a t e r p r o p o r t i o n o f t h e exam i n e e s , t h e r e b y a l l o w i n g c h a n c e and r e s p o n s e s e t v a r i a n c e to h a v e g r e a t e r o p p o r t u n i t y t o i n f l u e n c e e x a m in e e p e r f o r m a n c e . T h e re was a l s o some s u b j e c t i v i t y i n t h e s c o r i n g o f t h e SRA r e a s o n i n g t e s t when i n r e c a l l form at due to t h e n a tu r e o f 64 o f some o f th e t e s t i t e m s . The norms on th e SRA t e s t e were found to demand a h ig h e r l e v e l o f perform ance to a c h ie v e th e same grade p lacem en t s t a t u e . The mean grad e p lacem en t v a l u e s f o r th e CAT were 5 . 7 7 and 5 . 6 3 f o r th e r e a s o n in g and fun d am en tals s e c t i o n s r e s p e c t i v e l y ; th e v a l u e s f o r th e SRA w e re 4 . 8 8 , 5 . 1 3 , and 5 . 1 6 f o r t h e r e a s o n i n g , c o n c e p t s , and c o m p u ta - t i o n s e c t i o n s r e s p e c t i v e l y . S i n c e t h e r e was no s i g n i f i c a n t d i f f e r e n c e b e tw e e n t h e mean a r i t h m e t i c a c h i e v e m e n t (z = .0 9 ) and th e mean IQ*s (z x .9 ) o f t h e two g r o u p s a t th e b e g i n n i n g o f t h e yr>a r , i t m ust be c o n c l u d e d t h a t a h i g h e r l e v e l of a b i l i t y was r e q u i r e d on t h e SRA t o a t t a i n t h e same g r a d e p l a c e m e n t a s on t h e CAT. T h is w ould a p p e a r t o h av e r e s u l t e d from t h e norm d e f i n i t i o n s r a t h e r t h a n a d i f f e r e n c e i n th e s t a n d a r d i z e d p o p u l a t i o n . The CAT d e f i n e s a g r a d e p l a c e m e n t v a l u e a s t h e mean s c o r e o f p e r s o n s w i t h i n t h r e e m onths o f t h e t y p i c a l a g e - g r a d e r e l a t i o n f o r a g r a d e , and who have IQ s c o r e s from 98 to 102. S i n c e t h e a c h i e v e m e n t d i s t r i b u t i o n , as g i v e n in t h e m anual (7 8 ) i s skewed to w a rd th e h i g h e r s c o r e s , t h e s c o r e c o r r e s p o n d i n g to g r a d e l e v e l s t a t u s i s a c t u a l l y b elow t h e mean s c o r e o f t h e g r a d e l e v e l p o p u l a t i o n . A c o m p a r i s o n o f th e r e c o g n i t i o n v e r s u s r e c a l l s c o r e s on t h e v a r i o u s t e s t s a r e g i v e n in T a b le 16. The SRA t e s t s a r c p r e s e n t e d s e p a r a t e l y s i n c e a d e g r e e o f s u b j e c t i v i t y was i n t r o d u c e d i n t o th e s c o r i n g o f th e r e c a l l 68 TABLE 1 6 .- - - R e c a l l an d r e c o g n i t i o n mean s c o r e s on a l t e r n a t e fo rm s o f t h e CAT and SRA A r i t h m e t i c T e s t s CAT T o t a l SRA Comp. SRA R e a s . R e c o g n i 1 1 o n : F o rc e d c o m p l e t i o n (X^) 6 0 .5 8 1 8 .6 2 2 4 .6 6 F o r c e d c o m p l e t i o n w i t h c h a n c e c o r r e c t i o n (X^c ) 44 .48 1 .16 1 6 .5 5 R e c a l 1 (C ): 62 .a 1 15.21 23 .60 z - r n t i o s : Xf - C -2 .7 3 “ G .6 8 a .101 Xf c ' C -1 1 .2 8 a - 6 . 1 8a - 5 . 78 a S i g n i f i c a n t a t th e .01 l e v e l . 66 fo rm o f t h e r e a s o n i n g t e s t . A s t u d e n t was g iv e n c r e d i t i n b o r d e r l i n e d e c i s i o n s . The r e c o g n i t i o n form s o f t h e CAT w e re shown t o h a v e a s i g n i f i c a n t l y l o w e r mean s c o r e s t h a n t h e r e c a l l f o r m s . A l l o f t h e m u l t i p l e c h o i c e fo rm s h ad s i g n i f i c a n t l y l o w e r means t h a n t h e r e c a l l means when t h e t h e o r e t i c a l c o r r e c t i o n s f o r c h a n c e w e re m a d e . I t w o u ld a p p e a r t h a t e i t h e r t h e i t e m w r i t e r s h a v e b e e n s u c c e s s f u l i n c o n s t r u c t i n g i t e m s w h ich h a v e h i g h l y a t t r a c t i v e d e c o y s o r t h a t a g r e a t am ount o f i n a c c u r a c y i s i n t r o d u c e d i n t h e u s e o f t h e s e p a r a t e IBM a n s w e r s h e e t . The r e s e a r c h on t h i s i s s u e would seem t o d i s c o u n t t h e l a t t e r p o s s i b i l i t y . T r a x l e r ( 4 8 : 3 8 4 ) r e p o r t e d two u n p u b l i s h e d M a s t e r ' s t h e s e s w h ic h i n v e s t i g a t e d t h e i n f l u e n c e o f t h e s e p a r a t e a n s w e r s h e e t . I n one o f t h e s e s t u d i e s , Forms A and B o f t h e Iowa E l e m e n t a r y L a n g u a g e T e s t s w e r e a d m i n i s t e r e d t o 266 p u p i l s i n g r a d e s f o u r to s i x . No s i g n i f i c a n t d i f f e r e n c e s in mean s c o r e s w e r e fo u n d b e tw e e n t e s t s w i t h and w i t h o u t a n s w e r s h e e t s when t h e t e s t s w ere a d m i n i s t e r e d w i t h o u t n tim e l i m i t . In t h e o t h e r s t u d y , t h e i n v e s t i g a t o r fo u n d more tim e r e q u i r e d f o r t h e u s e o f t h e s e p a r a t e a n s w e r s h e e t t h a n f o r t h e t e s t b o o k l e t i n g r a d e s t h r e e and f i v e , b u t no s i g n i f i c a n t i n f l u e n c e on th e mean s c o r e s when t i m e was n o t a c r u c i a l f a c t o r . D u n la p (2 3 ) a l s o f o u n d t h a t s e p a r a t e a n s w e r s h e e t s w e re s u i t a b l e f o r u s e i n g r a d e s a s low a s f o u r . T h e se f i n d i n g s , a l o n g w i t h t h e f a c t t h a t a l l o f t h e c l a s s e s o f t h i s s t u d y had t a k e n f o u r t e s t s u s i n g t h e 67 s e p a r a t e IBM a n s w e r s h e e t a t t h e b e g i n n i n g o f t h e s c h o o l y e a r t w o u ld I n d i c a t e t h a t o n l y m i n o r e f f e c t s r e s u l t e d fro m t h e u s e o f s e p a r a t e a n s w e r s h e e t s . S t u d i e s r e p o r t e d i n C h a p t e r I I ( 4 6 , 8 2 ; showed t h a t a d u l t e x a m i n e e s g e n e r a l l y r e s p o n d e d w i t h g r e a t e r t h a n c h a n c e p r o b a b i l i t y on i t e m s on w h ic h t h e y " g u e s s e d . " The f i n d i n g s o f t h e p r e s e n t s t u d y showed t h e o p p o s i t e r e s u l t s w i t h c h i l d r e n , a s d i d a s t u d y by F a i r b e r ( 2 5 ) whose s u b j e c t s w ere fro m g r a d e s s i x a n d e i g h t . I t i s p o s s i b l e t h a t " t e s t s o p h i s t i c a t i o n " a c c o u n t s , t o some e x t e n t , f o r t h e l a c k o f a g r e e m e n t a t t h e d i f f e r e n t a g e l e v e l s . One a l s o d o e s n o t know i f t h e i t e m d i s t r n c t o r s u s e d i n t h e n o n - s t a n d a r d i z e d t e s t s in t h e a b o v e a d u l t s t u d i e s p o s s e s s e d t h e same d e g r e e o f r e l a t i v e a t t r a c t i v e n e s s a s d i d t h e t e s t s o f t h i s s t u d y . The f a c t t h a t t h e y w e re n o t a r i t h m e t i c t e s t s may h a v e a f f e c t e d th e c o n t r o l o f t h e " f a c e " a p p e a l a d v e r s e l y . The r e s u l t s on th e CAT c o u l d h a v e been h y p o t h e s i z e d a s r e s u l t i n g fro m an " o v e r c a u t i o u s n e s s " s e t on t h e p a r t o f some e x a m i n e e s , c a u s i n g them to r e s p o n d to t h e " d o n ’ t know" o p t i o n u n l e s s c e r t a i n o f t h e c o r r e c t n e s s o f t h e r e s p o n s e . I t i s i m p r o b a b l e , h o w e v e r, in l i g h t o f t h e f i n d i n g s on t h e o t h e r t e s t s , t h a t t h i s v a r i a b l e was r e s p o n s i b l e f o r t h e m a j o r p a r t o f t h e d i s c r e p a n c i e s b e t w e e n t h e t e s t s c o r e s . 66 C h a n c e - l e v e l s c o r e s . — In o r d e r t o more t h o r o u g h l y e v a l u a t e t h e e f f e c t s o f c h a n c e , a n e x a m i n a t i o n vas made o f s c o r e s f a l l i n g i n t h e r e g i o n o f t h e o r e t i c a l c h a n c e s u c c e s s . The s t a n d a r d e r r o r o f t h e c h a n c e s c o r e i s e q u a l t o ^ n p q , w here n i s t h e num ber o f i t e m s , p i s c h a n c e p r o b a b i l i t y o f s u c c e s s , and q i s t h e p r o b a b i l i t y o f m aking a wrong r e s p o n s e ( 3 6 : 4 8 c ) . The number o f e x a m in e e s who w e re fo u n d t o h av e o b t a i n e d s c o r e s below w hich th e n u l l h y p o t h e s i s o f no k n o w le d g e o f c o n t e n t c o u l d n o t be r e j e c t e d a t t h e .05 l e v e l , on t h e b a s i s o f t h e o r e t i c a l c h a n c e , was fo u n d t o be s i x f o r t h e CAT, t h i r t y - f i v e f o r t h e SRA r e a s o n i n g , and f i f t y - t w o f o r t h e SKA c o m p u t a t i o n t e s t s . V u l i d i t y c o e f f i c i e n t s o f t h e s e s c o r e s a r e g i v e n i n T a b le 17. Only one method, th e 4 -4 c u t o f f method, y i e l d e d s i g n i f i c a n t p o s i t i v e c o r r e l a t i o n s on b o th t e s t s b e f o r e c o r r e c t i o n s fo r ran ge r e s t r i c t i o n were made. It would appear t h a t the t e s t a were too d i f f i c u l t to g iv e a c c u r a t e d i s c r i m i n a t o r y in f o r m a t io n about t h e s e e x a m in e e s, p a r t i c u l a r l y i f s c o r e d in n c o n v e n t i o n a l manner. G u ilfo r d ( 3 7 : 4 4 8 ) lias recommended t h a t t e s t s sh o u ld be kept so e a s y that s c o r e s below two sta n d a rd d e v i a t i o n s above mean chance e x p e c t a n c y are a v o id e d . Such s c o r e s were o b t a in e d by s i x exam inees (5 r£) on the CAT, f i f t y - t w o (41.6?£) on the SRA r e a s o n in g t e s t , and s e v e n t y (56$) on th e SKA c :>mpu t a t i on t e s t s . The r e s u l t i n g v a l i d i t y c o e f f i c i e n t s from the v a r i o u s s c o r i n g methods are 69 TABLE 1 7 . — V a l i d i t y o f c h a n c e - l e v e l s c o r e s on t h e SRA a t t h e .0 5 l e v e l , w i t h and w i t h o u t c o r r e c t i o n f o r r a n g e r e s t r i c t i o n ' * S c o r i n g Method SRA R e a s . SRA O o a o • r x y Bxy r xy R xy Number c o r r e c t - . 4 5 4 c .041 .09 W ith t h e o r e t i c a l c h a n c e c o r r e c t i o n .1 0 4 .29 .1 2 8 .17 F o r c e d c o m p l e t i o n - . 3 6 5 c - . 0 5 4 c 3 -3 C u t o f f - . 2 6 4 c .088 .24 3 - 4 C u t o f f - . 3 1 8 c . 121 . 3 3 d 4 - 4 C u t o f f . 7 S 7 e . 7 8 e .273® .53® C h a n c e - t h r e s ho Id .4 4 0 * .6 4 ° .208 . 3 4 d S t a n d a r d e r r o r ( r ) o .171 .140 N 35 52 a C o m p u t a t i o n s not g i v e n f o r CAT an N = 6 . ^ C a l c u l a t e d from a b a c s ( 5 7 ) . c C o e f f i c i e n t w ould n o t h av e p s y c h o l o g i c a l m eaning and was t h e r e f o r e n o t c o m p u t e d . S i g n i f i c a n t p o s i t i v e c o r r e l a t i o n a t .05 l e v e l . S i g n i f i c a n t p o s i t i v e c o r r e l a t i o n a t .01 l e v e l . 70 g i v e n in T ab le 1 8 . A l l s c o r i n g methods on th e c o m p u ta t io n t e s t y i e l d e d v a l i d i t i e s s u f f i c i e n t t o r e j e c t t h e n u l l h y p o t h e s i s a t th e .0 1 l e v e l . T h ese v a l u e s w ere, how ever, s i g n i f i c a n t l y lo w e r , even w it h t h e c o r r e c t i o n f o r r e s t r i c t i o n o f r a n g e , than t h o s e o b t a i n e d on t h e e n t i r e e r o u p . On t h e r e a s o n i n g t e s t o n l y two m ethods, th e 4 - 4 c u t o f f and t h e c h a n c e - t h r e s h o l d , y i e l d e d s i g n i f i c a n t p o s i t i v e v a l i d i t y c o e f f i c i e n t s . I t i s a p p a r e n t t h a t fo r a c o n s i d e r a b l e p e r c e n t a g e o f th e group l i t t l e p r e d i c t i v e i n f o r m a t i o n was o b t a in e d on th e SRA t e s t s . T h is p r o b a b ly a c c o u n t e d , in p a r t , f o r t h e lo w e r v a l i d i t y c o e f f i c i e n t s o b t a in e d on t h e s e t e s t s as compared w ith t h e CAT. The b a t t e r i e s used in t h i s s tu d y were d e s i g n e d f o r g r a d e s f o u r to s i x . I t would be p r e d i c t e d t h a t th e unde s i r a b l e a s p e c t s r e s u l t i n g from th e d i f f i c u l t y l e v e l o f th e ite m s would have been m a g n if i e d had t h e s u b j e c t s been f o u r t h r a t h e r than f i f t h grade s t u d e n t s . C o n v e r s e l y , t h e s e e f f e c t s would have b een l e s s s e r i o u s i f the sam ple had been composed o f s i x t h g r a d e p u p i l s . I f t e s t pub l i s h e r s are g o in g to c o n t i n u e to produce t e s t s a p p r o p r i a t e f o r more than one g rad e l e v e l , th u s c a u s i n g th e ite m s to be v e r y d i f f i c u l t f o r th e low er grade l e v e l , i t would be d e s i r a b l e f o r an i n v e s t i g a t i o n of d i f f e r e n t i a l s c o r i n g a t v a r i o u s a b i l i t y l e v e l s b e c o n d u c t e d b e f o r e m a r k e tin g t h e i r p r o d u c t s . 71 TABLE 1 8 . — V a l i d i t y o f c h a n c e - l e v e l s c o r e s * a t th e .01 ^ l e v e l , w i t h and w i t h o u t c o r r e c t i o n f o r r a n g e r e s t r i c t i o n S c o r i n g Method SRA R e a s . SRA Comp. r xy r xy K Number c o r r e c t - . 3 0 3 o . 4 14d . 64d With t h e o r e t i c a l g * ch an ce c o r r e c t i o n .078 .16 .334 .4 7 Forced c o m p le t io n - . 1 5 5 c . 4 47d . 6 3 d 3 -3 C u t o f f - . 1 2 0 c . 3 75d . 6 0 d 3 -4 C u t o f f - . 0 9 5 c . 3 98d . 6 3 d 4 -4 C u t o f f . G78*1 . 8 0 d . 3 6 6 d . 5 6 d Choi ce — t ' r e s h o 1 d . 3 B l d .5 8 ° .3 6 3 d . 50d Standard e r r o r (r ) ' o ’ . 140 . 120 N 52 70 a C o m p u ta tio n s not g i v e n fo r CAT as N - 8 . ^Computed from a b a c s ( 5 7 ) . (J C o e f f i c i e n t would n ot have p sycho 1o g i c a l meaning and was t h e r e f o r e not com puted. ^ S i g n i f i c a n t p o s i t i v e c o r r e l a t i o n a t .01 l e v e l . 72 Summary e v a l u a t i o n o f s c o r i n g M ethods. — A c r i t i q u e o f th e v a r i o u s methods used in s c o r i n g th e CAT and SRA t e s t s w i l l now be made from th e in fo r m a t io n p r e s e n t e d in t h e p r e c e d in g s e c t i o n s . The c o n v e n t io n a l number r i g h t s c o r i n g method y i e l d e d th e s e v e n t h lo w e s t v a l i d i t y c o e f f i c i e n t o f the e i g h t methods used on the CAT; s i g n i f i c a n t l y low er than t h r e e : th e c o r r e c t e d s c o r e , d i s c o u n t i n g t h e " d o n 't know" r e s p o n s e s , th e 3 -4 and the 3 -3 c u t o f f m ethods. On the SRA r e a s o n in '7 t e s t th e number r i g h t method y i e l d e d s i g n i f i c a n t l y hi/rher v a l i d i t y than the c h a n c e - t h r e s h o l d and th e s c o r e s u s i n g the t h e o r e t i c a l ch an ce c o r r e c t i o n . T h is method was not s i f n i f i c a n t l y low er than the c o e f f i c i e n t s o b ta in e d by the o t h e r two methods u s e d . The d i f f e r e n c e s found u s in g the number r i g h t method on the CAT and SRA t e s t s p ro b a b ly r e s u l t e d from the CAT's use o f the a l t e r n a t i v e " d o n 't know" o p t i o n . The u se o f t h i s o p t io n d e c r e a s e d th e e f f e c t s o f chance as i n d i c a t e d by the low er r i s k - a c c e n t a n c e i n d e x e s . The use o f th e t h e o r e t i c a l c o r r e c t i o n fo r chance showed no s i g n i f i c a n t ad van tage o v e r the o t h e r s c o r i n g methods on the r e a s o n in g t e s t o f the SRA. Other methods a l s o y i e l d e d more d e s i r a b l e r e s u l t s o f th e SRA c o m p u ta tio n and CAT ( e x c e p t when th e " d o n 't know" r e s p o n s e s were d i s c o u n t e d ) . Table 16, page 63, showed t h a t th e a p r i o r a c o r r e c t i o n a c t u a l l y o v e r - c o r r e c t e d , c a u s in g th e r e c o g n itio n 73 s c o r e s to be s i g n i f i c a n t l y lo w e r than th e r e c a l l s c o r e s . A p p a r e n tly th e t e s t a u th o r s s u c c e e d e d in making th e d i s - t r a c t o r s a l l u r i n g enough to s e d u c e g r e a t e r th an chance p a s s e r s - b y . The ch a n ce c o r r e c t i o n was e s p e c i a l l y i n a p p r o p r i a t e on t h e CAT which u se d the " d o n 't know" o p t i o n . When th e r e s p o n s e s marked in t h i s o p t i o n w ere d i s r e g a r d e d , how ever, and th e c o r r e c t i o n f o r chance a p p l i e d , th e most f a v o r a b l e r e s u l t s o f any method were o b t a i n e d . The fo r c e d c o m p le t io n method was shown to have s t a t i s t i c a l a d v a n ta g e s o v er o n l y two s c o r i n g m ethods, the c h a n c e - t h r e s h o l d and c o r r e c t e d s c o r e , on the SRA computa t i o n t e s t . No s i g n i f i c a n t a d v a n ta g e s accompanied t h i s method on th e SRA r e a s o n in g t e s t and on the CAT i n f e r i o r r e s u l t s were o b t a i n e d . S e r i o u s p r a c t i c a l a d m i n i s t r a t i v e problems a l s o a r o s e from th e use o f t h i s method. In o rd er f o r a l l ex a m in e es t o c o n s c i e n t i o u s l y respond to e v e r y ite m , the sta n d a r d tim e l i m i t was a p p r o x im a t e ly d o u b le d . I t i s p r o b a b le t h a t g r e a t e r r e l i a b i l i t y and v a l i d i t y c o u ld have been o b t a in e d by h a v in g more item s on t h e t e s t s and a l l o w i n g a p p r o x im a t e ly 95*^ of th e p u p i l s to c o m p le te th e t e s t s . A d i s c i p l i n e problem was e n c o u n te r e d in a t t e m p t i n g to keep th o se who had f i n i s h e d th e t e s t s o c c u p i e d . I f th e exam inees were d i s m i s s e d or r e c r e a t i o n a l r e a d in g a l l o w e d , some o f th e r em a in in g h u r r i e d l y responded to the u n a t - tempted item s ii. o r d e r to be a llo w e d th e s e p r i v i l e g e s . There were alv.ays t h o s e , however, who wauld make a 74 d i l i g e n t , and o f t e n tim e -c o n su m in g , a tte m p t to answer each it e m . S e v e r a l appeared d is c o u r a g e d and d e p r e s s e d by th e on ero u s ta sk o f a t t e m p t in g to work problems t h a t were above t h e i r a b i l i t y lim e n . As shown in Table 6, page 40, most had reached t h e i r c e i l i n g d u r in g th e s ta n d a r d titde l i m i t . The c u t o f f methods were s u p e r i o r to th e f o r c e d - c h o i c e method on th e CAT. The 3 -3 and 3-4 c u t o f f methods a l s o had g r e a t e r v a l i d i t y than ('id th e number r i g h t and c o r r e c t e d s c o r e s . On th e SRA co m p u ta tio n t e s t , no method y i e l d e d s i g n i f i c a n t l y hiirlicr v a l i d i t y than the 4 - 4 , 3 - 3 , and 3 -4 c u t o f f m ethods. 'Vher th e t e s t s were e x tr e m e ly d i f f i c u l t ( . 0 5 a n d /o r .01 c h a n c e - l e v e l s c o r e s ) , t h e s e c u t o f f methods were s u p e r i o r ; o n ly th e 4-4 c u t o f f and c h a n c e - t h r e s h o l d methods c o n s i s t e n t l y y i e l d e d s i g n i f i c a n t p o s i t i v e v a l i d i t y c o e f f i c i e n t s . The i tier e a s e s in v a l i d i t y from u s in g th e e m p ir ic a l s c o r i n g formula based on the m u l t i p l e r e g r e s s i o n model are shown in Table 11, page 51, and Table 13, page 5 7 . The i n c r e a s e s were o f th e o r d e r o f .01 and .02 in most c a s e s . The l a r g e s t gain o c c u r r e d on th e SRA r e a s o n in g t e s t s where th e v a l i d i t y was i n c r e a s e d from .6957 to .7186 by t h e use o f th e e m i r i c a l f o r m u la . The o p tim a l w e i g h t f o r th e wrong r e s p o n s e s was d e te r m in e d by T h u r s t o n e ' s e q u a t io n ( 9 0 : 7 8 ) . The o p tim a l fo rm u la s fo r th e r e s p e c t i v e t e s t s and s c o r e s ranged as f o l l o w s : R ig h ts minus .014 v/ronr^s fo r the 75 number r i g h t s c o r e on the SRA co m p u ta tio n t e s t , t o R ig h t s minus .6 8 2 Wrongs f o r th e number r i g h t s c o r e on t h e SRA r e a s o n in g t e s t s . A lthough the I n c r e a s e s r e s u l t i n g from the u s e o f t h i s e m p i r i c a l s c o r i n g form ula were s m a l l , t h e r e a r e no s t a t i s t i c a l d i s a d v a n t a g e s to su c h a p r o c e d u r e . Optimal form u las co u ld be e a s i l y d e te r m in e d by t e s t p u b l i s h e r s from t h e i r s t a n d a r d i z a t i o n sa m n le . A s l i g h t i n c r e a s e i s n e v e r t h e l e s s an i n c r e a s e . H y p o th eses V i l a through V l l g were p a r t i a l l y sup p orted on th e CAT. They were not su p p o rted on the SRA t e s t s ; a lth o u g h t h e 3 - 3 , 3 - 4 , and 4 -4 c u t o f f methods y i e l d e d v a l i d i t i e s n ot s i g n i f i c a n t l y d i f f e r e n t from the nurnber r i g h t method, th e y were not h ig h e r as h y p o t h e s i z e d . <*ith the exam inees fo r whom th e t e s t s were v e r y d i f f i c u l t (a p p r o x im a t e ly 40^ on th e SRA t e s t s ) , th e 4-4 c u t o f f and c h a n c e - t h r e s h o l d s c o r e s were s u p e r i o r i v a l i d i t y to th e c o n v e n t io n a l s c o r i n g method. H y p o th eses V I I f and V l l g were s u p p o r te d under t h e s e c o n d i t i o n s . On th e CAT, h y p o t h e s e s V i l a , V l l d , and V i l e were s u p p o r te d ; th e c o r r e c t e d s c o r e , d i s c o u n t i n g th e "d on ’ t know" r e s p o n s e s ; the 3 -4 s c o r i n g method; and th e 4 -4 s c o r in g method, a l l y i e l d i n g s i g n i f i c a n t l y h ig h e r v a l i d i t i e s than th e c o n v e n t i o n a l number r i g h t method. 76 Sex d i f f e r e n c e s An e x a m i n a t i o n o f t h e v a r i a b l e o f s e x a s i t r e l a t e d t o t h e a r e a s i n v e s t i g a t e d i n t h i s s t u d y i s g i v e n i n T a b le 1 9. No s i g n i f i c a n t s e x d i f f e r e n c e e x i s t e d in mean a c h i e v e m e n t , t e s t v a l i d i t y , o r i n t h e p r o p o r t i o n sh o w in g s i g n i f i c a n t p o s i t i o n a l p r e f e r e n c e r e s p o n s e s e t s . The Mann-Whitney L t e s t showed th e boys had s i g n i f i c a n t l y h ig h e r in d e x e s o f r i s k a c c e p t a n c e on both the CAT and SRA t e s t s . T h is i n d i c a t e d t h a t g i r l s were more r e l u c t a n t to gamble on item s when in d o u b t. The g i r l s were a l s o shown t o ta k e more time t o c o m p le te the t e s t when a llo w e d to a ttem p t e v e r y ite m . Summary The f i n d i n g s o f 1he p r e s e n t i n v e s t i g a t i o n gave e v id e n c e f o r th e f o l l o w i n g summary s t a t e m e n t s . 1. The s t a n d a r d i z e d a ch iev em en t t e s t s used in t h i s stu d y were e s s e n t i a l l y power r a t h e r 1 linn snoed t e s t s . 2 . An e x a m in a tio n o f th e r i s k - a c o e p t a n c e in d e x e s o f th e exam inees showed a w id e sp r e a d i n c i d e n c e o f the s e t to gam ble. This te n d e n c y was s i g n i f i c a n t l y d e c r e a s e d by the u se o f the " d o n 't know" o p t i o n . The s e t was more p r e v a l e n t in boys than in g i r l s . 3 . There was a s i g n i f i c a n t n e g a t i v e r e l a t i o n s h i p betw een a b i l i t y and te n d e n c y to gam ble. 4 . Two o f th e t h i r t e e n t e s t s in th e s t a n d a r d i z e d 77 TABLE 1 9 . — S ex d i f f e r e n c e s in a c h i e v e m e n t , tim e , v a l i d i t y , and i n t h e g a m b lin g and p o s i t i o n a l p r e f e r e n c e r e s p o n s e s e t s G i r l s Hoys S i g n i f i c a n c e a Mean A ch ievem en t S c o r e s T o t a l CAT A r i t h m e t i c 5 0 .7 5 4 6 . 7 6 z = 1 .8 9 SRA A r i t h m e t i c C om putation 2 5 . 3 7 24 .13 z = .88 SRA A r i t h m e t i c R e a so n in g 19 .75 1 7 .8 2 z = 1 .5 6 Mean tim e r e q u i r e d to a tte m p t e v e r y ite m ( m in u t e s ) T o t a l CAT A r i t h m e t i c 62 .92 61 .48 X1 = 6 . 8 3 b T o ta l SRA A r i t h m e t i c 1 2 5 .5 2 TOO 1 O J . w 4 - , 0 1 b X 2 - 1 5 . 1 ° Val id i t y il T o ta l CAT A r i t h m e t i c .8 9 9 .893 z = .475 T o t a l SRA A r i t h m e t i c .7 4 0 .742 z = .0 0 2 Mean R i s k - a c c e p t a n c e Index^ T o t a l CAT A r i t h m e t i c .753 .819 z = 8 . 79C T o ta l SRA A r i t h m e t i c .9 10 .961 z = 1 1 . 3 ° Number s h o w i n g s i g n i f i c a n t i n t r a i n d i v i d u a l p o s i t i o n a l p r e f e r e n c e r e s p o n s e s e t s e 14 12 X 2 = .12 The s c o r i n r method y i e l d i n ' ' th' h i g h e s t v a l i d i t y c o e f f i c i e n t was used f o r e a c h t e s t . '’S i g n i f i c a n t a t th e .0 5 l e v e l . CS i g n i f l e a n t a t t h e .01 l e v e l . ^The Mann-Whitney U t e s t ( 6 7 : 1 1 6 ) was used to d e t e r m i n e s i g n i f i c a n c e . • * = .3 0 76 a c h ie v e m e n t b a t t e r i e s used In t h i s s t u d y showed s y s t e m a t i c k e y in g b i a s e s In t h e k e y in g o f t h e c o r r e c t o p t io n p o s i t i o n . 5 . I n t r a i n d i v i d u a l and i n t e r i n d i v i d u a l r e s p o n s e s e t s wore n o t p r e s e n t in t h e e l e m e n t a r y s c h o o l s a m p le . The two by two p o s i t i o n i n g o f th e o p t i o n s on th e answ er s h e e t may have c a n c e l e d out any " c e n t r a l ten d en cy " s e t . 6 . The r e s u l t s r e g a r d i n g th e most p r o p i t i o u s s c o r i n g method d i f f e r e d among t h e t e s t s . The b e s t r e s u l t s on th e CAT w ere o b t a in e d u s i n g th e c h a n c e c o r r e c t i o n , d i s c o u n t i n g i n c o r r e c t r e s p o n s e s in th e " d o n 't know" o p t i o n . This v a l i d i t y c o e f f i c i e n t was n ot s i g n i f i c a n t l y g r e a t e r , how ever, than t h o s e o b t a in e d u s i n g th e 3 -4 and 3 - 3 c u t o f f m e th o d s. On the SRA A r i t h m e t i c C om putation T e s t , th e f o r c e d — c o m p l e t i o n , th e number r i g h t , the 4 - 4 , 3 - 3 , and 3 -4 c u t o f f y i e l d e d th e h i g h e s t v a l i d i t y c o e f f i c i e n t s . None o f the o b t a in e d v a l i d i t i e s on th e SRA A r i t h m e tic R e a s o n in g l e s t d i f f e r e d s i g n i f i c a n t l y . 7. On e x a m in e e s fo r whom th e t e s t s were v e r y d i f f i c u l t th e 4 - 4 c u t o f f and c h a n c e - t h r e s h o l d methods were t h e o n l y methods c o n s i s t e n t l y y i e l d i n g s i g n i f i c a n t l y p o s i t i v e v a l i d i t v c o e f f i c i e n t s . The p o s s i b l e u se o f d i f f e r e n t s c o r i a r methods a t v a r i o u s a b i l i t y l e v e l s was s u g g e s t e d . 8 . The c o e f f i c i e n t s of c o i n c i d e n t v a l i d i t y were s i g n i f i c a n t l y h i g h e r (,or the CAT. I t was f e l t t h a t t h i s r e s u l t e d p r i m a r i l y from th e CAT's u se o£ th e " d o n 't know" 79 o p t i o n . A l t h o u g h t h i s o p t i o n I n t r o d u c e d a n o t h e r e e t t i t s a s s e t s s e e n t o o u t w e i g h i t s l i a b i l i t i e s . 9 . I t was s i g n i f i c a n t l y more d i f f i c u l t t o o b t a i n t h e same g r a d e p l a c e m e n t v a l u e on t h e SRA t e s t s as com p a r e d t o th e CAT t e s t s . T h is w ould seem t o h av e r e s u l t e d from the d i f f e r e n c e i n the t e s t r f o p e r a t i o n a l d e f i n i t i o n s o f g r a d e p l a c e m e n t . 1 0 . The t e s t c o n s t r u c t o r s appeared t o be s u c c e s s f u l in making d i s t r a c t o r s a t t r a c t i v e enough t o a l l u r e the m a j o r i t y o f e x a m in e e s who w e ie n ot su r e o f th e c o r r e c t answ er, r e s u l t i n g in the r e c a l l s c o r e s n o t b e in g s i g n i f i c a n t l y lo w er than the s c o r e s in the m u l t i p l e c h o i c e format. 11. No s i g n i f i c a n t s e x d i f f e r e n c e s were found in mean a c h ie v e m e n t s c o r e s , v a l i d i t y c o e f f i c i e n t s , or p o s i t i o n a l p r e f e r e n c e s e t s . S i g n i f i c a n t d i f f e r e n c e s were found, h ow ever, in the te n d e n c y to "amble s e t and in th e tim e r e q u ir e d to a t te m p t a l l i t e m s . CHAPTER V SUMMARY AND CONCLUSIONS, AND RECOMMENDATIONS Summary and C o n c lu s io n s The major p u r p o se s o f t h e p r e s e n t i n v e s t i g a t i o n were to d e t e r m in e w h eth er ( l ) gam bling and p o s i t i o n a l p r e f e r e n c e r e s p o n s e s e t s e x i s t e d in e l e m e n t a r y s c h o o l p u p i l s and (2) to e v a l u a t e th e e f f e c t s on v a l i d i t y when v a r i o u s s c o r i n g methods were used which a tte m p te d to c o r r e c t f o r v a r i a n c e in tr o d u c e d by chance and r e s p o n s e s e t s . The sample o f 246 f i f t h ?rade s t u d e n t s were d i v id e d i n t o two stroups on the b a s i s o f a c h ie v e m e n t and i n t e l l i g e n c e t e s t s g i v e n at th e b e g i n n i n g o f the s c h o o l y e a r . One group to o k th e a r i t h m e t i c s e c t i o n s o f th e SRA A ch ievem en t S e r i e s arid the o t h e r group to o k the c o r r e s ponding s e c t i o n s of th e C a l i f o r n i a A chievem ent T e s t s . A f t e r c o m p le t in g th e t e s t s in th e s t a n d a r d i z e d manner, each exam inee was a llo w e d as much a d d i t i o n a l time as n eces s a r y to a ttem p t e v e r y ite m . One week l a t e r an a l t e r n a t e form o f th e same a ch iev em en t s e r i e s was a d m in i s t e r e d , but w i t h o u t r e s p o n s e o p t i o n s , i . e . , in r e c a l l form. The p u p i l ' s s c o r e on t h i s r e c a l l form o f the t e s t was th e c r i t e r i o n measure as t h i s was i n t e r p r e t e d as an a p p r a i s a l o f h i s t r u e a b i l i t y t o d e a l w it h t h i s p a r t i c u l a r a c h i e v e ment a r e a w i t h o u t the c o n t a m in a t in g e f f e c t s from ch an ce and r e s p o n s e s e t s . 81 I t was n e c e s s a r y to e s t a b l i s h h y p o t h e s i s I, t h a t th e t e s t s were e s s e n t i a l l y power t e s t s , in o r d e r t o r u l e ou t t h e p o s s i b i l i t y o f sp e e d v a r i a n c e c o n t a m i n a t in g th e o t h e r f i n d i n g s . A ll t h r e e methods u sed t o a p p r a i s e sp e e d e f f e c t s showed the t e s t s t o b e e s s e n t i a l l y power t e s t s . To d e t e r m in e th e e x t e n t to which th e s e t to gamble was p r e s e n t in th e sa m p le , an i n d e x o f r i s k - a c c e p t a n c e d e v e l o p e d by Z i l l e r ( 8 5 ) was u s e d . This in d e x shows th e r a t i o o f th e number o f ite m s on which th e exam inee g u e s s e d but d id n o t know, to th e t o t a l number o f ite m s n ot known. H y p o t h e s i s I I , t h a t th e s e t to gamble was p r e s e n t in e l e m entary s c h o o l p u p i l s , was su p p o r te d as th e mean gam bling i n d e x e s were .614 and .9 3 0 f o r th e CAT and SRA t e s t s r e s p e c t i v e l y . The s i g n i f i c a n t l y low er in d e x on th e CAT would seem to have r e s u l t e d from i t s u se o f th e a l t e r n a t i v e " d o n 't know” o p t i o n where the ex a m in e e i s d i r e c t e d to respond to the l a s t o p t io n when he c a n n o t work a problem , or i f he t h i n k s none of the an sw ers g iv e n i s c o r r e c t . Boys were snown to have s i g n i f i c a n t l y high* r in d e x e s on both a c h ie v e m e n t s e r i e s . H y p o t h e s i s I I s u g g e s t e d t h a t t h e r e was a s i g n i f i c a n t p o s i t i v e r e l a t i o n s h i p betw een a b i l i t v and p r o b a b i l i t y o f f o l l o w i n g d i r e c t i o n s , i . e . , t h e " d o n 't know" d i r e c t i o n s . By o b s e r v i n g an e x a m i n e e ' s perform an ce a f t e r he had r e a c h e d h i s ch an ce t h r e s h o l d , i t was n ot d i f f i c u l t t o d e t e r m in e which e x a m in e e s were f o l l o w i n g d i r e c t i o n s . The 62 h y p o t h e s i s was s u p p o r t e d ; t h e o b t a in e d c o e f f i c i e n t o f c o n t i n g e n c y b e i n g . 3 2 3 . Two o f th e t h i r t e e n s t a n d a r d i z e d a c h ie v e m e n t t e s t s in th e b a t t e r i e s used in t h i s s t u d y w ere shown to have s y s t e m a t i c b i a s i n g it th e k e y i n g o f th e c o r r e c t o p t i o n p o s i t i o n . H y p o t h e s i s IV w as, t h e r e f o r e , n o t s u p p o r t e d in roost o f the t e s t s . No e v i d e n c e was o b t a in e d in s u p p o r t of the i n t r a - i n d i v i d u a l or i n t e r i n d i v i d u a l p o s i t i o n a l p r e f e r e n c e r e sp o n s e s e t s . H y p o th e s e s V and VI were c o n s e q u e n t l y r e j e c t e d . The p o s i t i o n i n g o f th e r e s p o n s e o p t i o n s on t h e answ er s h e e t s o f th e t e s t u se d may have c o u n t e r a c t e d any " c e n t r a l ten d en cy " s e t . The SRA answ er s h e e t a r r a n g e s th e r e s p o n s e o p t i o n s in a two by two c o n f i c u r a t i o n . The r e s u l t s r e g a r d i n g t h e roost e f f i c a c i o u s s c o r i n g method d i f f e r e d among th e t e s t s . The CAT y i e l d e d th e h i g h e s t v a l i d i t y when c o r r e c t e d f o r c h a n c e , d i s c o u n t b e in g g iv e n f o r r e s p o n s e s in th e " d o n 't know" o p ti in. T h is v a l i d i t y was not s i 'ru i f ica n t 1 y hi'/h^r than t h a t o b t a in e d u s in g th e 3-4 and 3 -3 c u t o f f m eth od s, how ever. On th e SRA c o m p u ta tio n t e s t none o f th e f o l l o w i n g methods y i e l d e d s i g n i f i c a n t l y d i f f e r e n t v a l i d i t y c o e f f i c i e n t s : f o r c e d c o m p l e t i o n , c o n v e n t i o n a l number r i g h t , 4-4 c u t o f f , 3 — 3 c u t o f f , and 3-4 c u t o f f . None of the methods used on th e SRA r e a s o n in ' t e s t y i e l d e d s i g n i f i c a n t l y d i f f e r e n t v a l i d i t i e s . 63 For th o s e exam inees who found the t e s t s v e r y d i f f i c u l t , i . e . , th e 41.69* f o r whom the n u l l h y p o t h e s i s o f no know ledge c o u ld be r e j e c t e d on the SHA t e s t s , th e 4— 4 c u t o f f and c h a n c e - t h r e s h o l d t e c h n i q u e s were th e o n ly methods c o n s i s t e n t l y y i e l d i n g s i g n i f i c a n t v a l i d i t y c o e f f i c i e n t s . The forced c o m p le t i o n method was found to have d i s t i n c t a d m i n i s t r a t i v e d i s a d v a n t a g e s ; t e s t - t a k i n g tim e was a p p r o x im a t e ly d o u b le d . I t a l s o y i e l d e d the lo w e s t v a l i d i t y c o e f f i c i e n t on the CAT w ith the e x c e p t i o n o f th e c o n v e n t i o n a l number r i g h t m ethod. On th e SRA t e s t s the f o r c e d c o m p le t i o n method was not s t a t i s t i c a l l y s u p e r i o r to o t h e r methods u se d , y e t had th e c o n c o m ita n t a d m i n i s t r a t i v e p ro b 1em s. H y p o th eses V i l a , V l l d , and V i l e w ere, t h e r e f o r e , s u p p o r te d on th e CAT. H y p o th eses V l l f and V l l g were sup p orted on th e SUA o n ly f o r th e low a b i l i t y s t u d e n t s . The c o e f f i c i e n t s o f c o i n c i d e n t v a l i d i t y were s i g n i f i c a n t l y h ig h e r for th e CAT. This appeared to r e s u l t from the CAT's use of th e " d o n 't know" o p t io n and, to some e x t e n t , th e s m a l l e r p e r c e n t a g e o f exam inees w ith c h a n c e — l e v e l s c o r e s . R e l i a b i l i t i e s o f the two t e s t s were com p a ra b le when c o n s i d e r i n g the e n t i r e b a t t e r i e s . The f i n d i n g s i n d i c a t e d t h a t i t was s i g n i f i c a n t l y more d i f f i c u l t to o b t a in the same grade placem ent v a l u e on th e SHA t e s t s as compared to th e CAT, This appeared t o r e s u l t from a d i f f e r e n c e in th e o p e r a t i o n a l d e f i n i t i o n s 84 o f t h e m eaning o f a grad e p la cem en t v a l u e . The t e s t c o n s t r u c t o r s appeared to be v e r y s u c c e s s f u l in making d i s t r a c t o r s a t t r a c t i v e enough t o a l l u r e th e m a j o r i t y o f e x a m in e e s who w ere not s u r e o f th e c o r r e c t a n sw e r . T h is was s'lowti by th e f a c t t h a t t h e r e c a l l s c o r e s were n ot s i g n i f i c a n t l y lo w e r than th e r e c o g n i t i o n s c o r e s . This f i n d i n g p r o b a b ly a c c o u n te d f o r th e r e l a t i v e l y i n e f f e c t i v e n e s s o f the c o r r e c t i o n f o r c h a n c e ( e x c e p t when the " d o n ’ t know" a l t e r n a t i v e was u se d ) in a f f e c t i n g r e l i a b i l i t y and v a l i d i t y f a v o r a b l y . I t would seem t h a t some e x a m in e e s were u n d u ly c a u t i o u s ..hen the " d o n ’ t know" o p t i on was p r e s e n t and resp o n d e d to t h i s o p t io n a t t im e s , when t h e y were c a p a b l e o f c o r r e c t l v w ork in g the p roblem . The r e c a l l s c o r e s were s i g n i f i c a n t 1y h i g h e r than th e r e c o g n i t i o n s c o r e s , when th e " d o n 't know" o p t i o n was p r e s e n t . IJven though a s e t i s p r e s e n t , th e o v e r a l l e f f e c t s are b e n e f i c i a l to v a l i d i t y . S i g n i f i c a n t s e x d i f f e r e n c e s e x i s t e d in the s e t to gamble arid th e mean tim e r e q u i r e d to a tte m p t a l l i t e m s . The boys w ere more l i k e l y to i' urss when t h e y d id n o t know how to work a problem and more tim e was n e c e s s a r y f o r th e g i r l s to r e sp o n d to e v ery i t e m . No s e x d i f f e r e n c e s e x i s te d in mean a c h ie v e m e n t , v a l i d i t y , or p o s i t i o n a l p r e f e r en ce s e t s . 85 Recommendations 1 . I f a p o s i t i o n a l p r e f e r e n c e s e t was p r e s e n t in t h i s sa m p le, i t was a p p a r e n t ly n u l l i f i e d by the u n u su a l p o s i t i o n i n g o f th e o p t i o n s on th e answ er s h e e t , i . e . , in a two by two s p a t i a l c o n f i g u r a t i o n . T h e r e f o r e , u n t i l i t has b een u n e q u i v o c a l l y e s t a b l i s h e d t h a t no such r e s p o n s e s t y l e e x i s t s in th e group f o r which a t e s t i s d e s i g n e d , t e s t c o n s t r u c t o r s would sta n d l e s s ch an ce fo r a l l o w i n g i r r e l e v a n t p o s i t i o n a l s e t v a r i a n c e t o e n t e r in to th e t e s t s c o r e s by u s in g t h i s answ er s h e e t f o r m a t . 2 . I t i s recommended t h a t t e s t p u b l i s h e r make e x p e r im e n ta l i n v e s t i g a t i o n s t o d e te r m in e which s c o r i n g procedure i s most a p p r o p r i a t e for- a p a r t i c u l a r t e s t . Much o f th e c o n f u s i o n in th e a r e a o f t e s t s c o r i n g ap p ea rs to have r e s u l t e d from i n v e s t i g a t o r s , a f t e r f i n d i n g an advan t a g e by u s i n g a p a r t i c u l a r s c o r i n g method on a p a r t i c u l a r t e s t , a t t e m p t in g to g e n e r a l i z e t h i s f in d i n g to a l l o t h e r t e s t s w ith o u t c o n s id e r i n ' ' th e g r e a t amount o f s i t u a t i o n a l f a c t o r s p r e s e n t , e . g . , the d e g r e e o f speed i n v o l v e d , item d i f f i c u l t y , r e l a t i v e a t t r a c t i v e n e s s o f d i s t r a c t o r s , p r e v i ous e x p e r i e n c e o f e x a m in e e s , e t c . VTien i t i s n e c e s s a r y fo r a t e s t to c o v e r a wide ran ge o f a b i l i t y , th e i n v e s t i g a t i o n o f use o f d i f f e r e n t i a l s c o r i n g methods at v a r i o u s a b i l i t y l e v e l s i s recommended. 3 . The p r e s e n t s tu d v has shown f a v o r a b le r e s u l t s a c c r u in g from th e u s e o f the a l t e r n a t i v e " d o n 't know” 86 o p t i o n , i . e . , i n s t r u c t i n g th e exam inee to respond t o a c e r t a i n o p t i o n p o s i t i o n i f he c a n n o t work a problem, o r i f he f e e l s none o f th e answers g i v e n i s c o r r e c t . C onsequent l y , more w id e sp r e a d u se o f t h i s ty p e o f o p t i o n i s recom mended. The t e s t s u s i n g t h i s o p t io n had g r e a t e r v a l i d i t y and lo w er mean r i s k - a c c e p t a n c e i n d e x e s . I t would appear t h a t t h i s i s a method by which chance v a r i a n c e on m u l t i p l e r e s p o n s e can be r e d u c e d . I t i s r e a s o n a b l e to b e l i e v e t h a t t h i s o p t io n c o u ld be used a d v a n t a g e o u s l y on a p t i t u d e t e s t s as w e ll as on a c h ie v e m e m t e s t s . T h is o p t io n , in o r d e r to i n c r e a s e v a l i d i t y , sh o u ld p ro b a b ly be i n c o r r e c t much more o f t e n than chance would d i c t a t e , but be keyed c o r r e c t on s e v e r a l l e s s d i f f i c u l t ite m s near th e b e g in n in g o f th e t e s t in o r d e r to r e i n f o r c e t h i s r e s p o n se p a t t e r n . A c o r r e c t i o n fo r chance a lo n g w ith the use o f the " d o n 't know" o p t io n i s a l s o recommonded. I f th e r e i s a p r a c t i c a l c r i t e r i o n a v a i l a b l e , an e m p i r i c a l l y d eterm in ed o ptim al w e ig h t f o r th e vvron," r e s i'o n s e s v.ould be p r e f e r a b l e . 4 . I t i s recommended th a t t e s t c o n s t r u c t o r s s tu d y and r e p o r t c o e f f i c i e n t s of c o i n c i d e n t v a l i d i t y where p o s s i b l e . S in c e th e p o s s i b i l i t i e s for e s t a b l i s h i n g em p i r i c a l v a l i d i t y o f a c h ie v e m e n t t e s t s are v e r y l i m i t e d , i t would seem r e a s o n a b le to i n s i s t th a t t e s t p u b l i s h e r s e s t a b l i s h to what d e g r e e th e r e s u l t s from t h e i r t e s t s p r e d i c t perform ance on the more l o r i c a l l y v a l i d r e c a l l 87 t e s t . W ithout such e v i d e n c e , chance and r e s p o n s e s e t v a r i a n c e c o u l d be a major s o u r c e o f t e s t v a r i a n c e and y e t go u n i d e n t i f i e d . As p r e v i o u s l y i n d i c a t e d , t h e s e f a c t o r s may a c t u a l l y enhance t e s t r e l i a b i l i t y , o f t e n th e o n l y s t a t i s t i c a l i n f o r m a t i o n g i v e n f o r a c h ie v e m e n t t e s t s . E v i dence o f a t e s t ' s c o i n c i d e n t v a l i d i t y i s recommended as a r e q u i s i t e s t e p b e f o r e p u r p o r t in g c o n t e n t v a l i d i t y . BIBLIOGRAPHY BIBLIOGRAPHY 1 . 2 . 3 . 4 . 5 . 6 . 7. 8 . 0 . 1 0 . 1 1 . 12. A n a s t a s i , Anne. P s y c h o l o g i c a l T e s t i n g . New York: The M acm illan Company, 1 6 5 4 . B e n n e t t , G. K. "Item D i f f i c u l t y and Speed o f Re s p o n s e s , n E d u c a t io n a l and P s y c h o l o g i c a l M easure m ent. 1 6 : 4 ^ 4 - 4 9 6 , 1 §56. Berg, I . A. "Response B ia s and P e r s o n a l i t y D e v i a t i o n H y p o t h e s i s , " J o u r n a l o f P s y c h o lo g y . 4 0 : 6 1 - 72, 1955. Bryan, M. M., and O th e r s. " C o r r e c tio n f o r G u e ssin g in S c o r i n g o f P r e t e s t s , E f f e c t Upon Item D i f f i c u l t y and T h e ir V a l i d i t y , " E d u c a t io n a l and P sy ch o l o g i c a l M easurem ent, 12 :45-3'6, 1 § 5 2 . B r o w n le e s, V. T ., and K e a t s , J . A. "A R e t e s t Method o f S tu d y in g P a r t i a l Knowledge and Other F a c to r s I n f l u e n c i n g Item R e s p o n se ," P s y c h o m e tr ik a , 2 3 : 6 7 - 73, 1958. C a l i f o r n i a T e s t Bureau. T e c h n i c a l R eport on C a l i f o r n i a A chievem ent T e s t s . Los A n g e le s , 1957. C a r r o l l , J . B. "The E f f e c t on D i f f i c u l t y and Chance S u c c e s s on C o r r e l a t i o n s Between Item s or Between T e s t s , " P s y c h o m e tr ik a , 1 0 : 1 - 1 9 , 1945. C l a r 1;, E. C. "General R esponse P a t t e r n s to F i v e - C h oice I te m s," J o u rn a l o f E d u c a t io n a l P s y c h o lo g y , 4 7 : l l o _ U 7 , 1953"; C l i f f , Rosemary. "The P r e d i c t i v e V alue o f C hance- L evel S c o r e s ," E d u c a tio n a l and P s y c h o l o g i c a l M easurem ents. 1 8 : 6 0 7 - 6 1 6 , 1958. Cook, D. L. "A Comparison o f Reading Comprehension S c o r e s O btained B e fo r e and A f t e r a Time Announce ment," J o u r n a l o f E d u c a t io n a l P s y c h o lo g y , 4 8 : 4 4 0 - 446, 19&T. Cook, W. ff. "Achievement T e s t s , " in W. S. Monroe (E d .) E n c y c lo p e d ia o f E d u c a tio n a l R e s e a r c h . New York: The M acm illan Company, pp. 1 4 6 1 -1 4 7 8 , 1950. Coombs, C. P ., and O th e r s . "The A s se s sm e n t o f Par t i a l K now ledge," E d u c a t io n a l and P s y c h o l o g i c a l M easurement, 1 6 : l i - 3 7 , 1956. 13. 1 4. 15. 16. 17. 18. 19. 2 0 . 21 . 22 . 23. 2 4. 90 C ro n b a c h , L. J . E s s e n t i a l s o f P s y c h o l o g i c a l T e s t i n g . New Y ork: H a r p e r and B r o t h e r s , 1 9 4 9 . E s s e n t i a l s o f P s y c h o l o g i c a l T e s t i n g . 2nd e d . New Y o rk : h a r p e r and B r o t h e r s , 1960. " F u r t h e r E v i d e n c e on R e s p o n s e S e t s and T e s t D e s i g n , " E d u c a t i o n a l and P s y c h o l o g i c a l M e a s u r e m e n t. 1 0 : 3 - 3 1 t 1 9 5 0 . . "Response S e t s and T e s t V a l i d i t y , " Educa- t l o n a l and P s y c h o l o g i c a l M easurem ent. 6 : 4 7 5 - 4 9 4 , 1946. ' ___________, and Merwin, J . C. "A Mode 1 f o r S tu d y in g th e V a l i d i t y o f M u l t i p l e C h oice I te m s ," Educa t i o n a l and P s y c h o l o g i c a l M easurem ent. 1 5 :^ 3 7 -3 5 2 , 1 9 5 5 . , and W arrington, W. G. "T im e-L im it T e s t s : E s t im a t in g T h e ir R e l i a b i l i t y and Degree o f S p e e d - i n g , " P s y c h o m e tr ik a , 1 6 : 1 6 7 - 1 8 8 , 1951. D a v id so n , »>. M,, and C a r r o l l , .7. B. "Speed and l e v e l Components in T im e-L im it S c o r e s : A F a c to r A n a ly s i s , " E d u c a t io n a l and P s y c h o l o g i c a l M easurem ent, 5 : 4 1 1 - 4 2 7 , 1 9 4 5 . D a v is , F. B. "Item A n a l y s i s in R e l a t i o n to Educa t i o n a l and P s y c h o l o g i c a l T e s t i n g , " Psycho l o g i c a l B u l l e t i n . 4 9 : 9 7 - 1 2 1 , 1952. ___________. "Use o f th e C o r r e c t i o n f o r Chance S u c c e s s in T e st S c o r i n g ," J o u r n a l o f E d u c a t io n a l R e s e a r c h , 5 2 : 2 7 9 - 2 8 0 , 1959. D o p p e lt, J . E. "The C o r r e c t i o n fo r G u e ss in g ," T est S e r v i c e B u l l e t i n , No. 4 6 . The P s y c h o l o g i c a l C orporat io n , 1954 . Dunlap, J , W., and O th e r s . "The E f f e c t s o f D i f f e r e n t D i r e c t i o n s and S c o r in g Methods on th e R e l i a b i l i t y o f a T r u e - F a ls e T e s t ," S ch o o l and S o c i e t y , 30: 3 7 8 - 3 8 2 , 1 9 29. Edwards, A l l e n L. E x p e r im e n ta l D esig n in P s y c h o l o g i c a l R e s e a r c h . 2nd e d . New York: R in e h a r t and Company, Inc ., 1 9 60. 91 2 5 . 2 6 . 2 7 . 2 8 . 2 9 . 3 0 . 3 1 . 3 2 . 33 . 3 4 . 3 5 . 3 6 . P a l r b e r , N. N. "An E x p e r i m e n t a l S t u d y o f t h e R e l a t i v e D i f f i c u l t y o f H a n d - S c o r e d and M a c h i n e - S c o r e d V e r s i o n s o f t h e Same T e s t , " B u l l e t i n o f t h e N a t i o n a l I n s t i t u t e o f P e r s o n n e l R e s e a r c h , 3 : 1 0 - 1 9 , F e d e r , D. D. " E f f e c t o f D i r e c t i o n s and A r r a n g e m e n t o f I te m s on S t u d e n t P e r f o r m a n c e i n a T e s t , " J o u r n a l o f E d u c a t i o n a l R e s e a r c h , 3 0 : 2 8 - 3 5 , 1 9 36. F r ie d m a n , M o rto n , and F l e i s h m a n , E. A. "A Note on t h e Use o f a 'D o n ’ t Know* A l t e r n a t i v e i n M u l t i p l e - C h o i c e T e s t s , " J o u r n a l o f E c ta c a t i o n a l P s y c h o l o g y , 4 7 : 3 4 4 - 4 9 , 1 9 5 6 . F r u c h t e r , B. " D i f f e r e n c e s in F a c t o r C o n ten t o f R i g h t s and Wrongs S c o r e s , " P s y c h o m e tr ik a , 1 8 : 2 5 7 - 2 5 6 , 1953. G a ie r , E. L ,, and O th e r s . "Response P a t t e r n s in a T e s t o f L o g i c a l I n f e r e n c e , " E d u c a t io n a l and P sy c h o l o g i c a l M easurem ent, 13 : 5& 0-567, l$f>3 . Gordon, L. V. "The E f f e c t o f P o s i t i o n on th e P r e f e r en ce V alu e o f P e r s o n a l i t y I te m s ," E d u c a t io n a l and h o l o g l c a l M easurem ent, 1 2 : 6 6 9 - ^ 7 6 , 1952. ___________ . " V a l i d i t i e s o f th e F o r c e d -C h o ic e and Q u e s t i o n n a i r e Methods o f P e r s o n a l i t y M easurem ent," J o u r n a l o f A p p lie d P s y c h o l o g y , 3 5 : 4 0 7 - 4 1 2 , 1 9 51. G r a e s s e r , Tt. F. " G u e ssin g on M u l t i p l e - C h o i c e T e s t s , " E d u c a t io n a l and P s y c h o l o g i c a l M easurement, 18: KlY-dStf, 195>8 .--------- G r a n ic h , L. A. "A T echn iq u e fo r E x p e r im e n tin g on G u e ss in g in O b j e c t i v e T e s t s , " J o u r n a l o f Educa t i o n a l P s y c h o l o g y , 2 2 : 1 4 5 - 1 5 6 , "1931. G reen e, E. B. Measurement o f human B e h a v i o r . Re v i s e d ed . The OdiysVey P r e s s " , T952™I G r i t t e n , F ., and Joh n son , D. . ’!. " I n d i v i d u a l D i f f e r e n c e s in J u d g in g M u l t i p l e C h o ice Q u e s t i o n s , " Jo u rn a l o f E d u c a t io n a l P s y c h o l o g y , 3 2 : 4 2 3 - 4 3 0 , 1 9 4 1 . G u i l f o r d , J . P. F u n d a m e n ta l S t a t i s t i c s i n P s y c h o l o g y and E d u c a t i o n . 3 r d e d . Mow V o r k : il c G r a w - H i l l Book Company, I n c . , 1 9 5 6 . 3 7 . 3 8 . 3 9 . 4 0. 4 1. 4 2 . 4 3. 44 . 4 5. 4 6. 47. 46 . 4 9. 5 0. 92 P s y c h o m e t r i c M e t h o d s . 2nd s d . New Y ork: M cG raw -H ill fiook Company, I n c . f 1 9 54. G u l l i k s s n , H. " I n t r i n s i c V a l i d i t y , " A m e ric a n P sy c h o l o g i s t . 5 : 5 1 1 - 5 1 7 , 1 9 50. . "The R e l i a b i l i t y o f S p eed ed T e s t s , ” P s y c h o m e t r i k a , 1 5 : 2 5 9 - 2 6 9 , 1950. __________. Theory o f Mental T e s t s . New York: W iley and Sons^ I n c . , 19 50. H am ilton , C. II. "Bias and Error in M u l t i p l e - C h o i c e T e s t s , ” P s y c h o m e tr ik a , 1 5 : 1 5 1 - 1 6 6 , 1950. Rawkes, h. E ., L i n d q u i s t , E, F ., and Mann, C. 11. The C o n s t r u c t i o n and Use o f A chievem ent Examina t i o n s . B o sto n : Houghton— M iff! in Company, 1 936. H e l m s t a d t e r , G. C. " P roced u res f o r O b ta in in g S e p a r a t e S e t and C ontent Components o f a T e st S c o r e , ” P s y c h o m e tr ik a , 2 2 : 3 8 1 - 3 9 3 , 1957. ___________, and Ortmeyer, 0. IT. "Some T echn iq u es fo r D eterm in in g th e R e l a t i v e Magnitude o f Speed and Power Components o f a T e s t ," E d u c a tio n a l and P sy c h o l o g i c a l M easurem ent, 1 3 : 2 8 0 - 2 8 7 , 1953. H o r s t , P. "The Chance Elem ent in the M u l t i p l e C h oice T est Item ," Jo u rn a l o f General P s y c h o lo g y , 6 : 2 0 9 - 211, 1 932. J a c k s o n , It. A. "G uessing and T est Perform an ce," E d u c a tio n a l and P s y c h o l o g i c a l Measurement, 15: "f4— 79, 1 9 55. Lee, J . M., and Symonds, P. M. "New-Type o f O b je c t iv e T e s t s , A Summary o f R ecen t I n v e s t i g a t i o n s , " J o u r n a l o f E d u c a t io n a l P s v c h o lo g y , 2 5 : 1 6 1 - 1 8 4 , T 9 ' 3 "4 ; “ L i n d q u i s t , E. F. ( e d . ) . E d u c a t i o n a l M e a s u r e m e n t. W a s h in g to n , A m erican C o u n c il on E d u c a t i o n , 1951. Lord, F. M. "The R e l a t i o n o f th e R e l i a b i l i t y o f M u l t i p l e - C h o i c e T e s t s to th e D i s t r i b u t i o n o f Item D i f f i c u l t i e s , " P s y c h o m e tr ik a . 1 7 : 1 8 1 - 1 9 4 , 1 9 52. Lorr, M, "I n t e r r e l a t l o n s h i p s o f Number C o r r e c t and Limen S c o r e s fo r an Amount-Lim it T e s t ," P sy c h o m e t r i k a , 1 6 : 2 1 - 3 0 , 1951. 93 5 1 . 5 2 . 5 3. 54. 5 5. 5 6. 5 7. 58. 59. 60 . Los A n g e le s C o u n ty S u p e r i n t e n d e n t o f S c h o o l s , D i v i s io n o f R e s e a r c h and G u i d a n c e . "A S tu d y C om paring t h e C a l i f o r n i a A c h ie v e m e n t T e s t w i t h t h e SRA A c h ie v e m e n t S e r i e s f o r R e a d in g and A r i t h m e t i c f o r G rad es 5 and 6 , " Los A n g e l e s , 1 9 59. (M im eo g rap h ed ) McNamara, W. J . , and W ertzman, E. "The E f f e c t o f C h o ic e P la c e m e n t on t h e D i f f i c u l t y o f M u l t i p l e - C h o ic e q u e s t i o n s , " J o u r n a l o f E d u c a t i o n a l P sy - c h o l o g y , 3 6 :1 0 3 -1 1 3 1 1943. M erwin, J . C. " R a t i o n a l and M a t h e m a t i c a l R e l a t i o n s h i p o f S i x S c o r i n g P r o c e d u r e s A p p l i c a b l e to T h r e e - C h o i c e I t e m s , " J o u r n a l o f E d u c a t i o n a l P s y c h o l o g y , 5 0 : 1 5 3 - 1 6 1 . M e t f e s s e l , N. S . "An E x p e r im e n ta l A n a l y s i s o f Re s p o n se S e t s in Forced C h oice T e s t P erform an ce." U n p u b lish ed Ph.D. d i s s e r t a t i o n , U n i v e r s i t y o f Southern C a l i f o r n i a , 1955. ___________ , and Sax, G i l b e r t . " S y s te m a tic H ia se s in the Keyin': o f C o r r e c t R esp o n ses on C e r t a in S ta n d a r d iz e d T e s t s , " E d u c a tio n a l and Psych o l o g i c a l M easurem ent, 1 8 : 7 8 7 -7 9 0 , 1958. M ic h a e l, i*. H. "Developm ent o f S t a t i s t i c a l Methods E s p e c i a l l y U s e fu l in T e s t C o n s t r u c t i o n and E v a lu a t i o n , " Review o f E d u ca tio n R e se a r c h , 2 6 : 8 9 - 1 0 9 , 1 9 5 6 . ___________and O th e r s. "Abaca fo r D e te r m in a tio n o f a C o r r e l a t i o n C o e f f i c i e n t C o r r e c te d fo r R e s t r i c t i o n o f Range." A b s t r a c t o f a paper p r e s e n te d a t the m e e tin g s o f the American P s y c h o l o g i c a l A s s o c i a t i o n , D i v i s i o n 5, September 9, 1959, C i n c i n n a t i , Ohio. Morton, U. L. T eachin g A r i t h m e t i c : .Vhat R esea rch Says to th e T e a c h e r . W ashington: N a tio n a l Edu c a t i o n a l A s s o c i a t i o n , 1953. M oaicr, C. I . " P s y c h o p h y sic s and Mental T e s t Theory: Fundamental P o s t u l a t e s and Elem entary Theorems," P s y c h o l o g i c a l R e v ie w , 4 7 : 3 5 6 - 3 6 6 , 1940. " P s y c h o p h y sic s and Mental T e s t Theory: 1i . The C o n sta n t P r o c e s s ," P s y c h o l o g i c a l Review, 4 8 : 2 3 5 - 2 4 9 , 1941. 94 6 1. 6 2 . 6 3 . 64 . 6 5 . 6 6 . 6 7. 6 8 . 69 . 7 0. 7 1. 72 . M y e rs , C. T. "The F a c t o r i a l C o m p o s i t i o n and V a l i d i t y o f D i f f e r e n t l y S p e e d e d T e s t s , " P s y c h o m e t r i k a , 1 7 : 3 4 7 - 3 5 4 , 1 9 5 2 . O s b u r n , H. G ., and O t h e r s . "The R e l a t i v e V a l i d i t y o f F o r c e d C h o i c e and S i n g l e S t i m u l u s S e l f D e s c r i p t i o n I t e m s , " E d u c a t i o n a l and P s y c h o l o g i c a l M e a s u r e m e n t . 1 4 : 4 0 7 - 4 1 7 , 1 $ 5 4 . P lu m le e , L. B. "The P r e d i c t e d arid Observed E f f e c t o f Chance S u c c e s s on M u l t i p l e - C h o i c e T e s t V a l i d i t y , " P s y c h o m e t r i k a , 1 9 : 6 5 - 7 0 , 1954. R a p a p o rt, G. M., and Berg, I . A. "Response S e t s in a M u l t i p l e - C h o i c e T e s t , " E d u c a t io n a l and P s y c h o l o g i c a l M easurem ent, 1 5 : 5 8 - 6 2 , 1 9 5 5 . Ruch, G. M., and S to d d a r d , G. D. "Com parative R e l i a b i l i t i e s o f F iv e Types o f O b j e c t i v e E xam inations," J o u r n a l o f E d u c a t i o n a l P s y c h o lo g y , 1 6 : 8 9 - 1 0 3 , 1 9 2 5 . ___________. T e s t s and M easurements in High S c h o o l In- s t r u c t i o n . Y on k ers, New York: World Book Company, T W F T .---------- S e i g e l , S. N o n p a ra m etrie S t a t i s t i c s f o r th e B e h a v i o r a l S c i e n c e s . New York: McGraw-Hill Book Com pany, Inc . , 1956. S h i r r i f f s , A. C ., and Boomer, IJ. S . "Who i s P e n a l iz e d by th e P e n a l t y fo r Guessing'?" J o u r n a l o f E d u c a t io n a l P s y c h o l o g y . 4 5 : 6 1 - 9 0 , 1 954. S o d e r q u i s t , P . "A New Method o f W eig h tin g S c o r e s in a T r u e - F a ls e T e s t ," J o u rn a l o f E d u c a t io n a l R e s e a r c h , 3 0 : 2 9 0 -2 9 2 1 1 9 3 6 . S t a n l e y , J . C. " P s y c h o l o g i c a l C o r r e c t i o n f o r C hance," J o u r n a l o f E d u c a t io n a l P s y c h o l o g y , 45: 1 2 9 - 1 4 2 , 1 9 5 4 . . "The Measurement o f a P e r s o n a l i t y T r a i t ; " J o u r n a l o f E d u c a t io n a l P s y c h o l o g y . 2 9 : 2 9 5 - 3 0 0 , 1 9 38. S w in e f o r d , I’. " A n a ly s is o f a P e r s o n a l i t y T ra it,," J o u r n a l o f E d u c a t io n a l P s y c h o lo g y , 3 2 : 4 3 6 - 4 4 4 , r m r ------------------------- -------------------------------- 95 73. 74. 75. 76. 77. 76. 79. 80 . 61 . 82 . 83. ___________, and M i l l e r , P. M. " E f f e c t s o f D i r e c t i o n s R e g a r d i n g G u e s s i n g on I te m S t a t i s t i c s on a M u l t i p l e C h o ic e V o c a b u l a r y T e s t , J o u r n a l o f Edu c a t i o n a l P s y c h o l o g y . 4 4 : 1 2 9 -1 3 9 , 1 9 5 3 . T a t e , M. W. " I n d i v i d u a l D i f f e r e n c e s i n S peed o f R e s p o n s e in M en ta l T e s t M a t e r i a l s o f V a r y i n g D e g re e s o f D i f f i c u l t y , " E d u c a t i o n a l and P s y c h o - o g l c a l M e a s u r e m e n t. 8 : 353 -3 7 4 , 1948. T h o r p e , L. P . , L e f e v e r , D. W e lty , and N a s lu n d , R. A. SRA A c h ie v e m e n t S e r i e s , Forms A & D. E x a m in e r M a n u a l^ 2nd e d . C h i c a g o : S c i e n c e R e s e a r c h A s s o c i a t e s , 1956. T e c h n i c a l S u p p l e m e n t . SEA A c h ie v e m e n t S e r i e s . 2nd e d . C h i c a g o : S c i e n c e R e s e a r c h A s s o c i a t e s , 1957. T h u r s t o n e , L. L. The R e l i a b i l i t y and V a l i d i t y o f T e s t s . Ann A r b o r, M i c h i g a n : Edwards B r o t h e r s ', TncT7 1931. T i e g s , E. W., and C l a r k , W . V . C a l i f o r n i a A c h i e v e ment T e s t s . E l e m e n t a r y B a t t e r y M a n u a l. Los A n g e l e s : C a l i f o r n i a T e s t B u re a u , 1957. T i n k e r , M. A. "Speed v s . C o m p re h e n s io n in R e a d in g as E f f e c t e d by L e v e l o f D i f f i c u l t y , " J o u r n a l o f E d u c a t i o n a l P s y c h o l o g y , 3 0 : 8 1 -9 4 , 1939. T o r r e n c e , E. P . , and Z i l l e r , R. C. " R i s k and L i f e E x p e r i e n c e D e v e lo p m e n t o f a S c a l e f o r M e a s u r in g R i s k - T a k i n g T e n d e n c i e s , " L o c k la n d A i r F o rc e B a se , T e x a s : A i r F o rc e P e r s o n n e l and T r a i n i n g R e s e a r c h C e n t e r , AFPTRC R e s e a r c h R e p o r t 5 7 -2 3 , 1957; c i t e d by H a m ilto n ( 4 0 ) . Votaw, D. F. "The E f f e c t o f ’Do Not G u e s s ' D i r e c t i o n s on t h e V a l i d i t y o f T r u e - F a l s e and M u l t i p l e - C h o ic e T e s t s , " J o u r n a l o f E d u c a t i o n a l P s y c h o l o g y . W est, P. V. "A C r i t i c a l S tu d y of t h e R i g h t Minus Wrong M e th o d ," J o u r n a l o f E d u c a t i o n a l R e s e a r c h . 8 : 1 - 9 , 1929. Whitcomb, M. A. "IBM A nsw er S h e e t s as a M ajo r S o u r c e i n V a r i a t i o n on H i g h l y S p eed ed T e s t s , " E d u c a t i o n a l and P s y c h o l o g i c a l M e a s u r e m e n t , 1 8 :7 5 7 -7 5 9 , 1958. 96 8 4 . 8 5. Wood, B. D. " S t u d i e s i n A c h ie v e m e n t T e s t s , " J o u r n a l o f E d u c a t i o n a l P s y c h o l o g y . 1 7 : 1 - 2 2 , 1926. Z i l l e r , E. C. "A M easu re o f th e G abm llng R e s p o n s e S e t i n O b j e c t i v e T e s t s , " P s y c h o m e t r i k a , 2 2 : 2 8 9 - 292, 1957.

Linked assets

University of Southern California Dissertations and Theses

Conceptually similar

PDF

An Empirical Study Comparing Various Methods Of Achievement Expectancy Bymeans Of Mental Ability

PDF

Positive And Negative Verbal Reinforcement In The Conditioning Of The Verbal Behavior Of Adolescent Underachieving Delinquents With Dependent, Neutral, And Aggressive Stimuli

PDF

The influence of social intelligence and social teaching methods upon academic achievement

PDF

A Methodological Investigation Of Affect Response Bias

PDF

The Effects Of Prior Part-Experiences On Visual Form Perception In The Albino Rat

PDF

Development Of A Mid Primary Screening Battery For Schools Of A High Socioeconomic Community

PDF

An Attitude Survey Of High School Dropouts By Means Of The Semantic Differential Process

PDF

Investigation Of A Self-Report Method In The Study Of Underachievement Among Elementary School Children

PDF

An Experimental Study Of The Effect Of Anxiety And Stress On The Group Intelligence Test Performance Of Elementary School Children

PDF

An Investigation Of Two Language, Two Memory, And Two Perceptual Abilities In Retardates Of Mental Age Four

PDF

The Effect Of Subject Sophistication On Ratio And Discrimination Scales

PDF

Effects Of Associative And Category Cuing On Recall Of Items From Exhaustive And Nonexhaustive Lists

PDF

Development And Analysis Of Some Tactual Measures Of Intelligence For Adolescent And Adult Blind

PDF

Predicting Success In The Study Of Descriptive Linguistics

PDF

An Analysis Of The Selection Criteria For Assignment Of Students To Advanced Placement Classes In The Los Angeles Unified School District

PDF

An Investigation Of Overt Versus Covert Responding In Programmed Instruction

PDF

Reversal And Nonreversal Shift Performance Of Retardates Under Various Motivational And Stimulus Conditions

PDF

An Experimental Analysis Of The Relationship Between The Reliability Of Amultiple-Choice Examination And Various Test-Scoring Procedures

PDF

The Effects Of Delay In Knowledge Of Results On The Amount Learned In Teaching Machine Programs Of Differing Cue Content

PDF

An Assessment Of Effects Of A Large, County Wide, Public School Program Of Special Training Classes On Trainable Retarded Children

Asset Metadata

Creator Hopkins, Kenneth Dean (author)

Core Title Validity Concomitants Of Various Scoring Procedures Which Attenuate The Effects Of Response Sets And Chance

Degree Doctor of Philosophy

Degree Program Education

Publisher University of Southern California (original), University of Southern California. Libraries (digital)

Tag education, educational psychology,OAI-PMH Harvest

Language English

Contributor Digitized by ProQuest (provenance)

Advisor Metfessel, Newton S. (committee chair), Guilford, Joy P. (committee member), Lefever, David Welty (committee member), Meyers, Charles Edward (committee member), Michael, William B. (committee member)

Permanent Link (DOI) https://doi.org/10.25549/usctheses-c18-100895

Unique identifier UC11358125

Identifier 6102532.pdf (filename),usctheses-c18-100895 (legacy record id)

Legacy Identifier 6102532.pdf

Dmrecord 100895

Document Type Dissertation

Rights Hopkins, Kenneth Dean

Type texts

Source University of Southern California (contributing entity), University of Southern California Dissertations and Theses (collection)

Access Conditions The author retains rights to his/her dissertation, thesis or other graduate work according to U.S. copyright law. Electronic access is being provided by the USC Libraries in agreement with the au...

Repository Name University of Southern California Digital Library

Repository Location USC Digital Library, University of Southern California, University Park Campus, Los Angeles, California 90089, USA