University of Southern California Dissertations and Theses

Performance Evaluation Based On Multidimensional Job Behaviors

(USC Thesis Other)

Performance Evaluation Based On Multidimensional Job Behaviors

PDF

Download a page range

Download transcript

Copy asset link

Request this asset

Transcript (if available)

Content PERFORMANCE EVALUATION BASED ON
MULTIDIMENSIONAL JOB BEHAVIORS
by
Edward Ziegler Hane
A Dissertation Presented to the
FACULTY OF THE GRADUATE SCHOOL
UNIVERSITY OF SOUTHERN CALIFORNIA
In Partial Fulfillment of the
Requirements for the Degree
DOCTOR OF PHILOSOPHY
(Psychology)
August 1970
71-12,f'V
, Ecl'.:.3TC ''in y lp .r , i r ?7-
mrcri'Aic1 r"A:'7.'"r'" :i n : 'riTiAriricinAAA
Aniver^lty o f loutrl.ern C a l l 'o r v . i a , fh .:' ., 127''
r- yc'.iOloyy, in lu o tr la l
University Microfilms, A XEROX Com pany, Ann Arbor, Michigan
o p :'n a n : o v
E nr.r'; Zie-’ler Har.e
1971
THIS DISSERATION HAS BEEN MICROFILMED EXACTLY AS RECEIVED
UNIVERSITY O F SO UTHERN CALIFORNIA
TH E G RA D U A TE S C H O O L
U N IV ER SITY PA RK
LO S A N G E L E S . C A L IF O R N IA S 0 0 0 7
This dissertation, ’ written by
E O W A B D . . . Z . I E . G . L E B . . . H A N E . ........
under the direction of h.is.. Dissertation C om
mittee, and approved by all its members, has
been presented to and accepted by The G radu
ate School, in partial fulfillment of require
ments of the degree of
D O C T O R O F P H I L O S O P H Y
Date..... August.. . 1 9 . 7 . 0 . .
LM^SERTATION COMMITTEE
- C hairm an
....
CONTENTS
CHAPTER PAGE
I. INTRODUCTION ............................. 1
II. BACKGROUND............................... 3
Job Performance Measurement
Dimensions of Job Performance
Development of Multidimensional Job
Performance Measures
III. PURPOSE AND GENERAL APPROACH ........... 29
IV. METHOD.................................... 34
Experimental Group and Subjects
Experimental Determination of Job
Behavior Dimensions
Analysis of Performance Ratings
Factor Analysis
V. R E S U L T S .................................. 47
Multidimensional Scaling
Performance Ratings
Relationship of Job Behavior
Dimensions with Age and Test
Scores
VI. DISCUSSION............................... 66
Performance Ratings and Factors
Comparison of Results of Factor
Analysis and Multidimensional
Scaling
ii
CHAPTER PAGE
Relationship between Age
and Job Performance
Relationships between Test Scores,
Job Performance and Age
VII. SUMMARY AND CONCLUSIONS.................77
REFERENCES...........................................84
APPENDIX 87
LIST OF TABLES
TABLE PAGE
1. Stress Values.................... 48
2. Loadings and Correlations Used
for Interpreting Job Behavior
Dimensions.................... 50
3. Performance Rating Distributions . 53
4. Correlations Among Rating Scales
and with A g e ................. 55
5.' Eigenvalues and Proportions of
Variance Accounted for in Factor
Analyses of Performance Ratings
and A g e ........................ 56
6. Loadings on Rating Scales for Six
Factors, After Varimax Rotation . 57
7. Loadings on Rating Scales for Seven
Factors, after Varimax Rotation . 58
8. Loadings on Rating Scales and Age
for Eight Factors, after Varimax
Rotation...................... 59
9. Squares of Loadings on Rating Scales
for Six Factors (After Varimax
Rotation)...................... 61
10. Interpretation of Factors (Six
Factors, after Varimax Rotation) 62
11. Correlation of Age and Aptitude
Test Scores with Factor Scores
and Performance Ratings ......... 64
iv
TABLE
12. Correlation of Personal Inventory
Scores with Factor Scores and
Performance Rating Scales . . . .
PAGE
65
v
FIGURES
FIGURE PAGE
1. A Model for Test Validation and
Selection Research ................ 16
2. Stress V a l u e s ........................ 48
CHAPTER I
INTRODUCTION
This research study has been concerned with the
measurement of job performance. In particular, the
dimensional structure of job performance has been
investigated, for a group of research specialists in the
physical and life sciences. The purpose has been to
determine whether useful performance ratings can be
obtained on the basis of multidimensional job behaviors,
and whether these performance ratings are related to
age or scores on aptitude tests and personal
inventories.
The investigation of performance measurement
has far-reaching application, because of the importance
of performance in all applied areas of psychology.
Human performance is a critical variable: for the
utilization of human resources in industry; for the
study and solution of community social problems; or for
the treatment of a psychiatric patient. In all of
these areas of application, and in many others,
psychologists have as objectives:
(a) The description and measurement of
of performance,
1
2
(b) The prediction of performance from
personal characteristics or
antecedent conditions, and
(c) The manipulation of other variables
in order to modify performance.
Although this study has been carried out in the
context of industrial job performance, there are
important implications for other areas of interest.
The multidimensional model of performance which was
employed, and the experimental and analytical methods
used, would be useful in any application which demands
a meaningful evaluation of human performance.
CHAPTER II
BACKGROUND
Job Performance Measurement
The Need for Job Performance Measurement.--The
importance of developing effective job performance
measures is based on their large number of practical
uses. Job performance measurement occupies a central
role in personnel administration and research due to
its functional relationships with job requirements,
selection procedures, work methods, performance
appraisal and counseling, compensation, and the planning
and evaluation of training.
The use of job performance as a criterion for
the development of selection techniques is one of the
most important applications. Thorndike (1949) has
stressed the role of job performance in the design of
selection procedures, summarizing as follows: "The
most fundamental and most difficult problem in any
selection research program is to obtain satisfactory
criterion measures of performance on the job, against
which to validate selection procedures" (Thorndike,
1949, p. 119).
In an analogous way, performance measures serve
3
4
as an indicator in evaluating the effectiveness of
training programs (Mager, 1962). Similarly, performance
measures function as dependent variables in research for
the evaluation of various aspects of the working
situation. In addition, training personnel use
performance measures as a diagnostic tool for the
identification of training needs.
The application of human factors to the evalu
ation and design of work methods also relies on job
performance measurement. For example, in the development
of improved methods for electronics inspection, per
formance measurement has been used both to identify
problems and evaluate proposed new methods (Harris and
Chaney, 1959).
Evaluations of performance are used by management
for several administrative purposes in the handling of
personnel. Typical uses are for promotions and
transfers, layoffs and discharges, and compensation.
A closely related application is the use of performance
measures as the basis for counseling of employees by
supervisors (Tiffin and McCormick, 1965).
5
Typical Methods.--The need for measuring job
performance has long been recognized in personnel
administration and research. Thorndike (1949) and
Guion (1965) have comprehensively presented the nature
of the problem and described typical procedures for its
solution. In order to be useful, job performance
measures should be relevant, reliable, discriminating,
and practical. In attempts to fulfill these require
ments, several general types of job performance measures
have been developed, both objective and subjective.
The principal types of objective measures have
been production rates, error rates, work samples, and
job knowledge tests. Subjective types of performance
measure include a variety of graphic rating scales,
ranking schemes, check lists, and comparisons among
individuals. For some purposes, measurements such as
absence rates, tenure, and accident rates are considered
as job performance, but for the purposes of this study,
job performance is considered more narrowly as an
indicator of the quantity or quality of work.
The objective kinds of performance measurement
are often unavailable or hopelessly contaminated by
6
extraneous factors. Much production output is paced
by variables which are a function of the total production
system. Even where individual production rates or
error measurements are feasible, they may be more
indicative of relative difficulty of work, availability
of materials, etc., than they are of individual
performance.
Work samples and job knowledge tests are
objective performance measures which avoid some of the
difficulties mentioned above. However, these methods
introduce an artificial environment, which may have
different motivational value for employees than normal
production. Consequently, their use as performance
measures is satisfactory only in certain situations.
For these reasons, subjective evaluations of
employee performance, such as supervisors’ ratings,
are usually required, even in production jobs. In
administrative and professional jobs, where there is
often no physical product, it is even more important
that performance measurement be accomplished as a matter
of judgment on the part of a rater. Ratings have the
advantage of utilizing the rater's mental capabilities
7
to properly weigh the multitude of factors involved in
evaluating job performance.
It is not surprising that a number of different
approaches have been attempted in the effort to obtain
performance ratings which have the characteristics of
relevance and reliability, identified above as require
ments of performance measures. Guion (1965) has des
cribed the principal variations. The graphic rating
scale is the most common format, utilizing a variety of
verbal definitions and numbers of scale divisions, and
sometimes employing forced distributions. In cases
where a set of scales has been used, each one for a
particular trait, a number of different kinds of weighted
combinations have been tried. Several procedures for
ranking employees have also been used, as well as
methods involving paired comparisons among employees.
Check lists of elements of job performance have been
devised, and sometimes arranged in a forced choice
format.
Problems in Job Performance Measurement.--Con
sidering the importance of the need and the variety of
8
approaches attempted, job performance measurement tech
niques have fallen far short of the degree of success
required. The principal weaknesses are pointed out
by Guion (1965). Rater bias, lack of reliability,
leniency, failure to discriminate among rating scales
(halo effect), and failure to disciminate among
individuals (central tendency) are cited as the most
common defects of job performance ratings.
The failure to devote sufficient attention to
the development of performance measures, and the failure
to use empirical methods, have both been pointed out as
causes of inadequacy in performance ratings. In
selection research, for example, more attention has been
given to the development of predictors than to the
understanding and evaluating of criteria (Ghiselli,
1956). The lack of empiricism can be seen in the
typical sequence of events followed in finding a per
formance criterion for selection research. The
investigator typically goes through a series of judgments
culminating in a decision to use someone else’s judgment
as the criterion (Guion, 1961).
Yet another cause of poor performance ratings,
and the problem of central interest in this study, is
the failure to recognize the multidimensional nature
9
of job performance. This problem will be examined in
the following section.
Dimensions of Job Performance
Unidimensional Approach and Its Weaknesses.--
The assumption that job performance is unidimensional
in nature has been implicit in most conventional per
formance measurement techniques. The concept is
apparent in Thorndike's discussion (1949) which em
phasizes the importance of the "ultimate" criterion,
which is "the complete final goal of a particular type
of selection or training" (Thorndike, 1949, p. 121).
Thorndike does distinguish a special kind of measurement,
the evaluation of performance in a specific defined
task. However, the operational methods of this kind of
evaluation consist of work samples and job knowledge
tests, rather than ratings on actual performance.
The concept of a single, ultimate criterion of
job performances has dictated a particular model for the
prediction of job performance, the multiple regression
model. Various predictions are combined with the most
suitable weights, in order to predict as large a
proportion of job performance variance as possible.
10
In utilizing this model, it has proven very difficult
to obtain selection batteries which predict more than
25 per cent of the variance in job performance measures.
In recent years, there has been considerable
criticism of the idea of a single, ultimate job per
formance measure, and of the corresponding model for
measurement and prediction.
Ghiselli (1956) was one of the first to point
out the "dimensional problems of criteria." The
unidimensional model, he said, requires the assumption
of the existence of a general factor in job performance,
analogous to a general factor in intelligence. However,
there is no evidence that such a general factor can be
extracted by factor analysis. He states further:
It would appear, therefore, that the criterion
dimensionality of the individual should be in
vestigated in the same way that Stephenson, Cattell,
and others investigate the personality dimensionality
of the individual. It is quite possible that workers
assigned to the same job perform quite differently
in a qualitative as well as in a quantitative
sense. (Ghiselli, 1956, pp. 3-4)
Korman (1968) reached a similar conclusion, in
conducting an evaluation of the research literature
relative to the prediction of leadership behavior. He
recommended adoption of the term "behavior" in place of
11
"criterion,” and further stated:
Basically, what should be predicted is not the
unknown but the known; i.e., psychologically-defined
behaviors and activities which we call leadership
behaviors in the formal organization. . . . If our
hypothesis is correct, such an approach would
raise the level of psychometric prediction to a
more satisfactory level than that found in this
review. (Korman, 1968, p. 317)
Rundquist (1969), in considering possible
reasons that the validity coefficients of predictor
instruments have been limited to approximately .50,
pointed out that different leadership styles may result
in the same output. He cited a study in which
"managers differed markedly, one being authoritarian,
one permissive, and one recessive. Yet there was no
difference in the department output. We cannot conclude
that differing styles of leadership do not differ in
their effects--only that each can prove equally product
ive under certain conditions" (Rundquist, 1969, p. 113).
He suggested that the improved determination of
predictor-criterion relationships requires an approach
which is psychological rather than statistical.
It involves the discovery of the mediating processes
used by all individuals in an entire sample of
applicants, in learning to do the job for which
success is to be predicted. . . . With this
information at hand, tests of these antecedent con
ditions can be constructed. (Rundquist, 1969, p. Ill)
12
Guion (1961) also critized the concept of a
single "ultimate" criterion stating that "there are in
may personnel situations dimensions of job performance
and of performance consequences that are quite
independent of each other" (Guion, 1961, p. 149).
Dunnette (1963) has convincingly argued against
the use of a single overall performance criterion. He
has criticized this concept as being oversimplified in
view of the complexities of most job behavior and the
available methods for observing job behavior.
Specifically, he has written: "Junk the criterion!
Let us cease searching for single or composite measures
of job success and proceed to undertake research which
accepts the world of success dimensionality as it
really exists" (Dunnette, 1963, p. 252).
The common contention of all of these points of
view is that job performance is too complex a concept to
be understood in terms of a single general dimension.
The satisfactory description and measurement of job
performance requires a multidimensional structure.
Furthermore, the most effective predictive relationships
can be obtained between predictor instruments and
independent measures of specific behavior, rather than
between predictor instruments and overall performance
criteria.
Multidimensional Approach.--If performance
varies along several relatively independent dimensions,
then it is necessary to determine a useful basis for
breaking down overall performance. Possibilities are
suggested by three different ways in which employees
have been described: according to personality traits,
job functions, and job behaviors.
Of these alternatives, the use of personality
traits to identify performance dimensions has the
least to offer. In practice one of its weaknesses has
been that identification of the traits has been
accomplished in a subjective, arbitrary manner. Further
more, there are in many cases mathematical procedures
for combining such ratings into a single overall
rating, although there is no evidence that such pro
cedures are justified by empirical relationships.
However, apart from these practical problems, it is
unreasonable to rate job performance, which consists of
dynamic behaviors, in terms of static personality traits
The use of such rating scales has generally
proved unsuccessful, in the sense that raters do not
actually discriminate among the traits. In one
example, ratings were made on twelve scales, each
14-
identified by word description of traits related to
job performance (e.g., versatility, productivity,
judgment). One of the scales was called "overall
performance." The ratings were correlated, the
correlations factor analyzed, and the solution rotated
to a psychologically meaningful solution. This pro
cedure resulted in the identification of three factors
which accounted for virtually all the variance in the
twelve scales. These three factors were interpreted as
ability, quality, and health (Tiffin and McCormick,
1965).
The breakdown of jobs into their component job
functions is the basis of the concept of synthetic
validity (Balma, 1959; Ghiselli, 1959). Synthetic
validity has been defined as an inferred validity, based
on a breakdown of jobs into their functional elements,
determination of predictive validity for each element,
and a combination of elemental validities into a whole.
This approach recognizes the fact that employees in a
given job classification may in fact be performing
different functions.
15
It is necessary, however, to make a further
distinction among employees concerning their job per
formance. Two employees with equal records of success
in job performance and identical job functions may have
accomplished their results by different patterns of
specific behavior (Dunnette, 1966). It would seem
reasonable, therefore, to validate predictors using a
performance criterion which recognizes the differences
in behavior.
Job performance needs to be separated into a set
of independent performance ratings, on the basis of
observed job behaviors, rather than job functions or
arbitrary personality traits.
Dunnette (1966) formulated this position in a
multidimensional model for performance measurement and
selection research. Dunnette's model is depicted in
Figure 1. The ultimate work objectives are expressed
as consequences, which must have a relationship to
organizational goals. Job performance would be measured
in terms of independent and mediating behaviors. The
presence of separate columns for predictors, individuals,
and situations represents the possibility of interaction
among all these variables. The arrows connecting sets
3
3
3
3
rT
rT
3
vO
O'
O'
T3
o
U l
s o
Pd
c/5
m
>
so
n
G
3
3!
O
Cd
Pd
r
^d
o
so
H
Pd
C /5
H
<
r
t —•
0
>
H
i —t
1
23
o
C /5
PI
r1
Pd
o
H
i —i
§
rq
n
a
G
SO
Pd
u>
N5
Pd C3
u>
Predictors (Individual
Difference Measures)
Individuals
____
C/5 C /5 C/5 C/5 C/5 C /5
05 U l -P~ UJ N5 I — 1
O
u>
Job Behaviors
Situations
Consequences(Related
to Organizational
Goals)
3
3
o
3
09
3
r r
3*
3
<
3
3
h“
3
CT
t —>
3
Cfl
r r
3*
3
3
o
a
3
o
i-h
<
3
P
H*
3
cr
I —*
3
C f l
Cfl
C
09
09
3
Cfl
r r
r r
3*
3
O
o
3
T3
I —1
3
X
T>
O
r r
3
3
rr
H-
3
H
3
i —*
3
rT
H*
O
3
cn
3*
H-
Td
c n
O'
17
This model serves as a point of departure for the
present study, the basic purpose of which is to evaluate
methods for measuring the mediating job behaviors.
Guion (1961) presented the outline of a planned
sequence of events designed to define and measure such
mediating job behaviors. The steps of Guion*s plan are
as follows:
1. Analyze the job and/or the organizational needs
by new, yet-to-be-developed techniques.
2. Develop measures of actual behavior relative
to the behavior expected, as identified in job
and need analysis. These measures are to
supplement measures of the consequences of
work--the so-called objective criteria
commonly tried at present.
3. Identify the criterion dimensions underlying
such measures by factor analysis or cluster
analysis or pattern analysis.
4. Develop reliable measures, each with high
construct validity, of the elements so
identified.
5. For each independent variable (predictor),
determine the predictive validity for each one
of the foregoing criterion measures, taking
them one at a time. (Guion, 1961, p. 148)
The successful development of multidimensional
performance rating techniques, consistent with
Dunnette*s model, would have practical value and provide
18
a useful research tool. The principal administrative
applications would be for more effective employer .
appraisal and counseling plans, placement and promotion,
and determination of job requirements. In personnel
research, performance dimensions could be used as
criteria for the validation of selection techniques,
and as dependent variables in determining the effect of
various situational and personal variables. Another
important research objective would be the relating of
performance dimensions to critical organizational goals.
Development of Multidimensional
Job Performance Measures
The sequence of activities in Guion's plan
requires consideration of appropriate research techniques.
First, empirical methods should be employed to overcome
Guion's objection that most criterion development is an
entirely subjective process. The development of
behavior measures necessitates the construction of
experimental measurement devices. The alternative pro
cedures which are available include ratings of observed
employee behaviors, ratings of the desirability or
importance of different kinds of behaviors, and ratings
19
of the similarities among a set of job behaviors. For
the identification of underlying dimensions by
statistical analysis, there are also alternative pro
cedures. The most likely statistical procedures are
multidimensional scaling analysis, multiple regression
analysis, and factor analysis. Finally, the results of
these experimental and analytical procedures must then
be translated into practical working tools.
Several investigators have carried out studies
which approximate Guion's sequence, and which involve to
some degree the experimental and analytical procedures
described above. It will be useful at this point to
review the methods and results of these studies.
Kirchner and Dunnette (1957) determined a set of
critical factors in job success for salesmen, utilizing
an empirical approach to the analysis of job requirements,
but relying on their own judgment to analyze the
empirical results into underlying dimensions. They
obtained 135 critical incident reports from 85 sales
managers, illustrating effective and ineffective sales
behavior. From these responses, they rationally
derived a set of fifteen factors which they consider to
20
be critical to sales success. The factors were sub
sequently applied to the development of a behavior
rating sheet for the use of managers in evaluating
salesmen.
Hemphill (1959) identified job performance factors
for executives, using a research design which reversed
Kirchner and Dunnette's strategy. The initial
determination of potential job behaviors was judgmental
rather than empirical, but the analysis of these into
underlying dimensions was accomplished with factor
analysis. He identified, initially, 575 job elements
on the basis of an extensive literature search. A group
of 93 executives responded to this list of job elements,
indicating the importance of each job element for
accomplishing their job functions. By correlating and
factor analyzing the ratings, Hemphill obtained a set of
ten independent dimensions of executive function. He
later developed a shorter listing of 191 job elements
for further research, reported to be equivalent to the
575 original elements.
Peres (1962) conducted a study to identify the per
formance dimensions of supervisory positions, using an
21
empirical approach for the initial determination of job
requirements, and an analytical approach for identifying
the underlying dimensions. A comprehensive listing of
303 job elements was obtained from a set of 414 essays
written by general supervisors, each describing the best
general supervisor they had known. The 303 job elements
were then used by the general supervisors for rating the
best and the worst supervisors each had known. The
ratings were analyzed by a factor analysis, the results
of which were orthogonally rotated using a simple
structure criterion. Peres identified seven broadly
defined factors, interpreted as follows:
I. Establishment of Work Climate
II. Management Ethics
III. Practice of Self and Subordinate
Development
IV. Personal Maturity and Sensitivity
V. Knowledge and Execution of Corporate
Policies and Procedures
VI. Technical Job Knowledge
VII. General Bias or Halo
22
The latter factor accounted for large inter
correlations among the job elements, and was interpreted
as a characteristic of the raters, not of the rated
employees. Thus, it was considered an artifact of the
experimental procedure.
In another application of factor analysis to
the problem, ten dimensions of supervisory behaviors
were identified for engineering supervisors (Ritti,
1964). The initial job elements consisted of a 76-item
supervisory behavior inventory, and the experimental
data were obtained by having 139 engineers rate their
supervisors on frequency of occurrence of each of the 76
behavior items. An important aspect of Ritti's study
was that the ratings were standardized by rater and by
variable, to reduce discrepancies in scaling due to
individual variation.
Multidimensional scaling analysis has been used
for similar purposes. Schultz and Siegel (1964)
employed this technique to identify four basic job
dimensions for electronics technicians. Their interest
was in job requirement dimensions, rather than job
performance dimensions, but the research strategy was
23
similar. Initially, eighteen tasks were empirically
identified as constituting the electronics technician
job. Supervisory personnel rated the similarity of each
pair of tasks, and the similarity ratings were analyzed
by multidimensional scaling. Four independent job
requirement dimensions were identified.
In a later study, Siegel and Pfeiffer (1965)
repeated the procedure for the same job using electronics
technicians themselves as similarity judges, instead of
supervisory personnel. This analysis also resulted in
four job requirement dimensions, interpreted as being
congruent with the dimensions obtained from the
supervisory personnel.
In another multidimensional scaling application,
Brown (1967) identified five independent dimensions of
interpersonal relation among the incumbents of a govern
ment management analyst position. Each pair of a set
of eighteen behavior statements was rated as to
similarity by six employees in the position, and the
underlying dimensions determined by multidimensional
scaling analysis, as in the previous example.
24
Generally speaking, the studies which have sought
to determine dimensions of job performance have produced
rather broadly defined activities which can better be
described as functions or objectives, rather than
behaviors. The lists of job dimensions are characterized
by items such as "technical job knowledge," and "super
vision of work."
Relationship of Job Dimensions to Test Results
and Personal Variables.--The potential value of relating
job performance dimensions to test results has been
covered in the above discussion. One of the principal
uses of multidimensional performance ratings would be to
serve as criteria for the validation of selection tests.
It is reasonable to expect higher validities with the
use of fundamental behavior dimensions as criteria, in
place of composite overall performance ratings.
Another useful research application would be in
relating job performance dimensions to certain personal
variables such as age, race, sex, and education. The
relationship with the age variable is of particular
interest. In many validation studies, it has been found
25
that age accounts for part of the relationship between
selection instruments and job performance, and its
effect needs to be statistically cancelled out. If
there is indeed a significant relationship between age
and particular job behaviors, an improved knowledge of
the relationship would have practical implications both
for employee placement and for job design. Several
studies have been made in an attempt to determine the
effects of age on overall performance, or on specific job
behavior. United States and Canadian Department of
Labor studies reported by Birren (1964) show a tendency
for little change in work output for women office workers
up to retirement, and some tendency for output to decline
among industrial workers after the age of fifty-five,
although individual differences are more prominent.
Anderson (1959) reports a study by King in which a slow
decline in output was found with women sewing machine
operators after the age of thirty-five, and a study by
Clay in which small declines in output were observed
among printing compositors, but not readers, after the
age of fifty. McFarland and O'Doherty (1959) report a
study by Palmer and Brownell in which there was no
26
relation between age and output in six companies which
were studied. In summary, the relationships discovered
between age and overall performance have been quite
small, especially when compared with individual
differences.
However, a number of sources indicate the
possibility of age-related changes in patterns of job
behavior. Griew (1964) argues for the necessity of break
ing the concept of "work capacity" down into com
ponents, since an older worker "may have an overall
capacity typical of a worker much younger than himself,
but it is more than likely that his ability for some
specific type of work will differ widely from that of
his younger colleague" (Griew, 1964, p. 25).
Griew also discusses the relation of speed in decision
making to age, and the reduced capacity for short-term
memory as age increases.
Kuhlen (1959) discusses several other indications
of possible age-related behavior changes. The results
of a study by Kuhlen and Johnson show a sharp reversal
of working goals for men at about age fifty, from
promotion and education to retirement. It is also
27
pointed out that changing age roles, or group expect
ations, may bring about a change in behavior with age.
Studies of personality are cited which show a trend with
age toward conservatism, dogmatism, and rigidity in
behavior. All of these attitudinal and personality
changes suggest the possibility that behavior patterns
change with age.
McFarland and O'Doherty (1959) suggest, as a
practical application, that jobs and individuals be
matched on the basis of job demands and behavior
potential. They have suggested that the reclassification
of older workers be based on the capacities of the
individual in relation to the critical psychological
requirements of the task.
It would appear likely, from this evidence, that
the effect of age on job performance could best be
understood in terms of changing behavior patterns,
rather than decrement in overall performance. In
particular, older workers could be expected to be more
deliberative, more conservative, and less flexible than
younger workers, and to exhibit sounder judgment but
28
less speed in their decision processes. With a better
understanding of age-related job behavior patterns,
improvements could be made in job placement and work
assignments.
These points of view support the argument that
job performance can best be measured in a multi
dimensional structure. If older workers differ from
younger workers in job behavior patterns, without
necessarily differing in overall job performance, it
follows that there must be some dimensions of job
behavior, along which individuals vary independently
of their overall job performance. The model proposed
by Dunnette (1966), and depicted in Figure 1, affords a
conceptual frame of reference for exploring age-related
job behavior variables by using a research design
incorporating the kinds of operations suggested in
Guion’s (1961) outline, and using subjects with an
appropriate age range within an occupation.
CHAPTER III
PURPOSE AND GENERAL APPROACH
The primary purpose of the present study was to
evaluate a multidimensional set of job performance rat
ing scales, developed on the basis of empirically
determined job behaviors. Specifically, it was intended
to analyze performance ratings obtained by the use of
these scales, in order to determine whether they dis
criminate well among individuals, and whether they are
independent of one another and of overall performance.
Additional purposes of the study were to test the
hypotheses that there are significant relationships
between independent job behavior dimensions and age,
and between behavior dimensions and selected personnel
tests.
This study is an important extension of the
research designs described above, which have not involved
the construction of performance rating instruments
based on the underlying dimensions. This extension is
useful in providing a verification of the validity of
the experimentally determined dimensions, and in
making it possible to investigate their relationships
with other variables.
29
30
The focus of interest in the present study is
on the elemental human behaviors associated with job
performance. This feature is in contrast to the
emphasis of some of the studies referred to above, which
were designed to reveal broader functional categories
(e.g., executive roles). It is felt that the use of
elemental behavior variables holds the greatest promise
for discovering relationships with other variables, and
for practical applications. It is these behaviors which
constitute the basic variables of performance measure
ment, and they correspond to the mediating behaviors as
expressed in Dunnette's (1966) model for performance
measurement and selection.
Another feature of this study is that in the
initial stage of empirical observation of job behaviors,
in-depth information is obtained from a limited number
of observers within a single organization. Most studies
have sought a limited amount of information from each
of a large number of observers. The approach utilized
in this study is consistent with the purpose of
explaining job performance in terms of fundamental human
31
behavior, rather than broad functions.
The reports of actual job behaviors were obtain
ed in interviews with supervisory personnel. The
investigator identified the behaviors in these reports,
eliminated repetitions, and expressed them in a list of
short action statements. The analysis of behaviors was
accomplished by multidimensional scaling, the same super
visory personnel providing similarity ratings for the
list of job behavior statements.
A set of underlying job behavior dimensions was
derived by interpreting the results of the multi
dimensional scaling analysis. Performance rating scales
were then constructed, corresponding to the underlying
behavior dimensions. The rating scales were used for
supervisory evaluation of their subordinates' job
performance. These ratings provided the data for
evaluation of the performance dimensions, and for in
vestigating the relationships of performance with age
and test scores. These procedures correspond rather
closely to the experimental sequence proposed by Guion
(see page 17), except that there was no attempt to
analyze the job requirements or organizational needs.
32
By these procedures, it was intended to test the follow
ing hypotheses:
(1) There is a set of job behavior dimensions
for the occupational group used in this
study, relatively independent of each
other and of overall performance.
(2) The methods used in this study will be
successful in identifying these behavior
dimensions.
(3) Job performance ratings based on behavior
dimensions will discriminate effectively
among individuals, and their distribution
will be approximately normal, about a mean
near the midpoint of each rating scale.
(4) Age will be positively correlated with
behavior dimensions which imply judgment,
deliberation, and conservatism, and
negatively correlated with behavior
dimensions which imply speed and
flexibility.
33
(5) It will be possible to predict behavioral
performance ratings with test scores more
accurately than overall performance can be
predicted with test scores.
This study was carried out within a scientific
group which is part of the Engineering Department of an
aerospace manufacturing firm. The subjects rated were
research specialists, many with advanced degrees, and
with complex job requirements. The supervisory personnel
who provided ratings were department managers who are
the direct supervisors of the research specialists.
CHAPTER IV
METHOD
Experimental Group and Subjects
All of the research procedures were conducted
within a scientific research department in the Engineer
ing organization of an aerospace manufacturing company.
The department is subdivided into four major groups,
with titles and principal functions as follows:
Group A: Aviation Medicine--Scientific research
on various physiological problems.
Group B: Physical Sciences--Scientific research
on physical, chemical, and meteorolo
gical problems.
Group C: Human Factors Engineering--Human
factors analysis in support of aero
space design.
Group D: Life Sciences Research--Scientific
research on problems in experimental
psychology.
Each of these groups is headed by a manager who
directly supervises the research scientists in the group.
The total population of research scientists in the
department was approximately fifty at the beginning of
34
35
the study. However, there has been considerable
turnover and, by using recent employees as well as
current ones, a total of sixty-eight subjects were used
for performance ratings. The division of subjects
among the four groups was as follows: Group A, eleven
subjects; Group B, eleven subjects; Group C, twenty-two
subjects; and Group D, twenty-four subjects.
The subjects were mostly of a very high level
of educational attainment in the various scientific
fields. The breakdown by degree level was as follows:
Doctor's 14 (21%)
Master's 22 (32%)
Bachelor's 20 (29%)
No degree 12 (18%)
There were three females among the sixty-eight
subjects. There was a rather broad age range, for an
industrial population. The mean age was forty-one, and
the standard deviation was ten years.
Thirty-six of the subjects (53 per cent) had been
employed in the department at least two years prior to
the time of the rating. The others had been employed in
the department for approximately one year or longer.
Experimental Determination of Job Behavior Dimensions
Collection of Observed Job Behaviors.--In this
first step, the investigator interviewed each manager
individually. After explaining the purpose of the
interview, the investigator asked each manager a standard
set of questions to elicit reports of favorable,
unfavorable, or neutral job behavior. The managers'
observations were recorded and reviewed later by the
investigator to eliminate redundancies and condense
wording. The result was a listing of thirty-one elements
of job behavior, expressed as brief action phrases. The
list of thirty-one job behaviors is included as
Appendix A.
Similarity Ratings.--The multidimensional scal
ing analysis which was used to determine job behavior
dimensions requires experimental data for the
estimation of the psychological distance between each
reference point. These data were obtained in the form
37
of managers' ratings of the similarity of each pair of
job elements. This is an application of the experi
mental method of paired comparisons for psychological
scaling (Torgerson, 1958).
These similarity ratings were obtained by the
use of a rating booklet in which each of the possible
pairs of job elements appeared once. The nine-interval
similarity rating scale is shown on the rating instruc
tion sheet (see Appendix B) which was included in the
rating booklet. A list of all the job elements was
included after the instruction sheet. The remainder of
the booklet contained the paired behaviors. On each
page, one of the thirty-one behaviors was paired with
fifteen other behaviors. First, a random arrangement of
pairs of behaviors was determined for each page. The
pages were then arranged in a different random sequence
in each rater's rating booklet. The first rating page
of each booklet was for an artificial job behavior item,
not used in the analysis. The artificial behavior item
was: "Considers all sides of a question." This was done
to allow the raters to stabilize their scale values on
standard material, prior to making actual ratings.
38
In addition to the managers, the research depart
ment's deputy director and assistant to the director
completed similarity ratings, for a total of six raters.
Multidimensional Scaling Analysis.--The similar
ity ratings were analyzed by a computer program using
Kruskal's (1964, 1968) procedure for multidimensional
scaling analysis. This is a nonmetric analysis in
which the only assumption made concerning the estimated
psychological distances (similarities) is that they are
correctly ordered. In this analysis, the minimum number
of dimensions is determined for a configuration of
points whose interpoint distances correspond to the
experimental distances with an acceptable goodness of
fit, with the constraint that the distances in the
solution are monotonically related to the experimental
distances. Each of these dimensions is defined by the
loadings on the original behavior items. The con
figuration can be rotated’to obtain the most meaningful
set of dimensions. The means of the similarity ratings
for all six raters were used as the experimental dis
tances for the multidimensional scaling analysis, and
the program was run for two, three, four, five, six,
39
seven, and eight dimensions. It is necessary to
evaluate each solution in terms of its relative goodness
of fit. Goodness of fit is measured in terms of stress,
computed according to Kruskal's Formula 1:
,. /
V El2
where d represents experimental distances, and $
represents points on the scale of distances, correspond
ing to each d value, such that the regression of on
d is monotonic, and a real solution exists in n
dimensions for the $ values. Summation occurs over
all interpoint distances in the configuration, where
each point corresponds to one of the thirty-one original
job behaviors. For each number of dimensions used, the
solution is that configuration which minimizes stress.
Stress is calculated and reported for each solution, in
order to measure its goodness of fit and evaluate its
adequacy.
Rotation and Interpretation of Dimensions.--
After the appropriate number of dimensions and the
corresponding configuration of points were determined
by the procedures described above, the configuration was
40
orthogonally rotated by the varimax method, using
Biomedical Computer Program BMD 03M (Dixon, 1968). The
dimensions were interpreted in terms of their loadings
on each of the original thirty-one job behaviors.
It should be pointed out that the origin of the
dimensions in this solution is placed at an arbitrary
location, and the same configuration with a different
point of origin would be an equally good mathematical
solution. This fact has implications for the sub
sequent use of the dimensions. For example, in the
resulting solution, each dimension has a negative and a
positive end, which implies that the job behaviors
representing opposite ends of the dimension are opposed
in meaning. The more correct psychological solution,
however, may be one in which the origin is near one end
of the dimension, in which case the apparently opposing
job behaviors would actually be uncorrelated or
positively correlated. The use of the behavior
dimensions as performance rating instruments should make
it possible to resolve the correct position of the
origin.
41
Analysis of Performance Ratings
Construction of Rating Scales.--A set of rating
scales was constructed, one scale corresponding to each
of the behavior dimensions identified in the multidimen
sional scaling analysis. The wording by which the rat
ing scales were defined was carefully chosen to reflect
the meaning conveyed in the original job elements.
Each scale consisted of nine categories representing a
continuum from least degree to maximum degree of the
defined behavior dimension.
The six dimensions resulting from the scaling
analysis were all bipolar in nature, with a positive and
negative end. For the reasons stated in the last section
it cannot be assumed that the opposite ends of each
dimension are actually opposed to one another. Further
more, in most cases, the behaviors which defined
opposite ends of dimensions did not logically appear to
be opposed. Therefore, for performance ratings and
factor analysis, the six dimensions were split into
twelve separate rating scales. It was felt that if
opposite ends of a dimension were actually opposed, this
would be obvious in the analysis of ratings.
42
In addition to these twelve scales, one addition
al scale was included to provide a measure of overall
performance on the job. The total of thirteen rating
scales were assembled into a booklet, with one rating
scale defined on each page, and provision for entering
names of subjects and rating for each. In completing
ratings, the managers were instructed to complete each
page, rating all subjects on that scale, before pro
ceeding to the next page. This is a common practice in
obtaining performance ratings, to minimize the possi
bility of halo effect.
Rating instructions were provided on the first
page of the rating booklet. The rating instruction
sheet is shown in Appendix C, and one of the rating
scale sheets is shown in Appendix D. A listing of the
complete set of rating scale definitions is included as
Appendix E.
Rating Distributions and Correlations.--Frequency
distributions and median ratings were calculated for
the ratings obtained on each scale, in order to evaluate
the scales in terms of discrimination among subjects,
normality of distribution, and leniency. Prior to com
puting correlations, the rating distributions were
43
examined for each rater and for eacn of the scales. It
was found that the form of the distributions varied
considerably among raters and among rating scales.
Therefore, the raw ratings were converted into standard
ized ratings for each rater and for each rating scale.
This is the procedure used by Ritti (1964), and was
necessary because the discrepancy among distributions
would have distorted the correlation coefficients.
Product-moment correlation coefficients were
calculated between the ratings obtained on each of the
thirteen rating scales. The sizes of the correlation
coefficients were used to evaluate the independence of
the scales from each other. In particular, each scale
was compared with its counterpart which represented the
opposite end of the same dimension, and each of the
behavioral rating scales was compared with the overall
performance rating scale.
Factor Analysis
The correlations among rating scales were
analyzed by a principal components factor analysis, and
a varimax orthogonal rotation, using Biomedical Computer
Program BMD 03M (Dixon, 1968). The results of the
44
factor analysis were used to examine the factor com
position of the performance ratings, and to compare it
with the dimensional structure obtained from the scaling
analysis. Factor scores were calculated for all subjects
on each of the factors. The factor analysis was re
peated with age used as an additional variable, along
with the thirteen rating scales, in order to determine
the extent to which age of subject accounts for variance
in the performance ratings.
Relationships of Job Performance with Age and
Test Score Data.--In addition to the performance rating
data, ages and scores on selected personnel tests were
obtained for use in this study. The age data were
obtained in order to determine whether age accounts for a
significant proportion of the variance in performance
ratings, which would indicate behavior patterns correlat
ed with age. Test scores were obtained to provide an
indication of potential predictive validity for tests,
using performance dimensions as criteria. Only a
limited number of the subjects were available for test
ing, and these only for a limited period of time; there
fore the results using test scores must be considered
45
only as rough indicators at best.
The tests selected for use included four short
aptitude tests and two personal inventory forms. The
following aptitude tests from the Employee Aptitude
Survey (Ruch, 1963) were used:
Verbal Comprehension
Verbal Reasoning
Numerical Reasoning
Symbolic Reasoning
The personal inventories used were the Gordon
Personal Profile and the Gordon Personal Inventory
(Gordon, 1963a, 1963b). These measures provide four
scales each on personality dimensions, as follows:
Gordon Personal Profile
Ascendancy
Responsibility
Emotional Stability
Sociability
Gordon Personal Inventory
Cautiousness
Original Thinking
Personal Relations
Vigor
46
All of these measures were selected because of
their use for industrial selection, and because of the
availability of norms and validation data.
The total number of subjects available for test
ing was thirty. Of these, two subjects did not take
the personal inventories. For purpose of analysis, the
means of all other subjects were used in lieu of actual
scores for these subjects.
Correlations of Age and Test Scores with Rating
Scales and Factor Scores.--Product-moment correlation
coefficients were calculated between age, test scores,
performance ratings on each of the thirteen rating
scales, and the factor scores obtained from the factor
analysis of performance ratings (the factor analysis
used was the one which did not include age as a
variable). These correlation coefficients were examined
to discover any indications of significant relation
ships. The correlations between age and test scores
were also examined, as a matter of interest.
CHAPTER V
RESULTS
Multidimensional Scaling
Stress Values.--The stress values from the
multidimensional scaling solutions are shown in Table 1.
In order to determine whether goodness of fit was
better than that expected by chance, the stress values
were compared with stress values obtained by Stenson
and Knoll (1969), using random numbers as data points.
The random number stress values are also shown in
Table 1, for comparison, and both sets of values are
graphically displayed in Figure 2.
A comparison of the curves shows that the stress
values for the similarity ratings data are substantially
lower than those for random numbers. (Stenson and Knoll
reported that the stress values obtained for random
numbers can be expected to vary not more than .01.)
Beyond six dimensions, however, the curve for similarity
ratings is relatively flat and approaches the random
numbers curve. It appears that the reduction in stress
value with more than six dimensions is due only to ran
dom effects. Therefore, six dimensions is considered to
be a satisfactory solution.
47
48
TABLE 1
STRESS VALUES
No. of Stress Values
Dimensions
Similarity
Ratingsa
Random
Numbers*5
2 .25 .35
3 .16 .25
4 .12 .19
5 .10 .16
6 .09 .13
7 .08 .11
8 .06 .09
9
- -
.08
10 .07
a31 variables
^30 variables. From Stenson and Knoll (1969)
.40
Random Numbers
Stress
.20
(Kruskal's
Formula 1)
Similarity
Ratings
Dimensions
FIGURE 2
STRESS VALUES
49
Dimensions and Loadings.--When it was determined
that the six-dimension solution of the scaling analysis
was satisfactory, the dimensional structure was rotated
orthogonally to simple structure, using a varimax
rotation. The resulting loadings of dimensions on job
behaviors (see Appendix F) were examined to interpret
the dimensions.
In order to make this interpretation, a list was
made of loadings with absolute values greater than .50.
Correlations were computed between job behaviors and
dimensions, corresponding to each of these loadings,
because the size of the original loadings is partly a
function of the total length of the dimension, and there
was considerable variation in the lengths of the dimen
sions. Some job b-jhaviors were eliminated from the list
because of correlations less than .70, and others were
eliminated because they had appeared on the list for
more than one dimension. The remaining job behaviors
are the ones used to interpret the dimensions. These
job behaviors, loadings, and correlations for each of
the six dimensions are shown in Table 2. Twenty-three
of the thirty-one job behaviors were used for inter
pretation of the dimensions.
TABLE 2
LOADINGS AND CORRELATIONS USED FOR INTERPRETING
JOB BEHAVIOR DIMENSIONS
Loading Correlation
FIRST DIMENSION "
.90 .92 Conforms to project schedules and budgets.
.77 .96 Plans and schedules work systematically.
-1.26 - .91 Spends too much time on relatively unimportant
work.
-1.20 - .86 Is frequently absent or unavailable when needed.
-1.19 - .84 Spends a great deal of time on non-work subjects.
SECOND DIMENSION
.59 .64 Writes clear, complete reports and letters.
- .82 - .94 Studies appropriate technical subjects.
- .71 - .97 Seeks out all available sources for related information.
- .61 - .84 Thoroughly explores the problem situation.
THIRD DIMENSION
1.05 .97 Performs mental operations very rapidly.
- .85 - .89 Accepts opinions and suggestions of others.
- .75 - .90 Coordinates well with the other personnel involved
in a project.
Ln
O
TABLE 2 (CONTINUED)
.90
.57
-1.30
.92
.86
- .94
FOURTH DIMENSION
Remains calm and works well under pressure
Performs job functions thoroughly and methodically.
Antagonizes ocher personnel with temperamental
attitude.
FIFTH DIMENSION
.93 .95 Interprets research results objectively.
-1.04 - .95 Works extra hours if necessary.
- .88 - .85 Tries to do as much work as he can.
SIXTH DIMENSION
.62 .83 Performs assignments on his own with minimum direction.
.57 .74 Obtains results which are responsive to practical needs.
.57 .78 Finds other ways to get things done when original plans
are blocked.
- .97 - .92 Convinces others that his opinion is correct.
- .89 - .90 Expresses himself verbally so that he is easily
understood.
52
In some respects, the interpretations of these
dimensions indicated that the managers had logical per
ceptions of job behavior dimensions. On each dimension,
the job elements clustered at the positive end seemed
to belong together, and those at the negative end also
seemed to belong together. However, the opposite ends
of a single behavior dimension were not usually opposed
in meaning. Since the point of origin of a multi
dimensional scaling solution is arbitrary, it was
realized that the clusters of elements at opposite ends
of a dimension were not necessarily opposites in the
managers' opinions.
Performance Ratings
Distributions.--The frequency distributions
and median ratings for each of the twelve job behavior
rating scales and the one overall performance rating
scale are displayed in Table 3. An indication of the
typical shape of rating distributions can be obtained
from the distributions of scale K, which is the most
skewed, and scale L, which is the least skewed.
53
TABLE 3
PERFORMANCE RATING DISTRIBUTIONS
(N = 68)
Rating
Scale
Rating Seale Categories
Median
1 2 3 4 5 6 7 8 4
A 1 5 5 4 13 13 15 11 2 6.0
B 12 13 13 6 11 2 6 2 3 3.2
C
-
3 8 7 10 11 17 20 6 6.1
D 1 3 4 7 7 9 19 11 7 6.7
E
-
2 6 3 14 13 19 6 6 6.2
F 1 4 2 5 13 10 13 19 2 6.4
G 1 1 4 2 7 15 21 16 2 6.7
H 18 12 8 7 9 6 4 4 1 3.1
I
-
3 3 6 10 17 20 7 2 6.2
J 3 2 4 5 14 8 12 10 11 6.3
K 3 1 4 7 7 10 16 25 6 6.7
L 1 5 2 7 16 14 10 9 5 5.8
oa 3 2 5 7 6 13 11 16 5 6.4
aOverall performance
54
Correlations.--The correlations among the twelve
job behavior rating scales, the overall performance
rating, and age are shown in Table 4. The lines drawn
through the matrix separate the six pairs of scales
which had been previously identified as opposite ends
of dimensions. The relatively large proportions of
variance in each scale which are accounted for by
overall performance are also shown in this table.
Factor Analysis.--Table 5 shows the eigenvalues
and proportions of variance accounted for by each
successive factor extracted in the principal components
factor analyses. These results are shown for two factor
analyses, the one excluding age as a variable, and the
one including age as a variable.
The factor matrix excluding age was rotated for
both six and seven factors, and the one including age
was rotated for eight factors. The factor loadings on
each variable, for each of these three rotations, are
shown in Tables 6, 7, and 8. The seven-factor rotation
with age excluded resulted in virtually the same matrix
as the six-factor rotation, except for the collection
55
TABLE 4
CORRELATIONS AMONG RATING SCALES AND
WITH AGEa
Scale A B C D E F G H I J K L O
A
B -76
C 66 -57
D 70 -67 77
E 68 -53 66 60
F 68 -65 59 61 42
G 62 -66 76 63 50 73
H -45 49 -38 -49 -26 -74 -65
I 57 -53 75 70 68 53 69 -55
J 63 -78 59 65 52 62 64 -44 50
K 81 -75 60 73 55 62 62 -33 47 75
L 58 -48 55 50 62 44 42 -24 51 49 52
0 82 -82 74 73 70 75 73 -48 67 75 83 64
Age 06 07 -07 -03 -01 18 -10 -01 -08 -09 10 07 12
V
P.V. 67 67 55 53 49 56 53 23 45 56 69 41
” 1
aDecimals omitted
^Proportion of variance accounted for by overall
performance
TABLE 5
EIGENVALUES AND PROPORTIONS OF VARIANCE ACCOUNTED FOR IN
FACTOR ANALYSES OF PERFORMANCE RATINGS AND AGE
Age Excluded Age Included
Factor Eigenvalue
Proportion
of Variance
Factor Eigenvalue
Proportion
of Variance
I 8.39 . 646 I 8.39 .600
II 1.15 .088 II 1.17 .083
III 0.91 .070 III 1.15 .082
IV 0.55 .043 IV 0.88 .063
V 0.40 .031 V 0.51 .036
VI 0.38 .029 VI 0.39 .028
VII 0.33 .025 VII 0.34 .024
VIII 0.24 .019 VIII 0.33 .023
IX 0.19 .014 IX 0.24 .017
X 0.18 .014 X 0.19 .013
XI 0.13 .010 XI 0.14 .010
XII 0.09 .007 XII 0.13 .010
XIII 0.05 .007 XIII 0.09 .007
XIV 0.05 .003
TABLE 6
LOADINGS ON RATING SCALES FOR
SIX FACTORS, AFTER VARIMAX
ROTATION3
Factors
Scale I II III IV V VI
A 76 28 19 24 26 33
B -47 -32 -14 -13 -68 -24
C 30 16 78 24 19 34
D 46 23 52 05 33 39
E 35 05 21 34 19 76
F 45 72 29 23 24 01
G 20 53 62 15 35 14
H -09 -93 -13 00 -17 -18
I 06 37 51 17 19 67
J 25 22 26 20 83 16
K 71 11 30 18 51 10
L 22 09 18 87 20 27
0 54 30 34 34 45 30
o
Decimals omitted.
58
TABLE 7
LOADINGS ON RATING SCALES FOR SEVEN
FACTORS, AFTER VARIMAX ROTATION3
Factors
Scale I II III IV V VI VII
A 74 25 16 21 30 35 18
B -42 -30 -12 -11 -71 -25 -13
C 27 14 70 23 20 37 37
D 34 26 26 15 33 28 71
E 34 04 18 28 21 81 10
F 46 69 33 20 26 04 07
G 22 48 69 08 36 22 06
H -08 -94 -11 -03 -17 -13 -15
I 01 38 39 18 18 65 36
J 20 21 23 20 84 15 17
K 66 09 23 18 54 10 28
L 21 09 14 89 21 28 11
0 52 27 33 29 48 34 13
Q
Decimals omitted.
TABLE 8
LOADINGS ON RATING SCALES AND AGE FOR
EIGHT FACTORS, AFTER VARIMAX ROTATIONa
Factors
Scale
I II III IV V VI VII VIII
A 36 04 26 54 23 18 19 59
B -24 09 -31 -80 -12 -11 -11 -20
C 39 -05 15 29 23 69 37 13
D 30 -02 26 45 15 25 71 15
E 83 00 05 32 27 16 09 21
F 07 19 68 43 18 34 07 25
G 22 -10 49 41 08 67 05 08
H -12 01 -94 -18 -03 -10 -13 -02
I 66 -07 39 18 17 36 34 -09
J 18 -08 22 86 18 20 13 -14
K 14 12 09 76 17 25 28 36
L 29 05 10 28 88 14 11 10
0 38 14 27 65 27 34 13 23
Age -02 99 03 -01 03 -04 -01 02
Q
Decimals omitted.
60
of variance for scale D in the one new factor. This
additional factor accounted for only 7.5 per cent of
the total variance, therefore the six-factor solution
is the preferred one. However, if there is particular
interest in measuring the kind of behavior represented
by scale D, then the seven-factor solution should be
used.
In order to evaluate the factor analysis results
and interpret factors for the six-factor rotated
matrix, it was useful to convert all the loadings into
squared loadings which represent proportions of variance
accounted for. These values are shown in Table 9 and
the loadings used to interpret factors are identified.
Table 10 identifies the rating scales on which each
factor had its largest loadings, and which were used
for interpretation of the factors.
Relationship of Job Behavior Dimensions
with Age and Test Scores
The correlations obtained for age and test
scores with performance ratings and factor scores are
61
TABLE 9
SQUARES OF LOADINGS ON RATING SCALES FOR SIX
FACTORS (AFTER VARIMAX ROTATION)3
Scale
Factors
Prop, of
Variance
I II III IV V VI
A 58b 08 04 06 07 11 94
B 22 10 02 02 46b 06 88
C 09 03 61b 06 04 12 95
D 21 05 27 00 11 15 79
E 12 00 04 12 04 58b 90
F 20 52b 08 05 06 00 91
G 04 28 38b 02 12 02 86
H 01 86b 02 00 03 03 95
I 00 14 26 03 04 45b 92
J 06 05 07 04 69b 03 94
K 50b 01 09 03 26 01 90
L 05 01 03 76b 04 07 96
0 29 09 12 12 20 09 91
P.T.V.C 18 17 16 10 17 13
aDecimals omitted.
^Loadings used in the interpretation of factors •
cProportion of total variance accounted for by
each factor. The six factors altogether account
for 91% of the total variance.
62
TABLE 10
INTERPRETATION OF FACTORS
(SIX FACTORS, AFTER VARIMAX ROTATION)
Factor
Scales with
Large Positive
Loadings
Scale with
Large Negative
Loadings
I
£
A Systematic
K Works on his own
II F Coordinates well H Antagonizes
III
C Writes clearly
G Remains calm
IV L Expresses well
V J Works hard B Wastes time
VI
E Mentally fast
I Objective
aFull definitions of scales are shown on the list in
Appendix E.
63
shown in Tables 11 and 12. The factor scores used are
those resulting from the six-factor rotation described
above. The correlations between age and test scores
are also shown.
TABLE 11
CORRELATION OF AGE AND APTITUDE TEST SCORES
WITH FACTOR SCORES AND PERFORMANCE RATINGSa
(For Age, N = 68; for Test Scores, N = 30)
Aze
vcb VRb NRb SRb
Factors
I 20 11 15 -16 04
II 08 -03 -02 04 -02
III -12 08 -55 -33 -42
IV 15 -24 23 -06 14
V -19 -01 -12 -05 -08
VI -i4 -20 -08 19 00
Scales
A 06 -19 -49 -33 -32
B 07 06 -02 -16 -10
C -07 -07 -01 -17 -02
D -03 -13 -20 -12 -10
E -01 05 17 28 21
F 18 -02 -38 -15 -28
G -10 -28 -10 00 02
H -01 10 12 -06 01
I -08 -11 -27 -03 -30
J -09 -23 12 -07 06
K 10 10 -44 -31 -32
L 07 -04 -08 -23 -20
0 12 -22 -13 -23 -17
Age 23 -63 -43 -46
aDecimals omitted.
^Identification of Tests:
VC: Verbal Comprehension
VR: Verbal Reasoning
NR: Numerical Reasoning
SR: Symbolic Reasoning
<)5
TABLE 12
CORRELATION OF PERSONAL INVENTORY SCORES
WITH FACTOR SCORES AND PERFORMANCE
RATING SCALES3
(N = 28)
i F R^ E^ S^ Cb 0^
pb
Vb
Factors
I 29 14 -11 25 -23 33 01 30
II -38 -02 22 -46 12 -37 12 -43
III 02 -10 03 05 16 36 04 -03
IV 11 01 00 -01 00 07 -12 -05
V 02 06 12 -01 20 18 27 14
VI -08 -31 -15 -03 -30 -13 -09 -22
Scales
A 04 -15 09 00 05 35 07 -14
B 31 06 -05 22 -11 53 08 27
C 13 05 -05 08 -12 28 11 02
D 06 00 07 02 06 28 23 03
E 08 -06 -22 23 -19 -22 -33 07
F 11 -01 05 09 01 47 29 11
G 05 -26 -21 05 -35 10 -11 -12
H 28 00 -25 35 -13 21 -20 34
I -14 -39 -06 -18 -17 02 -07 -38
J 14 01 02 03 00 22 00 -01
K -22 -04 07 -26 25 14 15 -24
L -16 -08 10 -33 -04 -01 06 -31
0 00 -37 -06 -14 -31 09 -13 -26
Age . 15
20 23 06 32 22 45 22
p y
Decimals omitted.
k Gordon Personal Profile
A: Ascendancy
R: Responsibility
E: Emotional Stability
S: Sociability
Gordon Personal Inventory
C: Cautiousness
0: Original Thinking
P: Personal Relations
V: Vigor
CHAPTER VI
DISCUSSION
Performance Ratings and Factors
The rating distributions on the performance
rating scales display a moderate degree of leniency, with
medians mostly between six and seven, although five was
the middle scale category. (On two negatively-oriented
scales, the medians were between three and four.) There
was also a moderate degree of distortion due to skewing,
thus discrimination among employees was not as good above
the median as it was below the median. (This was
reversed for the negatively-oriented scales.) However,
there was a good spread of ratings, with considerable
use of seven or eight out of the nine categories on each
scale. Even with some skewing, there were at least two
degrees of performance distinguished on the compressed
side of the median.
Considering the distortions typically obtained
with performance ratings, these scales provided useful
and realistic sets of performance evaluations. The
discrimination among individuals is sufficient for
practical uses, or for further analysis. It is concluded
66
67
that the raters are willing and able to do an effective
job of discriminating among employees on these rating
scales.
When the correlations among the performance
ratings were examined, it was obvious that a very strong
relationship occurred between the overall performance
rating and nearly all of the behavioral rating scales.
For nine of these twelve correlations, coefficients were
obtained at or above .70, and the absolute values of
the coefficients were .48 or above for all of the cor
relations. Two of the coefficients were negative, for
the two "negative" scales descriptive of undesirable
behaviors. These relatively large correlation coeffi
cients are a clear manifestation of the halo effect
commonly found in sets of performance ratings; the
individuals who are considered good tend to be rated
highly on all scales.
This strong and apparently controlling influence
of the overall evaluation corresponds to what Peres
(1962) called the rater bias factor. His interpretation
was that this variance represents bias, a characteristic
of the rater, thus it is not a part of the variation in
performance among employees.
Despite this strong halo effect, it was observed
68
that, for nine of the twelve behavioral scales, the pro
portion of variance not accounted for by overall per
formance was approximately .50, or greater. Thus, there
is a substantial amount of variance which cannot be pre
dicted from overall performance, and which may be
attributed to independent job behavior dimensions.
The intercorrelations among the twelve behavioral
rating scales were also relatively high. Of the sixty-
six intercorrelations, fifty-two had a value of .50 or
greater, although only twelve exceeded .70. It is quite
possible that these high intercorrelations are due to
that proportion of variance of each of them accounted for
by overall performance.
The principal components factor analysis naturally
resulted in a large first component, corresponding to the
halo effect. However, this factor accounted for only
65 per cent of the total variance; three factors are
required to account for 80 per cent of the total variance,
and six factors are required to account for 90 per cent
of the total variance. It was decided to rotate the
first six factors, accounting for a total of 91 per cent
of the variance, to a simple structure solution.
69
After the rotation of the six factors, it was
found that the variance was distributed rather evenly
among them, with proportions of total variance accounted
for ranging from .10 to .13. Furthermore, the halo
factor was spread rather evenly among the six factors.
The proportions of the overall performance variance
accounted for by each factor ranged from .09 to .29.
Each of the six factors had a large loading on one or
two of the behavioral rating scales (absolute values of
.63 to .93), and relatively small loadings on the other
scales.
An examination of the definitions of the rating
scales which have the highest loadings for each factor
reveals relationships which are generally logical and
psychologically meaningful.
The first factor is interpreted as representing
behavior which is systematic and self-directed. The
second factor is clearly concerned with personal
relations, loading an "coordinates well" on the positive
end, and "antagonizes" on the negative end. The third
factor is the least unified, apparently representing
written expression and composure under pressure. The
70
fourth factor is defined by a single rating scale,
quality of expression. The fifth factor represents
application of effort, loading on "works hard" at the
positive end, and "wastes time" at the negative end. The
sixth factor is interpreted as quality of mental activity,
with loadings on "mentally fast" and "interprets
objectively."
It appears that the performance ratings can best
be accounted for by six performance factors, independent
of one another, and of relatively equal importance. None
of the factors is controlled by the halo effect, which
is split up rather evenly among all factors. Each factor
is rather narrowly defined, in terms of the similarity
among the rating scales on which it has high loadings.
Based on these results, the so-called halo effect
is interpreted differently here than it was by Peres
(1962). The evidence from this study suggests that
raters integrate their evaluations on relatively indepen
dent behavior dimensions, when making overall performance
ratings. The overall ratings are of necessity highly
correlated with each of the behavioral dimensions. The
principal component in the initial factor analysis
71
solution represents this composite evaluation. When the
factor matrix is rotated, the independent behavior
dimensions are discovered, reversing the rater integration
process. As a result, the overall performance variance
is evenly distributed among the dimensions. In this
interpretation, the overall performance factor is con
sidered to represent real performance variance, as a
composite of independent behaviors. It is not, as Peres
concluded, an artifact attributable to bias among raters.
As an alternative solution, a varimax rotation
was accomplished with seven factors instead of six,
accounting for 93 per cent of the total variance. The
factor loadings which had been obtained in the six-factor
rotation were virtually unaltered, except that most of
the variance on scale D was pulled out and collected in
the new factor. This is the scale which had been defined
as "studies technical subjects, seeks out all sources of
information, and thoroughly explores problems." Inasmuch
as this new factor did not have sizable loadings on any of
the other rating scales, it was interpreted according to
the above definition. The seventh factor only accounted
for 7.5 per cent of the total variance, compared with over
72
10 per cent for each of the other factors. Therefore,
the six-factor solution is the preferred one, unless
there is particular interest in the kind of performance
described in the additional factor.
Comparison of Results of Factor Analysis
and Multidimensional Scaling
The factor analysis resulted in a dimensional
structure considerably different than that obtained
earlier from the multidimensional scaling analysis. This
is attributed to two conditions. First, the point of
origin was misplaced in the earlier structure, distorting
relationships among the job elements. The six apparently
bipolar dimensions were completely broken up, and the
apparently opposed clusters of job elements emerged
unrelated.
Secondly, these relatively independent clusters
of job behaviors were further combined, in the factor
analysis, into underlying performance factors. This is
interpreted as meaning that, in the abstract ratings of
similarities among job elements, the raters perceived a
certain number of potentially independent behavior
dimensions. However, in the concrete exercise of actual
73
performance evaluations, it turned out that a smaller
number of dimensions was adequate to account for job
performance in this situation.
The dimensional structure revealed by the factor
analysis of performance ratings is considered to be a
more accurate representation of job behavior dimensions
for practical uses than the structure obtained from the
multidimensional scaling analysis. In the first place,
it is a more parsimonious definition of performance
dimensions, without sacrificing psychological meaningful
ness. Furthermore, where there are behaviors at
opposite ends of a single dimension, the relationships
are logical. There are only two factors which are bi
polar, with a positive and a negative end. On both
of these factors, the sets of behaviors with high loadings
on the opposite ends are clearly opposed in meaning.
Relationship between Age and
Job Performance
Age was included as a variable in the matrix of
correlations among the thirteen performance rating scales.
There were no significant correlation coefficients between
74
age and any of the rating scales. A separate factor
analysis was also accomplished, including age as a
variable along with the rating scales. The second com
ponent which emerged had a very large loading on age, so
a varimax rotation was carried out using eight factors,
including the one loaded on age. As a result, a factor
was identified with a loading of .99 on age.
This factor had very small loadings on the rating
scales, and obviously no other factors were loaded on
age, since practically all of its variance was accounted
for. In short, age emerged as an almost totally indepen
dent variable, not accounting for any appreciable part of
performance rating variance.
It is concluded from this evidence that there are
no relationships between age and overall performance, or
any independent dimensions. There is no support for the
position that job performance patterns vary with age,
at least in terms of the kinds of performance measured
in this study.
Relationships between Test Scores,
Job Performance and Age
The number of subjects used in this portion of
75
the study was very small, so at best it would have been
possible to reach only tentative conclusions concerning
the relationships between test scores and job performance.
However, the results did not provide any basis even for
such tentative conclusions. The only relationships which
appeared to be the result of anything other than chance
variation were negative correlations between performance
factor III and three of the aptitude tests: verbal,
numerical, and symbolic reasoning. Factor III has high
loadings on two rating scales corresponding to remaining
calm under pressure and writing clearly. It is difficult
to explain a negative relationship between this factor
and three reasoning tests, and no conclusions are reached
on the basis of this evidence.
Correlations were also computed between test
scores and age for the group of thirty subjects who had
been available for testing. Significant negative cor
relations were found between age and three aptitude
tests: verbal, numerical, and symbolic reasoning.
Several possible interpretations can be suggested
for this evidence. One is that younger persons are, in
fact, superior in these aptitudes. However, it must be
76
considered that these tests are highly speeded, each with
a five-minute time limit. It is quite possible that the
superior performance of the younger subjects was due to
faster performance, rather than superior aptitudes.
Finally, it is known that selection factors which
occur in industry operate to create discrepancies between
groups of younger and older employees in terms of their
abilities. One effect is that of promotion and transfer
to better positions, which tends to remove the better
employees from the older group in any job classification.
An opposite effect is the dropout or termination of
unsuitable employees. The net effect of such trends is
not known, but it can be assumed that the personnel in
any job classification are affected over a period of time
by these selection factors.
Because of these various possible interpretations,
it is impossible to draw any conclusions concerning the
meaning of the observed relationship between age and test
scores.
CHAPTER VII
1
... ■»**'
(
SUMMARY AND CONCLUSIONS
This study was conducted to evaluate a multi
dimensional set of job performance rating scales, in
terms of their independence, their usefulness for
discriminating among individuals, and the strength of
their relationships with age and test results. It was
carried out within a scientific research organization,
using as subjects a group of sixty-eight research
scientists with a generally high level of education, and
with a broad age range within the age limits of
industrial employment.
A list of thirty-one elemental job behaviors was
obtained by eliciting descriptions of observed job
behavior from managers. A set of six independent job
behavior dimensions was derived by a multidimensional
scaling analysis, using as data similarity ratings made
by the managers on the thirty-one job behaviors. Multi
dimensional scaling results in bipolar dimensions, but
the opposite ends of the dimensions were not necessarily
opposed in meaning. Therefore, the six dimensions were
split into twelve separate behavior dimensions for
further analysis.
77
78
Twelve performance rating scales were constructed,
corresponding to the twelve dimensions obtained from the
multidimensional scaling analysis. These twelve scales
and an overall performance rating scale were used to
obtain performance ratings on the sixty-eight subjects.
The rating distributions and intercorrelations were cal
culated and examined in order to evaluate the rating
scales in terms of leniency, normality of distribution,
discrimination among individuals, independence from one
another, and independence from overall performance.
The rating scale correlation matrix was analyzed
by a principal components factor analysis and a varimax
rotation. The resulting factors were interpreted accord
ing to the principal loadings of each on the rating
scales.
Correlations were also calculated between age and
each of the performance rating scales, and a factor
analysis and varimax rotation were carried out with age
included as a variable along with the ratings.
Finally, the scores of thirty subjects on a short
series of aptitude tests and personal inventory scales
were correlated with the rating scales and the factor
scores obtained from factor analysis.
79
follows
The principal results of the study were as
1. A satisfactory solution was obtained for the
multidimensional scaling analysis, with six
dimensions accounting for the similarity
ratings data. However, the point of origin
of this kind of solution is arbitrary, so the
six dimensions were treated as twelve
separate dimensions for rating.
2. The performance ratings exhibited a high
degree of discrimination among individuals,
and only moderate amounts of leniency and
distortion due to skewing.
3. Relatively high intercorrelations were
obtained among the performance rating scales,
and very large correlations between overall
performance and each of the twelve behavioral
rating scales.
4. The factor analysis of performance ratings
resulted in a set of six independent per
formance factors, which accounted for 91 per
cent of the total variance. The six factors
were approximately equal in the proportion of
80
total variance each accounted for, and the
overall performance variance was rather
evenly spread among the six factors.
5. The interpretation of the factors was
realistic and psychologically meaningful.
The factors did not correspond to those
obtained from the multidimensional scaling
analysis. Not only did the opposite ends of
dimensions fail to remain in their bipolar
relationship, but the number of factors
was reduced by combinations among the
original dimensions.
6. Age was not related to the performance rat
ings or the factors.
7. Test scores were not related to the per
formance ratings or the factors. There was a
significant negative correlation between age
and three of the aptitude test scores
(verbal reasoning, numerical reasoning, and
symbolic reasoning).
The results support the first three hypotheses
set forth in this study, to the effect that there is a
81
set of independent job behavior dimensions for this
occupation, that they can be identified by the methods
used in this study, and that useful job performance
ratings can be obtained on the basis of these job
behavior dimensions. It is concluded that, for the
occupation studied, there is a set of six behavioral
performance factors which are independent of each other,
relatively independent of overall job performance, and
of equal importance in accounting for the variance in
the performance rating scales. These factors are inter
preted as representing the following behaviors:
1. Systematic and self-directed
2. Good personal relations
3. Clear writing and calmness under pressure
4. Good expression
5. Application of effort to the job
6. Rapid and objective in thinking
This set of factors is apparently the most
parsimonious structure for explaining differences in per
formance for this occupation. It is further concluded
that the performance ratings based on behavior dimensions
discriminated well among individuals, with only a
moderate discrepancy from a normal distribution.
The other two hypotheses, concerning the
relationships of age and test scores with job perform-
82
ance, are not supported by the results. No relationships
were demonstrated between age and job performance;
however, it is not possible to draw conclusions concern
ing the possibility of such relationships. The
selection factors in industry, described in the previous
chapter, operate to bias a comparison between the older
and younger employees in an occupation or work group,
and the age difference is confounded with ability
differences of an unknown direction and amount.
No relationships were demonstrated between
test results and job performance, but again it is not
possible to draw any conclusions. The limitation in
sample size for this portion of the study, the limit
ation in the scope of tests used, and the homogeneity
of the group, make it impossible to conclude that test
results are not related to job performance, even though
there is no evidence of a relationship.
The results suggest the need for certain
other lines of research. Inasmuch as it was possible to
identify job behavior dimensions as described above, this
type of research should be extended to bther levels of
employee, such as the subordinates rather than the
83
managers, and to other occupational groups. It is
believed that ultimately a domain of occupational
behaviors might be mapped out, and that particular
occupations can be described in terms of that portion
of the domain which is involved in the occupation.
It would also be useful to fully explore the
possibility of relationships between test scores and job
behavior dimensions. Other potential predictors of
job behavior should be used, and it may even be
necessary to develop new kinds of predictors.
The failure to detect any relationships between
age and job performance, and the many possible inter
pretations for relationships between age and test
scores, indicate that studies involving the age
variable in industry require new kinds of research
design. It will be necessary to control or measure the
complex factors determining the age distribution within
any occupational group, and longitudinal studies may be
the only feasible approach.
REFERENCES
Anderson
Balma, M
Birren, ,
Brown, K
Dixon, W
Dunnette
Dunnette
Ghiselli
Ghiselli
Gordon, ]
Gordon, ]
Griew, S
Guion, R
Guion, R
, J. E. The use of time and energy. In J. E.
Birren (Ed.), Handbook of aging and the
individual. Chicago: U"! of Cnicago Press, 1959.
. J. The concept of synthetic validity.
Personnel psychol., 1959, 12, 395-396.
J. E. The psychology of aging. Englewood
Cliffs, N.J.: Prentice-Hall7 1964.
. R. Job analysis by multidimensional scaling.
J. applied psvchol.. 1967, 51, 469-475.
. J. BMD, Biomedical computer programs.
Berkeley: U. of California Press7 1968.
M. D. A note on the criterion. J. appl.
psychol., 1963, 477 Z51-254.
M. D. Personnel selection and placement.
Belmont, Calif.: Wadsworth, l96o.
E. E. Dimensional problems of criteria. J.
appl. psychol., 1956, 40, 1-4.
E. E. The generalization of validity.
Personnel psychol., 1959, 2, 397-402.
V. Gordon personal inventory manual. 1963
revision. New York: Harcourt, Brace L World,
T O ; Ca)
.. V. Gordon personal profile manual, 1963
revision. New Vork: Harcourt, Brace & World,
T O — Cb)
Job redesign. Paris: Organisation for
economic co-operation and development, 1964.
M. Criterion measurement and personnel judg
ments. Personnel psychol., 1961, 14, 141-149.
M. Personnel testing. New York: McGraw-Hill,
1965.
84
85
Harris, D. H., & Chaney, F. B. Human factors in quality
assurance. New York: Wiley, 1969.
Hemphill, J. K. Job descriptions for executives,
Harvard business rev., 1959, 37^ 5, 55-6/.
Kirchner, W. K. & Dunnette, M. D. Identifying the
critical factors in successful salesmanship.
Personnel, 1957, 34, 54-59.
Korman, A. K. The prediction of managerial performance:
a review. Personnel psychol., 1968, 21, 295-322.
Kruskal, J. B. Multidimensional rating by optimizing
goodness of fit to a nonmetric hypothesis.
Psychometrika, 1964, 21, 1-27.
Kruskal, J. B. How to use M-D-SCAL, a program to do
multidimensional scaling and multidimensional
unfolding. Cambridge, Mass.: Marketing Science
Institute, 1968.
Kuhlen, R. G. Aging and life-adjustment. In J. E.
Birren (Ed.), Handbook of aging and the
individual. Chicago: tT of Cnicago Press, 1959.
pp. 852-897.
Mager, R. F. Preparing instructional objectives.
Palo Alto, CalT: Fearon, 1962.
McFarland, R. A., & O'Doherty, B. M. Work and occupa
tional skills. In J. E. Birren (Ed.), Handbook
of aging and the individual. Chicago: U. of
Chicago Press, 1959, 452-500.
Peres, S. H. Performance dimensions. Personnel
psychol., 1962, 15, 405-410.
Ritti, R. R. Control of halo in factor analysis of a
supervisory behavior inventory. Personnel
psychol., 1964, 17, 305-318.
Ruch, F. L., & Ruch, W. W. Employee aptitude survey,
technical report. Los Angeles: Psychological
services, 1963.
86
Rundquist, E.'A. The prediction ceiling. Personnel
psychol., 1969, 22, 109-116.
Schultz, D. D. & Siegel, A. I. The analysis of job
performance by multidimensional scaling tech
niques. J. appl. psychol., 1964, 48, 329-335.
Siegel, A. I., & Pfeiffer, M. G. Factorial congruence
in criterion development. Personnel psychol.,
1965, 18, 267-279.
Stenson, H. H., & Knoll, R. L. Goodness of fit for
random rankings in Kruskal's nonmetric scaling
procedure. Psychol. bull., 1969, 71, 122-126.
Thorndike, R. L. Personnel selection. New York:
Wiley, 1949"!
Tiffin, J. & McCormick, E. J. Industrial psychology.
(5th ed.) Englewood Cliffs, N.J.: Prentice-Hall,
1965.
Torgerson, W. S. Theory and methods of scaling.
New York: Wiley, 1958.
87
APPENDIX A
LIST OF JOB BEHAVIORS USED IN MULTI
DIMENSIONAL SCALING ANALYSIS
1. Performs assignments on his own with minimum
direction.
2. Finds other ways to get things done when original
plans are blocked.
3. Works extra hours if necessary.
4. Thinks of many new ideas and approaches.
5. Performs mental operations very rapidly.
6. Plans and schedules work systematically.
7. Thoroughly explores the problem situation.
8. Seeks out all available sources for related
information.
9. Writes clear, complete reports and letters.
10. Spends a great deal of time on non-work subjects.
11. Defends his opinion when it is questioned.
12. Follows instructions of his superior.
13. Accepts opinions and suggestions of others.
14. Applies appropriate research/engineering principles
to his work.
15. Remains calm and works well under pressure.
16. Obtains results which are responsive to practical
needs.
88
17. Is frequently absent or unavailable when needed.
18. Expresses himself verbally so that he is easily
understood.
19. Performs job functions thoroughly and methodically.
20. Provides enough information to manager on status
of his work.
21. Coordinates well with the other personnel involved
in a project.
22. Willingly assists other personnel with their
problems.
23. Antagonizes other personnel with temperamental
attitude.
24. Spends too much time on relatively unimportant work.
25. Conforms to project schedules and budgets.
26. Recognizes the important problems in a situation.
27. Studies appropriate technical subjects.
28. Convinces others that his opinion is correct.
29. Tries to do as much work as he can.
30. Interprets research results objectively.
31. Maintains pleasant personal relationships.
89
APPENDIX B
INSTRUCTIONS FOR SIMILARITY RATINGS
SIMILARITIES AMONG JOB BEHAVIORS
Please indicate your judgments concerning the similarities
among a set of job behaviors, by following the instruc
tions given below.
1. Read the complete list of job behaviors on the next
page.
2. Turn to the following page and read the job behavior
shown at the top of the page. Notice that all the
other job behaviors are listed on the page, and
there is a numbered scale for each of them.
3. Circle the number on each scale which represents
your judgment as to the similarity between that job
behavior and the one at the top of the page.
NOTE: The left-hand end of each scale (Category 1)
represents the least degree of similarity possible.
The right-hand end of each scale (Category 9) repre
sents the maximum degree of similarity possible.
The categories in between represent intermediate
degrees of similarity. (See examples below.)
4. Now repeat this procedure for each page in the book
let. A different job behavior is shown at the top
of each page. Once you have completed a page, do
not go back to it.
Please make your judgments and mark the scales rapidly,
spending a minimum amount of time on each judgment.
Examples of Similarity Ratings
For each of the job behaviors listed below, circle the
number which indicates its similarity to:
90
Least
Simi
larity
Max.
Simi
larity
Understands technical
problems
Solves problems
1
rapidly ------------ 1 2 3 4 5 6 7 (8^
) 9
Shares credit for
(?) 3 4 5 6 7 3 success ------------ 1 9
The scales are marked to show that the person
making the judgments feels that "Understands technical
problems" is quite similar to "Solves problems rapidly,"
but not very similar to "Shares credit for success."
91
APPENDIX C
INSTRUCTION SHEET FOR JOB PERFORMANCE RATINGS
JOB PERFORMANCE RATINGS
I. RATING SCALES
Ratings are to be made on thirteen rating
scales, each on a separate page of this booklet.
The first twelve scales measure specific
dimensions of job behavior, and the final rating
scale is a measure of overall job performance. The
persons to be rated are listed on each page, with
the rating scale.
II. GENERAL WORK CATEGORIES
A set of general work categories is listed
below, along with the names of the persons you will
be rating. Please enter, for each person, the
number of the one work category which best describes
the work he has been doing.
WORK CATEGORIES
1. Human Engineering Studies and Applications
2. Engineering Studies other than Human Factors
3. Life Sciences Research
4. Physical Sciences Research
5. Analysis, Statistics and Computer Technology
6. Design and Development of Research Equipment
7. Administration, Planning, Scheduling, Super
vising
8. Layout, Art, and Presentations
9. Other (Describe opposite name)
NAME WORK NAME WORK
CATEGORY CATEGORY
92
APPENDIX D
EXAMPLE OF A JOB PERFORMANCE RATING SCALE
RATING SCALE C. The kind of job behavior to be rated
on this scale is described below:
Individual writes clear, complete reports and
letters.
Indicate your rating of each person by placing a check
opposite his name, in one of the rating scale
categories.
Scale category 9 represents the highest degree of this
job behavior, and scale category 1 represents the least
degree.
The other scale categories, from 2 through 8, represent
regularly increasing degrees of the described behavior.
R A T I N G SCALE C A T E G O R I E S
NAMES
(Least)
9
HigH
est)
93
APPENDIX E
LIST OF PERFORMANCE RATING SCALE
DEFINITIONS
A. Plans and schedules work systematically, and conforms
to schedules and budgets.
B. Spends too much time on unimportant or non-work
activities, and is frequently absent or unavailable
C. Writes clear, complete reports and letters.
D. Studies technical subjects, seeks out all sources of
information, and thoroughly explores problems.
E. Performs mental operations very rapidly.
F. Coordinates well with other persons, and accepts
their opinions and suggestions.
G. Remains calm under pressure, and performs work
thoroughly and methodically.
H. Antagonizes other persons with temperamental
attitude.
I. Interprets research results objectively.
J. Tries to do as much work as he can, and works extra
hours if necessary.
K. Works on his own with minimum direction, finds
other ways to get things done when necessary, and
obtains practical results.
94
L. Expresses himself well verbally and convinces others.
0. Overall job performance.
95
APPENDIX F
LOADINGS OF DIMENSIONS ON JOB BEHAVIORS,
OBTAINED FROM MULTIDIMENSIONAL SCALING
ANALYSIS (AFTER VARIMAX ROTATION)
Job
Behaviors
Dimensions
I II III IV V VI
1 .31 .06 .24 .06 .18 .62
2 .17
-
.36
-
.13 - .06
-
.21 .57
3 - .19 .06 .15 - .08
-1.04 .26
4 - .14
-
.46 .55 - .31 .02 .35
5 - .14 .23 1.05 .05 .01 .05
6 .77 .02 .01 .23 .03 .02
7 .32
-
.61
-
.09 - .16 .07 .14
8 - .06
-
.71
-
.07 .13
- .03 .13
9 .33 .59 .27 .31 .41 - .17
10 -1.19
.50 .14 - .10
-
.31 - .46
11 - .02 .12 .37 - .70 .21 - .86
12 .37 .48
-
.56 - .29
-
.08 .54
13 - .26
-
.17
-
.85 - .11 .02 .26
14 .33
-
.27 .44 .07 .37 .09
15 .26 .10
-
.20 .90 .17 .05
16 .19
-
.07 .03 .17 .45 .57
17 -1.20 .52 .15 - .41 .18 .20
18 .05 .28
-
.23 .11 .24 - .89
19 .16
-
.08 .21 .57
-
.18 .09
20 .49 .42
-
.10 - .18 .53 .12
21 .23 .03
-
.75 .10 .20 - .16
22 - .70 .44
-
.45 .65
-
.16 - .08
23 - .23 .14
-
.04 -1.30
-
.30 - .27
24 -1.26 .02 .05 - .25
- .45 - .26
25 .90 .15
-
.01 .05
-
.38 .07
26 .20
-
.57
-
.08 .37 .57 - .01
27 .17
-
.82 .22 .06 .07 - .03
28 .06
-
.05 .00 - .37
-
.23 - .97
29 .23
-
.37 .20 .23
-
.88 .13
30 - .12
-
.20 .21 .10 .93 .02
31 - .05 .58 - .74 .16 - .09 - .13
t

Linked assets

University of Southern California Dissertations and Theses

Conceptually similar

PDF

Electrophysiological Properties Of Reaction Time And Aging

PDF

Biofeedback Control Of The Eeg Alpha Rhythm And Its Effect On Reaction Time In The Young And Old

PDF

Psychomotor Performance And Change In Cardiac Rate In Subjects Behaviorally Predisposed To Coronary Heart Disease

PDF

The Effect Of Anxiety And Frustration On Muscular Tension Related To The Temporomandibular-Joint Syndrome

PDF

Simple Auditory Reaction Time Of Young Adult And Elderly Subjects In Relation To Perceptual Deprivation And Signal-On Versus Signal-Off Conditions

PDF

Signal Processing Time And Aging: Age Differences In Backward Dichoptic Masking

PDF

The Effects Of Diphenylhydantoin On The Galvanic Skin Responses Of Psychopathic And Normal Prisoners

PDF

Effects Of Group Behavior Therapy Imagery On Basketball Performance

PDF

Differential effects of epinephrine and propranolol on shuttle box avoidance learning in rats of different ages

PDF

Operant Discrimination Of An Interoceptive Stimulus In The Urinary Bladder Of Intact And Dorsal Root Transected Female Rhesus Monkeys

PDF

Age Differences In Serial Reaction Time As A Function Of Stimulus Complexity Under Conditions Of Noise And Muscular Tension

PDF

Audience Effect And Age Differences On Tasks Of Increasing Complexity

PDF

A Monte Carlo Evaluation Of Interactive Multidimensional Scaling

PDF

Imagery And Response Styles In Desensitization

PDF

Cue-Dependent Amnesia

PDF

Age, Sex, And Task Difficulty As Predictors Of Social Conformity: A Search For General Tendencies Of Conformity Behavior

PDF

Influences Of Interoperative Experience And Age On Recovery Of Visual Function Following Two-Stage Lesions Of The Striate Cortex

PDF

Organization, Conflict, And Change: A Test Of A Multivariate Model Within Two Types Of Simulated Social Systems

PDF

Anxiety Level And The Repression-Sensitization Dimension In Desensitization Therapies

PDF

The Conditioning Of A Discriminative Stimulus Measured As An Orienting Reaction In Profoundly Retarded Blind Children

Asset Metadata

Creator Hane, Edward Ziegler (author)

Core Title Performance Evaluation Based On Multidimensional Job Behaviors

Degree Doctor of Philosophy

Degree Program Psychology

Publisher University of Southern California (original), University of Southern California. Libraries (digital)

Tag OAI-PMH Harvest,Psychology, industrial

Language English

Contributor Digitized by ProQuest (provenance)

Advisor Birren, James E. (committee chair), Galbraith, Gary C. (committee member), Holmen, Milton G. (committee member)

Permanent Link (DOI) https://doi.org/10.25549/usctheses-c18-455787

Unique identifier UC11362320

Identifier 7112390.pdf (filename),usctheses-c18-455787 (legacy record id)

Legacy Identifier 7112390

Dmrecord 455787

Document Type Dissertation

Rights Hane, Edward Ziegler

Type texts

Source University of Southern California (contributing entity), University of Southern California Dissertations and Theses (collection)

Access Conditions The author retains rights to his/her dissertation, thesis or other graduate work according to U.S. copyright law. Electronic access is being provided by the USC Libraries in agreement with the au...

Repository Name University of Southern California Digital Library

Repository Location USC Digital Library, University of Southern California, University Park Campus, Los Angeles, California 90089, USA