Close
About
FAQ
Home
Collections
Login
USC Login
Register
0
Selected
Invert selection
Deselect all
Deselect all
Click here to refresh results
Click here to refresh results
USC
/
Digital Library
/
University of Southern California Dissertations and Theses
/
Measuring the alignment of high school and community college math assessments
(USC Thesis Other)
Measuring the alignment of high school and community college math assessments
PDF
Download
Share
Open document
Flip pages
Contact Us
Contact Us
Copy asset link
Request this asset
Transcript (if available)
Content
Copyright 2008 Alison R. Shelton
MEASURING THE ALIGNMENT OF HIGH SCHOOL AND COMMUNITY
COLLEGE MATH ASSESSMENTS
by
Alison R. Shelton
_____________________________________________________________
A Dissertation Presented to the
FACULTY OF THE ROSSIER SCHOOL OF EDUCATION
UNIVERSITY OF SOUTHERN CALIFORNIA
In Partial Fulfillment of the
Requirement for the Degree
DOCTOR OF EDUCATION
August 2008
ii
DEDICATION
For my family.
iii
TABLE OF CONTENTS
DEDICATION ii
LIST OF TABLES v
ABSTRACT vi
CHAPTER 1: INTRODUCTION 1
Statement of the Problem 1
Background and Context of the Problem 4
Alignment 7
Purpose of the Study 10
Research Questions 10
CHAPTER 2: A REVIEW OF THE LITERATURE 12
K-12 and Higher Education Disconnect 12
The Bridge Project 13
Standards for Success 20
Achieve and the American Diploma Project 22
The Context of Placement Testing in California 25
The Structure of California’s Public Education System 25
K-16 Articulation 28
Assessments in California 30
Remediation in California 32
Validity of Placement Tests and Policies 34
Measuring Alignment 37
What is Alignment? 37
Alignment Models 41
Previous Alignment Studies 45
CHAPTER 3: METHODOLOGY 53
Research Questions 54
Alignment Rating Process 55
Alignment Criteria 57
Analysis 62
CHAPTER 4: RESULTS 64
Analysis 64
Indices of Alignment 64
Research Questions 69
iv
CHAPTER 5: DISCUSSION 75
Implications for Policy and Practice 76
Limitations of the Study 83
Suggestions for Future Research 83
Conclusion 85
REFERENCES 87
v
LIST OF TABLES
Table 1: Example Standard Content Matches 59
Table 2: Example DOK Ratings 60
Table 3: Rater Reliability 64
Table 4: Categorical Concurrence Ratings 65
Table 5: Depth of Knowledge Ratings 66
Table 6: Range of Knowledge Ratings 68
Table 7: Balance of Representation Ratings 69
vi
ABSTRACT
More than 60% of all community college students are placed into
remedial, non-credit bearing courses. Concerns over the lack of articulation
across the K-12 and postsecondary educational systems have led to
questions regarding whether students have had the opportunity to learn and
demonstrate the skills required for success in college level classes. To
measure the degree to which the expectations across these systems are
consistent, the degree of alignment between the examinations at these two
levels was explored. The California Community College placement test
content was compared to the high school level California Standards Tests in
General Mathematics, Algebra 1, and Geometry. Only the General
Mathematics Test was aligned across a substantial number of standards.
Taking into consideration past studies, it appears that the major source of
misalignment between the two testing systems occurs within the content
areas of Integers and Rationals, Trigonometry, and Graphing.
1
CHAPTER 1
INTRODUCTION
This study is designed to explore the consistency of high school and
community college level expectations. In California, secondary and
postsecondary educational systems largely operate independently of one
another. There is little incentive for the state universities, community
colleges and high schools to communicate and work together. One way this
has been manifested is in the high rates of high school graduates needing
postsecondary remediation.
Statement of the Problem
High Remediation Rates. The majority of California high school
graduates who go on to attend California’s community colleges need some
form of remediation. In general, this means that students end up taking
classes that they had previously passed in high school (Willett, 2007).
Increasing college enrollments and fiscal concerns have brought issues
associated with remediation to the forefront of today’s educational policy.
Educational leaders across the system are looking for ways to reduce the
need for students to take remedial courses once they enroll in postsecondary
institutions.
2
High rates of remediation suggest that there is a pervasive lack of
preparation for college among recent high school graduates. The low
proficiency rates on high school standardized tests support this claim
(California Department of Education, 2008). However, low performance is not
necessarily the issue for every remedial college student. For instance, the
mean GPA of California State University remedial math students is well over
a 3.0 ("Fall 2006 final regularly admitted first-time freshmen remediation
systemwide"). This indicates that students who are earning A’s and B’s in
high school, and meet the minimum course requirements for university
admission, are still being placed into remedial courses. Furthermore, there is
evidence that many students who are placed in remedial classes could have
been successful in college level courses (Armstrong, 1999, 2000). This
suggests that the remediation problem is more complex than just the lack of
preparedness of high school graduates. To get an accurate picture of the
issue we need to look beyond merely preparation to other issues such as
motivation, school quality, and the appropriateness of the tests themselves.
High school GPAs indicate that college bound students are meeting
expectations for college preparation but remediation rates indicate that they
are not meeting these standards. Students who have taken a college
preparatory curriculum with over a B average have reason to believe that
they are well prepared for college and are rightly surprised when they are
placed into remedial coursework. Furthermore, the question of how well
3
prepared students are for college level courses is confounded by the
appropriateness of the placement policies in effect. Placement decisions
need to be valid and fair if we are going to make decisions that impact
students. Without knowing if the placement tests are validly assessing
students, we can’t assume more remediation in high school or other
interventions will help to address the remediation problem.
Lack of validity evidence. Addressing concerns about widespread
remediation necessitates looking at placement policies. Are students
adequately placed in appropriate classes? Are the assessments valid and
fair? There are many dimensions of validity that could be explored. We
could look at cut scores and how well they predict student performance. We
could explore issues of reliability or examine the assessment for bias.
However, we need to first establish that the subject matter of the exam is
appropriate for its intended use, that it has content validity. Many studies
have been conducted that explore the predictive validity of placement tests
used by the California Community College System (Armstrong, 1994; Isonio,
1992). However, placement test scores alone are not necessarily better
predictors of success than high school transcripts (Marwick, 2004).
Furthermore, no matter how predictive a score is of student future success, it
is not a fair assessment if it is predicated on content that students have not
had the opportunity to learn. Entry-level college placement tests that are
based on content outside of the standard college preparatory curriculum are
4
biased against students who do not have the additional resources to prepare
for them.
Current community college matriculation policies utilize assessments
to make placement decisions without evidence that they adequately
assesses the prerequisite knowledge that is necessary for student success.
Therefore, before we worry about cut scores and related issues we need to
establish the content validity of these tests.
The problem. High school students prepare for college by taking
college preparatory courses, which is defined in California as the a-g
coursework requirements for university admission (Education Trust-West,
2004; University of California, 2008). In the absence of specific matriculation
standards, the content of college placement tests becomes the “de facto”
standard for college readiness. However, there is no clear evidence that this
placement test content assesses students on the standards associated with
prerequisite high school level courses. In other words, there is a lack of
evidence as to whether the secondary and postsecondary educational
systems agree about the content skills that are necessary to prepare
students for college level work.
Background and Context of the Problem
There is clear evidence that nation-wide expectations between high
school and college are not well articulated (Achieve Inc., 2007a; Conley,
5
2003b; Kirst & Venezia, 2004; Venezia, Kirst, & Antonio, 2003). In a recent
study of the disconnect between high school and college, the Stanford
University Bridge Project researchers made several suggestions to help
improve student success through better articulation between K-12 and
postsecondary educational systems. Included in these suggestions was the
recommendation that states examine the relationship between high school
standards and postsecondary placement examinations (Venezia, Kirst, &
Antonio, 2003). In order to improve the articulation between these levels in
California, it is important to have some measurement of the degree to which
these expectations agree.
California Community Colleges are open access institutions with the
mission to serve anyone who can benefit (California Community Colleges,
2006). In practice, this means that anyone with a high school diploma, and
even some without, can enroll in these institutions. Students take placement
exams that determine if they will be placed in college level or basic-skills
level courses. These course placements can have a significant impact on an
individual student’s course-taking trajectory and ultimate degree attainment.
For those who are placed in the lowest level remedial courses, the chance of
completing a degree is quite low. For instance, students placed in arithmetic
level courses have only a 10% chance of eventually attempting transfer level
math (California Community Colleges, 2006). This issue disproportionately
affects students of minority background; as 65% of African American, 53% of
6
Mexican American, and 54% of other Latino students are placed in remedial
math courses, compared to only 26% of White students.
Standards. One component of the No Child Left Behind Act is the use
of standards based, high stakes testing to assess the proficiency level of
students ("No Child Left Behind Act of 2001", 2002). These assessments
function as messages to teachers and students about what is important to
teach and learn (Herman, Webb, & Zuniga, 2007). They do not provide
feedback to the students on their progress towards college readiness
(Lundell, Higbee, Hipp, & Copeland, 2004). However, students who are
performing well in college preparatory classes have reason to believe that
they are successfully preparing themselves for college level work.
Community colleges are, by mission, open access institutions
(California Department of Education, 1960). They have no explicitly defined
entrance standards or standards for college level work. The University of
California system has outlined minimum standards for college preparation by
defining the course requirements for admission and requiring high schools to
petition for approval of all courses designed to meet these requirements
(University of California, 2008). Part of the California Community College
mission includes preparing students to transfer to the state four-year
university system (California Community Colleges, 2006); consequently, their
definition of college preparation should be in accordance with the UC/CSU
systems. In other words, students who have mastered the content in the
7
college preparatory curriculum should be able to place into college level
coursework. Therefore, if the placement test content is valid for its intended
use in the California Community College System, it should test students on
content that is part of the stated college preparatory curriculum. In
mathematics, this curriculum requires students to take math through Algebra
II. For this reason, remedial placements should be determined by the
standards of these courses and any content outside of this course sequence
that is necessary for future math success should be included in credit-
bearing college math courses.
Alignment
Many suggestions have been made for improving the alignment
between high school and college, most of which will not be discussed here.
One common recommendation for improving alignment is to ensure that
placement exams used in colleges are aligned with the standards used to
prepare students for college. Preferably this could be achieved through the
implementation of a systemwide set of exams, benchmarked on explicit K-16
and university standards (Kirst, 1998). These aligned standards and
assessments can improve the quality of signals and incentives provided to
students as they prepare for college (Venezia, Kirst, & Antonio, 2003). As a
first step in addressing these suggestions, the degree to which college
8
placement tests currently align with high school standards should be
explored.
Alignment and Validity. Alignment is essentially a measure of content
validity (Koretz & Hamilton, 2006). Exams that are well-aligned test
essentially the same content at appropriate levels of cognitive complexity.
Therefore, measuring the alignment between the high school end of course
exams and the community college placement tests provides content validity
evidence.
Should these tests be well-aligned, then placement decisions are
made by assessing students on pre-requisite knowledge they have had the
opportunity to learn. Furthermore, community college personnel will have the
option to utilize past standards scores to make placement decisions more
economically and efficiently. When tests are well-aligned, earlier
assessments can be used to benchmark progress towards meeting later
requirements. In California, an aligned testing system is important if one is to
make judgments regarding individual student progress. More specifically, a
well-aligned testing system across grade levels would allow educational
personnel to diagnose problems relating to individual students’ preparation
for college, while there is still time to correct them.
Should these tests fail to be well-aligned, placement decisions that
have an enormous impact on students’ future successes are being made
based upon standards that are not emphasized in their pre-requisite, pre-
9
collegiate courses. If the tests are not well-aligned, then the content taught
in most high school math classes will be significantly different from what
postsecondary institutions expect students to have mastered before
beginning college level work. The differential focus between the two levels
will create barriers to student achievement.
Alignment methodologies. Norman Webb has designed one of the
most comprehensive and adaptable models for judging test alignment
(Bhola, Impara, & Buckendahl, 2003). Webb’s model assesses alignment in
four dimensions: categorical concurrence, depth of knowledge, range of
knowledge and balance of coverage. These categories give a
comprehensive picture of how well aligned assessments are in content,
complexity and breath of coverage across and within standards. (see Brown
& Niemi, 2007; N. L. Webb, 1997; N. L. Webb, 1999)
A study conducted by Conley and Brown, examines standardized test
items to ascertain the degree of alignment with Knowledge and Skills for
University Success (Brown & Conley, 2003) but does not look specifically at
California standards or the placement tests which serve as “de facto”
standards for embarking upon college level coursework. Brown and Niemi
recently conducted a study assessing the alignment between the
aforementioned California Community College placement test content and
the Algebra II and Summative High School Mathematics Standards Tests
(Brown & Niemi, 2007). These standards tests, however, are taken by only
10
18% of the best academically prepared California high school students.
Community colleges, in contrast, are designed to provide a college education
to every student who can benefit.
Purpose of the Study
No previously conducted study measures the alignment between the
community college entrance expectations and the tests taken by the majority
of California students. The California Standards Tests in General
Mathematics, Algebra I, and Geometry are cumulatively taken by 81.2% of
high school students, therefore, it is both important and informative to
investigate the alignment between the standards associated with these tests
and the “de facto” standards defined by the college placement tests. This
study, combined with the work of Brown and Niemi, will provide a clear
picture of how well the community college expectations are aligned with the
California Standards Tests taken by 99.2% of California students.
Research Questions
How well do the California Standards Tests (CST) in General
Mathematics, Algebra I, and Geometry align with the content of the most
commonly utilized math placement tests in the California Community College
(CCC) System?
11
Are there areas emphasized on the mathematics placement tests that
are not measured by the California Standards Tests?
To what degrees can performance on the CSTs measure progress
towards meeting the “de facto” college readiness standards in mathematics,
as defined by CCC placement test content?
12
CHAPTER 2
A REVIEW OF THE LITERATURE
Assessment alignment is part of the larger issue of the consistency of
expectations across secondary and postsecondary educational systems.
Therefore, this literature review will begin by introducing studies that explore
the transition from high school to college. Also, in addressing the alignment
of assessments across levels, it is vital to understand the structural context
within which the assessments are administered. Consequently, the context
of testing in California will be briefly explored. Finally, since the consistency
between secondary and postsecondary assessments in California will be
measured in this study using alignment methods, an overview of alignment
will be provided.
K-12 and Higher Education Disconnect
Historically, there has been little articulation between what occurs at
the secondary and postsecondary levels of education across the country.
Awareness and concern over the disconnect between secondary and
postsecondary expectations has been steadily increasing. The increased
awareness can be attributed, in part, to several recent studies that explored
the transition from high school to college. Researchers from Stanford’s
13
Bridge Project, Standards for Success, and the American Diploma Project
have helped illuminate some of the barriers to achievement faced by today’s
youth as they transition to postsecondary training.
The Bridge Project
The Bridge Project is the result of a six-year study conducted by
researchers for Stanford University (Kirst & Venezia, 2004) on the transition
from K-12 education to college. Six states were explored, including
California. The purpose of the Bridge project was to encourage policy
development that supports student access and achievement in
postsecondary education, under the assumption that these policies serve as
signals that influence students’ actions and motivations.
The results of surveys given by researchers for the Bridge Project
indicate that nearly 90% of students entering high school intend to pursue a
higher education; including more than 80% of the most underrepresented
minority students (Venezia, Kirst, & Antonio, 2003). Furthermore, more than
70% of students actually attend some form of postsecondary training after
high school, often choosing to enroll at the local community colleges.
Students are attracted to the option of community colleges for many
reasons. Some choose this option for reasons of convenience such as the
location or cost. Others utilize the open access of community college as a
second chance (Bueschel, 2004). Regardless of the reason they choose to
14
attend these institutions, students often find a lack of consistency between
what is expected of them in college and their secondary school experiences
(Kirst, 2007). They find that even open access institutions have academic
expectations and that completing a high school diploma is not necessarily
sufficient to meet those expectations. This discovery is particularly
significant for low-income students, of which a greater percentage are found
to be unprepared or marginally qualified for college (Venezia, Kirst, &
Antonio, 2003).
There are many barriers that contribute to the disconnect between
high school and colleges. These include, but are not limited to, the disparate
expectations between secondary and postsecondary institutions, unequal
opportunities for preparation and “a confusing array of state and institutional
exams” (Venezia, Callan, Finney, Kirst, & Usdan, 2005, p. 2). These barriers
are intensified by the lack of connections between the different educational
levels.
The findings of the Bridge Project indicate that the disparate
institutional expectations are significant barriers to student success. High
school graduation standards do not align well with community college
entrance expectations (Kirst, Venezia, & Antonio, 2004). Students are
unclear about college expectations; and, their high school teachers lack the
information necessary to help clarify these expectations (Kirst, Venezia, &
Antonio, 2004). Furthermore, although California has been successful in
15
aligning assessments and standards in the K-12 arena, little has been done
to address the connection between these standards and higher education in
this state (Antonio & Bersola, 2004). K-12 reforms are occurring largely
without regard to what students are expected to know when they enter
college. Additionally, community college respondents to the Bridge Project
studies view these reforms as essentially irrelevant to what is happening at
their institutions and have little interest in the associated standards and
assessments (Bueschel, 2004).
Bridge Project findings point to marked differences between high
school and college level assessments. These differences highlight a
difference in expectations at the two levels. What is expected for students to
know and demonstrate to exit the K-12 system is markedly different from
what is expected of students only months later when they enter college.
These different standards, which originated in a time when a much smaller
proportion of students attended college, are confusing to students in an era
when most students enter high school with the aspiration to complete at least
some level of college education (Kirst, Venezia, & Antonio, 2004).
Bridge Project researchers (Antonio & Bersola, 2004) found that very
few students were aware of college entrance and placement test
requirements. College placement requirements and related policies are
particularly confusing and frustrating to community college bound students
(Kirst, Venezia, & Antonio, 2004). They expect, due to the policy of open
16
admission, to be able to register for any course they desire. They are often
surprised when course placement policies prevent them from taking college
level courses right away (Bueschel, 2004).
One area that contributes to the students’ confusion over college
preparatory expectations is the bewildering and varied set of assessments
they are expected to undertake between high school and college. In
California, many students are expected to attempt more than 20
examinations (Kirst & Venezia, 2004), and they are startling different in
content. For instance, the math assessments analyzed by RAND
researchers tested students at different cognitive levels. High school tests
assessed low-level content where college admissions and placement
assessments tested higher-level material (Kirst & Venezia, 2004). Some of
the differences in content stem from differences in use, for instance
assessing readiness for college level coursework or determining whether a
student has met minimum high school exit standards. However, this
contributes to students taking a myriad of standardized assessments that do
not align well in either purpose or content (Antonio & Bersola, 2004). As a
result, both students and their teachers are unclear about what students are
expected to know before they begin college. The policies that lead to these
multiple assessments promote a culture in which postsecondary and K-12
educators are largely unaware of the standards and policies of their
counterparts, let alone the content of their standardized assessments (Kirst,
17
Venezia, & Antonio, 2004). It is therefore unsurprising that students are
unaware of postsecondary expectations.
High school educators have a tendency to focus their energies on
helping students do well on high school state assessments. In higher
education, however, the focus is on admissions forward. There is little
connection between the two levels. The interviews conducted throughout the
Bridge Project exposed a pervasive absence of communication between
higher education and K-12 policymakers. This lack of communication
exasperates the problem of student confusion regarding expectations, as it
leads to mixed messages about what is important. Although some schools
are effectively addressing this problem through high school – college
articulation agreements, most students attend high school where
communication between their high school educators and potential colleges is
minimal, if it exists at all (Antonio & Bersola, 2004).
Researchers for the Bridge Project recommend three main actions
(Kirst, Venezia, & Antonio, 2004). First, steps should be taken to improve the
quality and accuracy of information available to students, parents, and
secondary educators. This recommendation necessitates improving the
communication and articulation across secondary and postsecondary
education systems. Second, attention should be shifted away from more
selective postsecondary institutions and focus on broad access institutions,
such as state schools and community colleges. Therefore, it is beneficial to
18
look at placement policies in the California Community College System.
Finally, existing policies and programs should be expanded, from just looking
at equal access to college, to looking at equal success in college. Therefore,
policies should be evaluated to ensure that they are promoting, rather than
interfering with, students’ success.
With better coordination among the levels, student knowledge about
what is necessary to be prepared for college would improve. High school
standards would be related to college placement and admissions
expectations. Tests students take during high school would be utilized as
benchmarks towards meeting college readiness goals. Teachers would have
the necessary information to help their students understand what is expected
of them by the time they enter college and their progress towards meeting
that goal. As a first step towards meeting these conditions, and thereby
improving the quality of information to available to students, the Bridge
Project researchers recommend that we “examine the relationship between
the content of postsecondary education placement exams and K-12 exit-level
standards and assessments to determine if more compatibility is necessary
and possible” (Kirst, Venezia, & Antonio, 2004, p. 308). They note that if
high school assessments are designed with appropriate college ready
standards, and placement exams are in congruence with the same
standards, students will get clear signals on academic deficiencies while
there is still time to do something about them.
19
The California State University system has taken a step in the right
direction with its Early Assessment Program (Olson, 2006). Juniors in high
school may take an augmented form of the California Standards Test that, if
passed, takes the place of the tradiational English and math placement tests.
Students scoring lower on this test have the opportunity to make up
deficiencies during their senior year (Kirst, 2004b). The mathematics portion
of this exam, however, is only available to the small percentage of students
taking the Algebra II or the Summative Math Test. Furthermore, it does not
take the place of mandatory placement exams at the two-year college level.
It would be worthwhile to explore whether a similar solution could be helpful
for community college bound students.
The Bridge Project brought to light many of the barriers that interfere
with the transitions from K-12 education to college. The current system
sends mixed signals as to what it is important for students to know. Policies
across the nation are disconnected and promote confusion and inadequate
preparation for college. One method of addressing this confusion is to look
at the K-12 assessments and college placement exams, to determine if the
assessments students take in high school can serve as benchmarks towards
progress in attaining college readiness expectations. This will help to
determine the degree to which expectations in California are aligned across
levels.
20
Standards for Success
Standards for Success (Conley, 2005a) was a research project,
sponsored by both the Association of American Universities and the Pew
Charitable Trusts, that attempted to address the mixed signals students
receive regarding what colleges expect students to know before they
commence college level coursework. First, college-ready standards were
explicitly defined as the Knowledge and Skills for University Success
(KSUS). Then, steps were taken to assess whether state standardized
assessments adequately measure progress towards meeting these college-
readiness standards. Since most high school graduates immediately attend
college or another form of postsecondary training, K-12 standards for college
preparatory courses should prepare students for and assess students on
these skills. Therefore, it is hoped that the KSUS standards will help to
improve the connection between what is expected of students in high school
and what they need for college.
Once the KSUS objectives were developed, a study was conducted to
determine how well the high school standards and assessments of 20 states
align with these goals (Conley & Venezia, 2003). An item-by-item analysis
was conducted by trained raters on 66 English and mathematics
assessments. These raters consisted of university educators who teach
freshman-level courses and high school teachers. The results of these
analyses were combined to assign alignment ratings for each test on
21
individual KSUS standards categories as well as to determine a rating for the
assessment as a whole. Assessments were determined to be well-aligned,
inconsistently aligned, or not well aligned. Overall, states fared poorly in their
alignment in such cognitive skills as critical thinking and mathematical
reasoning. The results clearly illustrate that state standards tests are not
designed to measure college readiness standards.
Conley recommended several steps to help states develop secondary
standards that align more appropriately with university expectations and can
be used to benchmark student progress towards college readiness. First,
states should perform content analyses to ensure that their K-12 standards
align well with university expectations. Second, states should compare
student performance on exams with measures of postsecondary success.
Third, states should consider modifying their K-12 assessments to measure
more complex cognitive tasks. Finally, educational leaders should work
towards increasing the alignment of K-12 assessment systems with the
policies for college admissions and placement (Conley, 2003a).
The Standards for Success project found a marked lack of alignment
between secondary high-stakes assessments and university expectations.
Therefore, the messages that students and their teachers receive from these
assessments is different from what university educators expect students to
have learned by the time they enter the doors of their institutions. The KSUS
standards explicitly outline what many universities expect of their students for
22
entry-level college coursework. These standards can be used to address
some of the disconnect between K-12 and university standards, if states take
steps to align their standards and assessments with these expectations.
Achieve and the American Diploma Project
Achieve and the members of the American Diploma Project took a
different approach to looking at the disconnect between high school and
postsecondary success. They took steps to identify what students need to
know in order to be successful in either college or the workplace. They found
that the skills were quite similar, and therefore preparing students for the
workforce is essentially the same as preparing them for college (Achieve
Inc., 1997, 2002). However, there is an “expectations gap” between what is
required of students in high school and what they are expected to be able to
do once they graduate (Achieve Inc., 2007a, 2007b). To close this gap, they
focus on strategies to improve college and workforce readiness.
Researchers found that the expectations held by employers and
postsecondary educators were fundamentally the same (Achieve Inc., 2004).
They developed a core set of standards that reflect these expectations.
Achieve researchers utilized these standards to determine how well the
admissions and placement tests students are required to take for college
align with these college and workplace readiness standards (Achieve Inc.,
2007a). Researchers found a gap between these expectations and what
23
students are actually required to learn in high school, which they defined as
the expectations gap.
Some progress has been made towards closing the expectations gap.
As of 2007, the graduation requirements of thirteen states compel students to
complete a curriculum designed to prepare them for college and work. Nine
states assess students on their progress towards meeting college readiness
standards and twelve states, including California, claim that their standards
align with the expectations of college and work (Achieve Inc., 2007b). Since
this report is based upon self-report measures rather than an analysis of the
expectations across levels, steps should be made to validate this claim.
Achieve found that expectations held by postsecondary faculty are
consistent, yet not well advertised. Postsecondary education systems can
help to close the expectations gap through the explicit definition of college
readiness standards. Furthermore, they need to verify that the admissions
and placement tests that they utilize assess students on these standards and
not arbitrary concepts. In addition, postsecondary institutions could
collaborate with their corresponding K-12 educational systems to design high
school tests that measure progress towards meeting the defined college
readiness standards. These tests can then be used for placement decisions,
no matter what level of postsecondary training students need. However, if K-
12 tests are used in this manner, they should be evaluated for predictive
validity.
24
California’s use of the 11
th
grade tests to exempt students from
placement exams in the California State University (CSU) system is a
positive step toward closing the expectations gap. It serves as a great
indicator of college level readiness for the small percentage who participate.
However, the program is optional and intended only for students entering the
CSU system. It is not useful for students who are community college bound;
nor, for students who have not taken Algebra II by their junior year.
Summary
Stanford’s Bridge Project, Standards for Success, and the American
Diploma Project have all helped to demonstrate how the pervasive lack of
alignment between secondary and postsecondary standards creates barriers
to achievement in the transition from high school to college. These barriers
disproportionately affect poor and minority students. As a step in addressing
these barriers, researchers for these projects have suggested that the
alignment of assessments across the K-12 and postsecondary education
systems be explored.
Since most students begin their college education immediately upon
graduation from high school, it is reasonable to expect the courses students
take in high school to prepare students to successfully place into and
complete college level courses. Ensuring that these exams and other
25
measures of secondary and postsecondary expectations are well-aligned is
important in providing consistent expectations across the levels.
The Context of Placement Testing in California
To understand the degree to which expectations are consistent
throughout California’s public education system, it is important to understand
the context in which placement testing occurs. Therefore, the structure of
California’s public education system will be outlined. This will include an
overview of the secondary and postsecondary assessment systems as well
as an introduction to remedial education in California. Finally, the importance
of placement test validity will be addressed.
The Structure of California’s Public Education System
California’s educational system consists of three distinct higher
education systems in addition to the K-12 system of education. These
include the University of California System (UC), The California State
University System (CSU), and the California Community College System
(CCC). These systems are fairly independent of one another; however, the
California Postsecondary Education Commission (CPEC) aids in the
coordination between these systems and reviews policies that impact all
three (UC Office of the President, 2007).
26
Although education has traditionally been a venue of local control, the
power has been steadily shifting to one of greater state control (Kirst,
Hayward, & Fuller, 2000). The stakeholders at each level are varied, with
many interested parties competing with the government at the local, state,
and federal levels for influence over policy decisions (Conley, 2003c).
Although there has been traditionally little cohesion across the levels,
stakeholders are beginning to work together to improve communication and
consistency.
UC and CSU Systems. California has two levels of four year
institutions, forming the UC and CSU systems which together constitute the
largest four-year university system in the United States (Kirst & Venezia,
2001). The UC system guarantees admission to the top eighth of California’s
graduating class and is responsible for all law and medical graduate training
as well as most doctoral programs. The CSU system admits the top third of
California’s graduating class and is primarily responsible for education
through the master’s level. Both systems give priority to transfer students
from California Community Colleges in the admissions process. In fact, the
universities are designed around a transfer model and offer more upper than
lower division courses (UC Office of the President, 2007).
Community College System. With over 2.5 million students and well
over 100 colleges, the California Community College System forms the
largest postsecondary system in the world. It has a primary mission of
27
providing education to anyone whom can benefit from it. This includes
remedial and vocational education in addition to academic courses designed
for transfer to four-year programs. The colleges are primarily controlled
locally through a system of 72 districts, but are overseen by the Board of
Governors of the California Community Colleges (California Community
Colleges, 2006; CCC Chancellor's Office, 2007; Kirst & Venezia, 2001; UC
Office of the President, 2007).
Access is important to the California Community College System;
nevertheless, recently the CCC system has been increasing its attention on
other measures of success. In its strategic plan (California Community
Colleges, 2006), the CCC system has developed many goals to improve the
opportunities of those in the communities it serves, going beyond simply
removing access barriers. They are taking steps to promote the college
readiness of incoming students, looking at both basic skills programs and
placement and assessment policies. Specifically, they are interested in ways
that articulation can be improved between the CCC system and its K-12
counterparts. Improving the alignment between K-12 and CCC expectations,
in the form of standards, curriculum and assessments, is a particular goal for
the California Community College System.
K-12. California’s K-12 system is overseen by an elected official, the
Superintendent of Public Instruction, and an appointed State Board of
Education. In addition, the Governor appoints a Secretary of Education, a
28
member of his cabinet. The Governor has veto power over policy decisions,
which gives him a significant amount of power over public education.
Locally, K-12 education is administered through county and district
superintendents and boards of education. Traditionally, these local bodies
have had a great deal of control over policies and their implementation; but,
this power has been weakening as the state has been assuming a greater
portion of the control and responsibility, particularly over the last quarter
century (Alpert, Alquist, & Strom-Martin, 2002; Hill, 1999). Much of these
changes have occurred as the state takes a greater interest in improving
student achievement.
K-16 Articulation
There has been traditionally very little articulation across education
levels in California. Postsecondary and K-12 policymakers tend to be
centered on their own levels and have been less concerned with what is
occurring at other levels. The differing missions and disconnected curricula
and assessments are inefficient in providing equal opportunities throughout
various education levels across the state (Bueschel, 2004). However, there
has been increasing pressure on educational systems to better align
expectations within and across systems (Conley, 2005b), and stakeholders
from many directions are calling for P-16 reform.
29
The Superintendent of California’s K-12 educational system has
established a new P-16 council that promises to improve the coordination
across levels to improve education for students at all levels. Council goals
include addressing achievement gaps and improving the articulation across
systems to better prepare students for college (Jung, 2007). Policy
alignment across systems is not new, and is “relatively common outside the
United States” (Conley, 2005b, p. 121). Developing a coordinated vision and
better communication is an important first step in creating a system that is
well articulated and responsive to the educational needs of the individuals in
California’s ever changing society (Alpert, Alquist, & Strom-Martin, 2002).
This better aligned system will help to provide students with consistent
signals and information.
Curriculum. In California, students desiring to enroll directly in a four-
year program are expected to complete a college preparatory curriculum.
This is defined by the a-g requirements for university admission, which
include four years of English and three years of mathematics through
Algebra II (Education Trust-West, 2004). California’s high school graduation
requirements, in contrast, only require three years of English and two years
of mathematics through Algebra I (Editorial Projects in Education Research
Center, 2007).
Only a small minority of California students graduate having
completed the necessary courses to enroll in a four-year university (Ali,
30
2002), and this proportion has been dropping over time (California
Postsecondary Education Commission, 2005). Most students follow a more
general course of study (Ali, 2002; Planty, Bozick, & Ingels, 2006). Therefore,
for most recent high school graduates, access to a four year degree
necessarily begins at the community college level.
Assessments in California
High School Standards and Assessments. K-12 content standards
were adopted in California beginning in the late 1990’s with the purpose of
improving student achievement through making explicit the knowledge and
skills that students should acquire for each grade level and course (California
State Board of Education, 2007). Students are given annual assessments to
assess proficiency on these standards in core courses such as English,
mathematics, social studies, and science. Schools are held accountable for
the results of these assessments. In addition to these content tests, high
school students are expected to take and pass an examination in English
and mathematics in order to graduate from high school. However, as the
graduation tests assess middle school and freshmen level standards (Callan,
Finney, Kirst, Usdan, & Venezia, 2006), they don’t do a good job of
measuring the knowledge and skills that students should have acquired by
the time they have completed the 12
th
grade.
31
The high school standards can not be expected to naturally align with
university expectations, as they were not designed with the express purpose
of preparing students for college (Conley, 2005b). However, teachers tend to
concentrate on those standards that appear most often on the tests, as
schools are held accountable for student results on these assessments
(Herman, Webb, & Zuniga, 2007). In California, these standardized
assessments are fairly well aligned with the state standards (Bueschel,
2004), so teachers are essentially concentrating on the key standards for any
given content area. However, most postsecondary instructors have little faith
in the ability of the state standards to prepare their incoming students for
college-level work (ACT Inc., 2007).
Postsecondary Standards and Assessments. Most colleges and
universities require students to take placement exams in order to take
college level coursework (Perin, 2006). Due to the fact that community
colleges do not have explicit entrance standards, these placement tests
serve as “de facto” standards for college level work; standards that are set at
a higher level than those signaled by high school graduation requirements
(Venezia, Kirst, & Antonio, 2003). However, students generally become
aware of this fact after it is too late, when they are surprised by placing into
non-credit bearing, remedial courses (Bueschel, 2004).
There is a wide variety of assessments used for placement purposes
and most students are unaware of the existence of these exams, let alone
32
their content (Bueschel, 2004). One exception is CSU’s Early Assessment
program; students are given the opportunity to take an augmented form of
the K-12 state assessments at the end of their junior year. Students who do
well are exempted from taking placement tests, provided they attend a
university in the CSU system. In addition, as they obtain their scores in
August, they receive diagnostic information on their level of college
preparation while there is still time to do something about it (Callan, Finney,
Kirst, Usdan, & Venezia, 2006). Unfortunately, this program is unavailable
for UC and community college bound students.
Remediation in California
In California, students are placed in remedial education at alarming
rates. More than half of all students who enter the California State University
System (CSU) are placed in some form of remediation (Moses, 1999), even
though the CSU system enrolls only the top third of graduating seniors. In
2006, 37.5% of first-time students entering the CSU system were placed in
remedial mathematics, despite the fact that the mean high school GPA of
these students was 3.15 ("Fall 2006 final regularly admitted first-time
freshmen remediation systemwide"). At the community college level the
remediation rates are even higher with 60% of all entering students being
placed in remedial courses (Kirst, 2004a).
33
Inequities in Remediation. For the fall of 2006, CSU bound African
American and Latino students were placed in English and mathematics
remedial courses at a rate of more than twice that of White, Non-Latino
students ("Fall 2006 final regularly admitted first-time freshmen remediation
systemwide"). Approximately half of all community college students who
enroll in these courses are students of minority background (California
Community Colleges, 2006). These remedial placements keep students
from taking college-level coursework, which at minimum delays student
progress toward degree attainment.
Peter Riley Bahr (2004) explored racial inequities in the successful
completion of postsecondary remediation in mathematics. He followed a
cohort of remedial math students in the California Community College
System over a period of six years, to compare outcomes. He found that only
one quarter of the students placed in remedial math were able to complete
their remediation program successfully, as defined by taking college-level
math. Furthermore, he found that the likelihood of White students eventually
completing a college-level math course was 136% greater than Black
students and 42% greater than Hispanic students.
The ethic differences in remedial outcomes that were observed can be
partly explained by the initial course level taken by students. Black and
Hispanic students are more likely to enroll in the lowest-levels of remediation,
such as arithmetic, whereas White students are more likely to enroll in
34
classes closer to college level mathematics, such as intermediate algebra or
geometry. This is significant due to the fact that these differing course levels
represent a difference of 726% in the probability of remediating successfully.
Due to the differential outcomes attributable to course placement, it is
important to ensure that only students who can benefit from basic skills
education are placed in remedial courses. Improperly placing students in
basic skills courses is unfair and disheartening. Individuals who are placed
in these courses often feel stigmatized and the time that it takes students to
progress through these courses impedes their degree progress (Academic
Senate for California Community Colleges, 2004).
Validity of Placement Tests and Policies
Legal regulations in California require institutions to provide validity
evidence for tests used to make placement decisions (Bryant, 2001). College
placement tests that lack validity are unfair to students and fiscally
irresponsible. Budget cuts across California have reduced funding for
community colleges resulting in reduced course offerings (Research and
Planning Group for California Community Colleges, 2005). Ensuring that
students are not needlessly placed in remedial courses could potential free a
great deal of funding for other purposes.
It is important to note that validity is established for a specific use of a
test and is not an element of the test itself (Koretz & Hamilton, 2006;
35
Shepard, 1993). It is possible for a test to be valid in one context and invalid
in another. In order to claim that a test is valid for a specific use, it is
necessary, but not sufficient, to establish that the content of the test is
suitable for the purpose of the test (Shepard, 1993).
Content Validity. Content validity measures the suitability of the
subject matter of the test for the intended use. The process of determining
content validity involves both determining whether the content domain that
the test purports to assess is appropriate for the intended use and that the
test adequately samples from that domain (Shepard, 1993). One of the
assumptions of placement testing is that skills developed in prerequisite
courses are necessary for later courses (Koretz & Hamilton, 2006).
Therefore, the California Community College Assessment Association
recommends measuring content validity through considering both the extent
to which pre-course skills are measured by a given instrument and the extent
to which measured skills are not pre-course expectations (California
Community College Assessment Association, 2001). Since the pre-course
expectations for entry level college courses are the high school classes
students take to prepare for college, it is reasonable to compare college
placement tests with high school standards as a measure of content validity.
Placement Test Validation in the CCC. In its strategic plan, the
California Community College System has outlined numerous goals,
including improving the effectiveness of student placement and the
36
assessment of student preparation (California Community Colleges, 2006).
Lack of funding has prevented many California colleges from beginning and
completing the validation process (Academic Senate for California
Community Colleges, 2004). In a study conducted for the San Diego
Community College District, the Mathematics Diagnostic Testing Program
(MDTP) demonstrated sufficient predictive validity, using grades in the
subsequent math course as evidence (Armstrong, 1994). However, the
study also found that the placement process had a negative, disproportionate
impact for younger, non-white students. It is important to investigate the
factors that contribute to this trend.
The potential that students could have been misplaced into lower level
courses, when they have the capability of succeeding in higher level courses
can be partially addressed by investigating the alignment between the
secondary and postsecondary expectations. Internal reports have estimated
that approximately one-quarter of community college students may be
misplaced by the policies currently in effect (Academic Senate for California
Community Colleges, 2004). Furthermore, recent data from the California
Partnership for Achieving Student Success suggests that most students
transfer down at least one level of math as they transition from high school to
the community college level (Willett, 2007). This means that students are
often placed in courses lower than those that they have already successfully
completed. Therefore, it is important to determine the degree to which this
37
effect is due to misaligned expectations between the secondary and
postsecondary education systems, rather than the students not learning high
school content.
Measuring Alignment
One of the most significant issues that influences the transition from
high school to college is the concept of alignment. There appears to be a
marked lack of alignment between the K-12 and postsecondary systems
nationwide. Measuring the alignment that already exists is an important step
in addressing the problems that arise from a misaligned system.
What is Alignment?
When considering the transition from high school to college, alignment
is a measurement of the relationship between different components of an
educational system. In the field of measurement, it is often defined as the
extent to which expectations in the form of standards match the assessments
intended to measure them. Alignment is a measure of the relationship
between the system components rather than an attribute of any particular
element within the system (N. L. Webb, 2002). The important distinction is
that alignment indicates how well the parts of the systems work together to
guide student learning and expectations. Two important types of alignment
38
are important in the transition form high school to college, vertical and
horizontal alignment.
Vertical Alignment. Vertical alignment refers to the alignment between
what is expected in differing levels in the educational pipeline. It is an
indicator of how well expectations are articulated, as students progress from
one educational level to the next. These expectations can take many forms
including course content, standards and exams. A system that is vertically
well aligned vertically provides students with clear and consistent signals
regarding what is important for them to learn as they progress through the
educational system. They can be confident that the knowledge they acquire
at one level of education will prepare them for the next.
A vertically aligned system is logically structured. Each component
has a purpose that prepares students for the next level. The disconnect
between high school and college suggests that there are some vertical
alignment issues inherent in this transition. In fact, although alignment
between secondary and postsecondary systems is common in other
countries, it is quite rare in the United States (Conley, 2003c). Addressing
the high school-college disconnect necessitates looking at issues of vertical
alignment. However, if assessments, courses, and standards are to achieve
a strong degree of vertical alignment, there must first be some measure of
consistency within each level (Martineau, Paek, Keene, & Hirsch, 2007), a
concept that is termed horizontal alignment.
39
Horizontal Alignment. Horizontal alignment refers to alignment within
a specific level of expectations. This can take many forms, such as the
alignment between curricula, courses, exams, exam blueprints, and
standards. In the testing field, the basic goal is to determine the degree to
which the exams measure the same content, at the same level, as the
associated standards. In the Standards for Educational and Psychological
Testing (American Educational Research Association, American
Psychological Association, & National Council on Measurement in Education,
1999), alignment is mentioned as an important component to finding and
reducing testing bias. In Standard 13.3, for instance, test users are
cautioned to find evidence for the extent to which the test represents
curricular standards. The Webb alignment methodology, which will be
discussed shortly, was designed to measure the horizontal alignment
between standards and exams.
Consequences of Misalignment. A system that lacks horizontal or
vertical alignment sends unclear and inconsistent signals regarding what is
important for students to learn in preparation for postsecondary success. It
makes tracking student progress through the system difficult. Insufficient
alignment makes issues such as poor student performance, achievement
gaps, high dropout rates, and high remediation rates more difficult to
diagnose. Therefore, investigating the alignment among components in the
40
educational system is an important step in addressing the widespread
underpreparation of California students for college level work.
Lack of alignment confuses students about what knowledge and skills
are necessary and important. The lack of measures of college readiness and
the differing expectations between the secondary and postsecondary
education systems lead to confusing messages regarding what will be
expected of students in college (Rodriguez, 1995). Without the alignment
between standards, exams, and expectations, students do not receive clear
and consistent information regarding their progress toward meeting college
readiness expectations (N. M. Webb, Herman, & Webb, 2007). Evidence
suggests that, on a national basis, secondary assessments do indeed lack
the necessary alignment required to provide students with this vital feedback
(Conley, 2003a).
Learning is more difficult to measure in a misaligned system (Baker,
2004). It is quiet possible that capable students are misplaced into remedial
courses because they have not had the opportunity to learn the content they
are expected to demonstrate on the placement exams. This generates
questions regarding the validity of any placement decisions made upon these
potentially misaligned tests.
Improving Alignment. Many suggestions have been made on
improving the alignment between high school and college, most of which will
not be discussed here. Researchers for the Bridge Project made a few
41
suggestions on improving alignment through addressing admissions and
placement policies. One of these recommendations is to ensure that
placement exams used in colleges are aligned with the standards used to
prepare students for college. Preferably this could be achieved through the
implementation of a systemwide set of exams benchmarked on explicit K-16
and university standards (Kirst, 1998). These aligned standards and
assessments can improve the quality of signals and incentives provided to
students as they prepare for college (Venezia, Kirst, & Antonio, 2003). As a
first step in addressing these suggestions, the degree to which the currently
utilized college placement tests align with high school standards should be
explored.
System alignment is an important part of the transition from high
school to college. For the system to be adequately aligned vertically, there
must be some degree of horizontal alignment between the high school exit
level and college entrance level expectations. One way of measuring the
degree of this alignment is to examine the relationship between the high
school standards tests and college placement exams.
Alignment Models
Alignment between standards and exams has been measured in
various ways in the past, ranging from the simple to the complex. A thorough
literature review of these methods has been completed which summarizes
42
and critiques the most commonly used models (Impara, 2001) and will not be
reproduced here. Rather, a brief introduction to alignment options will be
followed by an overview of the Webb alignment procedure that is increasingly
the preferred method.
Simple Alignment Models. There are many ways in which alignment
has been measured in the past, with some of the simplest, and least robust
methods being a verification on the surface that similar content is being
measured. In this category belongs alignment that is determined by
matching items with a blueprint during test construction. The Achieve
(2007b) study provides a second example of simple alignment, in which
California reported that they have aligned their standards with postsecondary
expectations by involving postsecondary leaders in the development of the
state standards. However, most simple alignment procedures are a little
more robust than these examples. Most procedures involve, at minimum,
using raters to verify the content match between standards and exams.
While content consistency is important, it is not enough. For
alignment to truly be verified, matched items should measure content at the
same level of complexity. For example, two items could measure solving for
an unknown variable and hence match in content. However, the complexity
of the problem depends on the complexity of the equation and the way it is
presented. There is a big difference between solving a one-step equation
and solving one containing radicals. The cognitive difficulty can also be
43
varied by whether the problem is multiple choice or requires a student-
produced response. One allows students to plug in the choices; the other
forces them to solve the problem.
La Marca (2001), recognizing the importance of the complexity of
items, uses a second criteria, depth. He emphasizes that alignment involves
matching test items to objectives both in terms of content and cognitive
complexity. It should be mentioned that La Marca is aware that content and
depth are not the only significant measures of alignment. Tests should also
be aligned with regard to how much emphasis is placed on each of the
objectives and what type of performance is required, while remaining
accessible to students of all ability levels (La Marca, Redfield, & Winter,
2000). Nevertheless, he focuses on alignment as a construct of “content
representativeness”, as it relates to content validity, and therefore
emphasizes content and depth as the fundamental aspects to explore.
This short introduction does not begin to explore all the methodologies
that have been developed to explore testing alignment. The methods often
vary with the purpose for which the alignment procedures were designed.
For instance, one methodology was designed to look in depth at individual
items rather than entire tests or systems (Kulm, Dager Wilson, & Kitchen,
2005). Others quantify the alignment of content instruction with standards or
other curricular frameworks (Blank, 2002; Blank, Porter, & Smithson, 2001;
Porter, Smithson, Blank, & Zeidner, 2007). However, one model developed
44
by Norman Webb has been found to be more comprehensive and therefore
adaptable for many contexts (Impara, 2001).
Webb alignment model. Webb’s model is a measure of content
alignment that focuses on four components categorical concurrence, depth of
knowledge consistency, range of knowledge, and balance of representation
(N. L. Webb, 1997, 1999, 2002, 2007; N. M. Webb, Herman, & Webb, 2007).
Categorical concurrence refers to how well items match in content. Depth of
knowledge indicates whether that match occurs at the same level of
complexity, which Webb defines as recall, skill/concept, strategic thinking
and extended thinking. Range of knowledge measures the portion of
objectives that are measured by at least one test item. Whereas, balance of
representation refers to how well the items are balanced among the available
objectives. Sometimes a fifth criteria, source of challenge, is also considered
in the analysis (N. L. Webb, 2002). This is used to identify items for which
characteristics unrelated to the underlying standard make the item more
difficult.
An alignment exercise is conducted using trained reviewers (Herman,
Webb, & Zuniga, 2007; N. M. Webb, Herman, & Webb, 2007) to identify the
content match and depth of knowledge of standards and test items. A matrix
is thereby formed that can be utilized to analyze the range and balance of
standards measured on the test. In addition to reporting on the four
alignment criteria, rater reliability is provided as part of the analysis.
45
Previous Alignment Studies
Despite the relatively recent development of these alignment
methodologies, several important studies have been conducted on alignment
that are important to mention when considering the transition from high
school to college.
Webb’s Studies. In two early alignment exercises, Norman Webb
(1999, 2002) conducted workshops to solidify the alignment techniques he
had been developing. One of Webb’s primary purposes was to evaluate and
obtain feedback on the alignment process itself. In the 1999 study, raters
expressed frustration regarding the instruction to refrain from judging the
quality of the standards or exams. They also had some difficulty with the
process of determining depth of knowledge levels. Webb therefore
recommends that future alignment studies provide more training for the
reviewers and the means for raters to comment on the quality of standards
and assessment items they are reviewing.
In the 2002 study, Webb reported on several issues related to the
alignment in general. The first issue involves situations in which an item
matches multiple objectives. Should multiple matches be allowed? If so,
how many? A second issue is arises when the assessment is only designed
to measure part of the standards. Should the standards that are not included
be excluded from the alignment exercises? Or, should the lack of inclusion
be emphasized as part of the alignment analysis? Assessments are often
46
designed to measure a range of standards, or conversely standards are
designed for a range of grade levels where only part of that range needs to
be measured by a particular set of assessments. This can lead to difficultly
when assigning depth of knowledge levels. Finally, it is very common for
some objectives within a standard, especially those that are easier to test, to
be overemphasized. Resolving these issues is partially subjective; it
depends on the purposes of the tests and the educational values of those
running the assessment systems. When conducting alignment exercises,
these issues should be considered and resolved beforehand.
California K-12 and College Assessments. Le, Hamilton, and Robyn
(2000) conducted an analysis on the alignment of K-12 assessments, college
admissions tests and postsecondary placement exams for six states,
including California. They found large amounts of misalignment in English
and math content and in how skills were measured. For example, the SAT
and ACT are both designed with the same purpose, they are both
admissions tests and are even used interchangeably by many universities.
However, they measure mathematics achievement quite differently. The
ACT requires the extensive use of memorized formulas while the SAT
provides these formulas and emphasizes abstract reasoning. Therefore,
even when measuring the same content, items from these tests are vary
different in the type of cognitive demand they place on students.
47
The analysis conducted by Le, Hamilton, and Robyn had discouraging
results. There was very little evidence of alignment between tests across
educational levels. Furthermore, the various tests analyzed were structurally
different, so that individuals preparing for one or another of the tests would
need to utilize varying preparation techniques, which greatly influences
student performance. This study seems to indicate that there is a marked
lack of alignment in California between exams taken at the high school exit
and college entrance levels. However, it should be noted that several of the
exams that were analyzed are either no longer in use, or exist today in a
markedly different forms. Therefore, it would be beneficial to repeat this
study with contemporary assessments.
The Golden State Exam. Herman, Webb and Zuniga (2003)
compared the mathematics portion of California’s Golden State Exam with
the University of California Statement on Competencies in Mathematics
Expected of Entering College Students (see University of California
Academic Senate, 1997). These exams were abolished in 2003 (see Flores,
2003; Sigman, 2006). However, as they were based on California standards
for Algebra I & II, Geometry, and Probability and Statistics, this alignment
exercise is still informative to some degree regarding the alignment of state
standards with postsecondary expectations in mathematics. The purpose
was to determine the degree of alignment of this test with university
expectations, for potential use in admissions. This helps to provide
48
information regarding how well the content students learn in high school
aligns with university expectations. All items on the exam, except one, were
able to be matched to the Statement on Competencies. Conversely, slightly
less than half of the topics on the Statement on Competencies were
addressed on the test. Variables, Equations, and Algebraic Expressions
received the most emphasis, with 75% of the objectives corresponding to a
test item. None of the items on the test addressed the category of
Argumentation and Proof.
Knowledge and Skills for University Success. Concerned over the
messages students receive about what is important from high school
assessments and its impact on college preparation, David Conley (2003a)
investigated the alignment of various state standardized assessments with
the Knowledge and Skills for University Success (KSUS) objectives. He
found that, in general, the distribution of items across KSUS objectives was
weak. However, nearly 70% of the items that matched did so with an
appropriate level of cognitive complexity. Although this looks promising, it
should be noted that, in mathematics, range of knowledge was particularly
weak, with less than one-third of the objectives being addressed.
In a similar study (Brown & Conley, 2007), the KSUS objectives were
compared with standardized assessments in twenty states. The authors
clarified that complete alignment was not expected, nor was it necessarily
required. The purpose of the study was to investigate the degree of
49
alignment between expectations at the two levels, given the expanding use
of high school level assessments for multiple purposes.
When averaged across all standards categorical congruence was
adequate, primarily due to the influence of the objectives of Reading
Comprehension in English, as well as Computation and Math Reasoning in
mathematics. High school math assessments met the depth of knowledge
criteria almost universally, excepting the area of Trigonometry. In English,
depth of knowledge aligned best with Reading and Comprehension and
Writing. Only 30% of English and 13% of mathematics assessments met the
criteria for range of knowledge. Balance of representation was adequate for
more than three quarters of all exams, meaning that where objectives and
standards matched, they did so with an appropriate level of balance. The
authors conclude that overall the assessments were not well aligned with
KSUS standards, and that this alignment was worse for mathematics than
English.
High School and Community College Assessments. Brown and Niemi
(2007) recognized the utility of determining the alignment between high
school assessments and the “de facto” standards for college level work, as
defined by community college placement tests. They utilized the augmented
forms of the California Standards Tests (CST) in English, Algebra II, and
Summative High School Mathematics. Students receiving a passing score
on the augmented CSTs can be exempted from taking placement exams at
50
universities in the California State University system, provided they continue
to meet other requirements. Therefore, these tests are utilized to identify
students who are ready for college level work. It is reasonable to assume,
therefore, that since they are designed to measure the same construct,
college readiness, that there might be at least some degree of alignment
between the augmented CSTs and the community college placement tests.
They began by defining the “de facto” standards for college level work,
the placement exam content. Utilizing the most commonly administered
placement exams throughout the California Community College (CCC)
system, they conducted a content analysis to identify the central objectives of
the CCC placement tests in reading, language arts and mathematics. The
content was determined through examining test blueprints and actual test
items for these examinations. These objectives were reduced to the core
content and then similar items were grouped under relevant category
headings (see Brown & Niemi, 2007).
Once the core objectives were identified an alignment workshop was
held, employing a modified form of the Webb alignment methodology to
investigate the alignment between the identified CCC objectives and the
augmented CSTs. The only change in methodology was that the first five
levels of Marzano’s scale were used to measure depth of knowledge
consistency in place of the one developed by Webb. Marzano’s levels were
designed to address problems with earlier classification techniques that did
51
not sequentially measure higher levels of cognition (see Marzano, 1993) and
are based on more updated understandings about cognitive processes
(Colwell, 2006; Kendall & Snyder, 2003; Marzano, 2001). Therefore,
Marzano’s scale is an appropriate measure for depth of knowledge
consistency.
The augmented English CST exhibited good categorical concurrence
with the CCC objectives. The augmented forms of the mathematics CSTs
displayed less. Some of the categories that were not well aligned were
whole numbers and fractions, decimals and percents and other topics that
are found in earlier math courses; however, categorical concurrence was
also found lacking in higher-level topics such as trigonometry and graphing.
As in the KSUS studies, the balance of representation was good where
matches occurred.
Overall, the tests were strongly aligned in English Language Arts.
Math, however, was well aligned only in depth of knowledge and balance of
representation. A possible explanation for this lack of alignment could be
that the concepts not measured on the augmented CSTs are standards that
measure skills much lower than or standards beyond the Algebra II test.
It should be noted that while all 11
th
grade students have the
opportunity to take the augmented form of the English CST, the augmented
Algebra II and Summative High School Math tests are only available to
students who have completed Algebra II or higher by the 11
th
grade. Many
52
more students take the Algebra 1, Geometry and General Mathematics tests.
To have a more complete picture of how well the California high school
standards prepare students to achieve a passing score on the placement
exams the alignment of these additional tests with CCC standards should
also be investigated. This will help to identify areas in which the consistency
between high school assessments and postsecondary expectations can be
improved.
53
CHAPTER 3
ALIGNMENT METHODOLOGY
How well aligned are the mandatory tests taken by California high
school students with the entrance expectations of California Community
Colleges? In the absence of formal entrance standards in the open-access
California Community College (CCC) System, the placement exam content
defines college-level expectations. To help provide a clear picture on how
consistent these expectations are with the high school curriculum, the
alignment between these “de facto” entrance standards and K-12 content
standards in mathematics was explored.
The investigation concentrated on the alignment of CCC placement
test content with the California Standards Tests in General Mathematics,
Algebra I, and Geometry. This study is similar to, and designed to expand
upon, the study conducted by Brown and Niemi which investigated the
alignment between the CCC placement exam content and the augmented
forms of the California Standards Tests (CST) in 11
th
grade English, Algebra
II, and Summative High School Mathematics (see Brown & Niemi, 2007). As
the authors indicate, the Algebra II and Summative High School Mathematics
tests are taken by only 18% of California high school students; whereas, the
math tests addressed in the current study are taken by 81.2%. Combining
the results of the two studies provides an indication of how well the
54
community college standards are aligned with the CST’s taken by 99.2% of
California students.
There are several methodologies available to investigate the
alignment between these exams. One approach, developed by Norman
Webb, has become widely accepted due to its complexity and adaptability
(Bhola, Impara, & Buckendahl, 2003; Impara, 2001). This study followed a
modified Webb approach of content focus alignment (see Brown & Conley,
2007; Brown & Niemi, 2007; N. L. Webb, 1997, 1999, 2002; N. L. Webb,
2007) to investigate the alignment of the “de facto” California Community
College entrance standards with the California K12 content standards in
mathematics.
Research Questions
This purpose of this study is to investigate the degree of alignment
between high school and college expectations. To that end, three research
questions were explored:
I. How well do the California Standards Tests (CST) in General
Mathematics, Algebra I, and Geometry align with the content of the
most commonly utilized math placement tests in the California
Community College (CCC) System?
II. Are there areas emphasized on the mathematics placement tests that
are not measured by the California Standards Tests?
55
III. To what degrees can performance on the CSTs measure progress
towards meeting the “de facto” college readiness standards in
mathematics, as defined by CCC placement test content?
Alignment Rating Process
Participants. Consistent with past alignment studies, 9 subject matter
experts from California high schools and colleges were recruited to serve as
raters for this alignment study. For the purposes of this study, a subject
matter expert is defined to be someone who is qualified to teach and has
experience teaching high school level or introductory college level math
courses. Raters were recruited from local high schools, community colleges
and universities. Each rater was familiar with the California Content
Standards in mathematics and/or the content in remedial or entry-level
college math classes. The alignment exercises took place within the context
of a one-day alignment workshop, where participants were compensated for
their time at the rate of $250.
Activities. The alignment workshop began with a training session.
First, the raters were introduced to the purposes of the study and important
definitions, including the concepts of categorical concurrence and depth of
knowledge. Second, raters were introduced to the 5-point scale being used
to quantify depth of knowledge. Raters practiced assigning depth of
knowledge levels to sample, training items. The purpose of this training was
56
to help reviewers develop a shared understanding of these levels. It also
helped calibrate the concepts of content match and depth of knowledge
coding, improving consistency between raters.
Once training was completed, the raters analyzed the CST test
elements in General Mathematics, Algebra 1, and Geometry and compared
them to the California Community College placement test objectives in two
steps. The first step was to determine the depth of knowledge assessed by
individual objectives on a 1-5 scale using Marzano’s taxonomy (Marzano,
2001; Marzano & Kendall, 2007). The depth of knowledge ratings for the
community college objectives had been previously determined with a high
degree of rater reliability (Brown & Niemi, 2007) and was not repeated here.
Second, each rater compared the content of the high school assessment
elements with the California Community College placement exam content to
determine if and where matches occurred. For each rater, this created three
matrices of CCC objectives by CST items; one each for General
Mathematics, Algebra I, and Geometry. Analyses were performed for each
rater and then averaged to determine the results on a per standard basis.
Rater Reliability. Due to the fact that this study employs multiple
raters, the reliability of ratings was assessed for rater agreement. This was
done utilizing the generalizability coefficient (see Brennan, 2001; Mushquash
& O'Connor, 2006; Shavelson & Webb, 1991, 2006; Shavelson, Webb, &
Rowley, 1989), a measure of reliability that, unlike the reliability coefficient of
57
classical test theory, distinguishes between systematic and unsystematic
sources of variability. This is quantified in the generalizability coefficient.
The coefficient was utilized to provide an estimate of the consistency
between raters.
Alignment Criteria
Brown and Niemi identified the CCC placement test content, through
the analysis of the most commonly utilized placement tests in the California
Community College System. These tests include the Accuplacer Arithmetic,
Elementary Algebra, and College Level Math tests, ACT Compass’
Numerical Skills/Pre-Algebra, Algebra, College Algebra, and Geometry tests,
and MDTP’s Algebra Readiness and Elementary Algebra tests. The content
objectives discovered were arranged into 12 major groupings (see Brown &
Niemi, 2007):
• Whole Numbers and Fractions,
• Decimals and Percents,
• Applications and Interpreting Tables/Graphs,
• Integers and Rationals,
• Algebraic Expressions and Operations,
• Operations with Exponents,
• Equations, Inequalities, and Word problems,
• Functions,
58
• Trigonometry,
• Geometry,
• Graphing and,
• Other Algebra Topics.
These community college standards were compared with the California
Standards Tests in General Mathematics, Algebra I, and Geometry. The
modified Webb alignment procedure that was followed was used to assess
the alignment between these exams in four dimensions: categorical
concurrence, depth of knowledge, range of knowledge, and balance of
coverage.
Categorical Concurrence. Categorical concurrence (CC) is a measure
of how well two assessments are matched in content. It helps to quantify the
extent to which the mandatory high school assessments measure the same
content as the assessments used for community college placement. A test
item and an objective show categorical concurrence if they address the same
topic. Raters assigned each test item to one or more community college
objectives, or to no match. Categorical concurrence is analyzed on a per
standard basis, with six assessment items mapping into the objectives for
that standard indicating that categorical concurrence has been achieved.
Each standard either meets or fails to meet this criteria. This metric does not
allow for partial achievement.
59
Table 1 provides example content matches within a generic standard,
for illustrative purposes. The letters A through E represent all the objectives
within the given standard. Test items are numbered 1 through 5. Each
check mark represents a hit, a test item measuring the same content as the
associated objective. This example meets the criteria for Categorical
Concurrence because there are a total of 8 hits, which is greater than or
equal to 6.
TABLE 1: Example Standard Content Matches
Letters represent objectives within the standard and
numbers represent test items.
1 2 3 4 5
A
B
C
D
E
It is possible for categorical concurrence to have been achieved with
all items mapping onto the same objective and/or with questions that do not
match the cognitive complexity of the standard. For this reason, each
standard was also analyzed to determine whether the cognitive levels
assessed by the items are comparable to the associated objectives and to
determine the ratio of objectives addressed within the standard.
Depth of Knowledge Consistency. Depth of knowledge consistency
(DOK) measures cognitive complexity on a 1 - 5 scale using Marzano’s
definitions of levels of cognition. These include Retrieval, Comprehension,
Analysis, Knowledge Utilization and Metacognitive Processes. DOK
60
measures the degree to which placement exams and high school tests
assess matched content to a similar degree of complexity, quantified by a
value from 0 to 1.0. This indicates the degree to which California high school
tests measure content at higher or lower levels of complexity compared to
community college placement exams. An item matches an objective on
depth of knowledge if it has been assigned a complexity rating equal to or
greater than the associated objective. The depth of knowledge criteria is
determined to have been met if at least half of the items matched to a given
standard are tested at a level of complexity at or above that of the standard.
Allowances are made for standards in which 40-50% of the items meet the
required benchmark. In this case, the DOK criterion is determined to have
been weakly met.
Table 2 provides example depth of knowledge ratings for illustrative
purposes. Within this example standard, 7 out of 8 test items measure
content at a cognitive level at least as great as the associated objective. This
yields a DOK level of 7/8, approximately .88, which meets the minimum DOK
value.
TABLE 2: Example DOK Ratings
Is the cognitive complexity of the test item greater than or equal to the objective?
Test Item # (DOK Rating)
Objective (DOK Rating) 1(2) 2(4) 3(3) 4(4) 5(2)
A(3) yes yes yes
B(2)
C(1) yes yes
D(3) yes
E(3) yes no
61
Range of Knowledge. Range of knowledge (ROK) refers to the
number of objectives tested by items mapped into a given standard. This
measure indicates whether the span of knowledge measured by high school
standards tests and community college placement exams are comparable,
with a range of 0 to 1.0. The higher the percent of objectives assessed, the
higher the range of knowledge value assigned to that test. The minimum
criteria is that at least half of the objectives within the standard have at least
one question mapped to them, reflected in a desired ROK value greater than
.5. A score between .4 and .5 indicates that the ROK criterion has been
weakly met.
In order to illustrate the calculation of range of knowledge, please refer
once again to Table 1 on page 59. Notice how 4 out of 5 of the objectives, A,
C, D, and E, are assessed by at least one test item. Therefore, the ROK
value is calculated as 4/5, .80. This standard exceeds the minimum ROK
value of .50.
Balance of Representation. The DOK and ROK criteria allow an exam
to have a large number of questions clustered around a few objectives. In
order to determine if the questions are evenly spread among the various
objectives within a standard, the balance of representation index is
computed. Balance of representation (BR) refers to the distribution of
assessment items among the various objectives. It quantifies the overall
balance of items, ranging from 0 to 1.0. The formula used to determine this
62
is
!
1"
1
O
"
I(k)
H
k=1
O
#
$
%
&
'
(
)
2
, where
!
O refers to the number of objectives earning a
hit within a given standard,
!
H is the overall “number of items hit for the
standard” and
!
I(k) stands for the total of items hit for each objective
!
k (N.
L. Webb, 2007, p. 14). A specific assessment is high in balance of
representation if the assessment items are distributed evenly across the
given objectives. The desired level of balance is .70 or higher, allowing
standards to be determined to have weakly met this criteria if the balance is
measured between .60 and .69.
To calculate the balance of representation we refer once again to
Table 1. Notice that there are four objectives with at least one hit, therefore
!
O= 4. Since there are eight hits total for all objectives,
!
H = 8.
!
I(k) is
uniquely determined for each objective based on the associated number of
hits. For instance,
!
I(A)= 3 because there are three matched test items for
objective A. Likewise,
!
I(B)= 0 because there are no matches. Substituting
these values into our equation, we obtain a value above the minimum of .70:
!
1"
1
4
"
3
8
+
1
4
"
0
8
+
1
4
"
2
8
+
1
4
"
1
8
+
1
4
"
2
8
#
$
%
&
'
(
/2 = .75.
Analysis
Each rater’s examination of CST test elements creates an individual
matrix of CCC objectives by CST items. This matrix provides information
63
about content match and DOK level. Each matrix was analyzed separately
to determine the resultant values for categorical concurrence, depth of
knowledge, range of knowledge and balance of representation. These
results were averaged to determine the final results by standard and by
assessment. Rater reliability was analyzed using the g-coefficient.
Analyzing the data by standard allows for more than merely the
determination of whether assessments are well aligned or not. The various
alignment criteria can be utilized to identify sources of misalignment between
the CSTs and CCC placement tests. This information can then be further
utilized to take steps to improve California’s consistency and articulation
across standards between the high school and community college levels.
64
CHAPTER 4
RESULTS
Analysis
Indices of Alignment
Reliability of Ratings. This study relies on the judgment of multiple
raters in analyzing test alignment. Therefore, before examining the results of
these judgments, it is important to determine the degree to which the raters
are in agreement. For this purpose, the generalizability coefficient (g-
coefficient) was employed to measure the degree of rater consistency. The
generalizability coefficient (see Brennan, 2001; Mushquash & O'Connor,
2006; Shavelson & Webb, 1991, 2006; Shavelson, Webb, & Rowley, 1989) is
a measure of reliability that, unlike its counterpart in classical test theory,
distinguishes between systematic and unsystematic sources of variability.
TABLE 3: Reliability Ratings (DOK)
CST Subject Area
Relative G- Coefficient(ρ) Absolute G-Coefficient(φ)
General Mathematics .90 .90
Algebra 1 .84 .83
Geometry .79 .77
The g-coefficient values were calculated independently for each of the
three tests (see Table 3). Rater agreement for both the General
Mathematics and Geometry tests exceeded the preferred threshold of .80.
65
The relative and absolute g-coefficients for the General Mathematics test
were both calculated at approximately .90, whereas the ratings for the
Geometry test were .84 and .83 respectively. The Algebra I test showed
slightly less consistency, while still meeting the minimally acceptable level of
.70, with relative and absolute g-coefficients calculated at .79 and .77
respectively.
TABLE 4: Categorical Concurrence Ratings
Criterion value for alignment: 6
CCC Math Placement Test Content General Math CST Algebra I CST Geometry CST
Whole Numbers and Fractions 16.89 4.67 0.00
Decimals and Percents 13.33 0.78 0.00
Applications and interpreting tables/graphs 12.67 4.33 1.78
Integers and Rationals 3.11 1.78 0.00
Algebraic Expressions and Operations 5.67 16.67 0.33
Operations with exponents 6.67 2.78 0.00
Equations, Inequalities, and Word Problems 6.44 8.00 0.44
Functions 2.33 2.78 0.11
Trigonometry 0.00 0.00 4.22
Geometry 11.22 0.33 18.56
Graphing 3.44 4.78 0.89
Applications and Other Algebra Topics 5.56 0.11 4.78
Categorical Concurrence. Categorical concurrence was analyzed on
a per standard basis, with six assessment items mapping into the objectives
for that standard indicating that categorical concurrence has been achieved
(see Table 4). The Algebra I standards test achieved categorical
concurrence two areas: (1) Algebraic Expressions and Operations and
66
(2) Equations, Inequalities, and Word problems. The Geometry test met the
categorical concurrence criteria in the area of Geometry only. The General
Mathematics test met the categorical concurrence criteria in most categories
including a near match in Algebraic Expressions and Operations as well as
Applications and Other Algebra topics, which averaged 5.67 and 5.56
respectively. Categorical concurrence was not achieved for (1) Integers and
Rationals, (2) Functions, (3) Trigonometry, or (4) Graphing.
Depth of Knowledge consistency. The depth of knowledge (DOK)
criterion is met for a standard if at least half of the items matched are tested
at a level of complexity at or above that of the standard. The calculated DOK
values for all standards are reported in Table 5.
TABLE 5: Depth of Knowledge Ratings
Criterion value for alignment: 50%
CCC Math Placement Test Content General Math CST Algebra I CST Geometry CST
Whole Numbers and Fractions 52.51% 46.49% N/A
Decimals and Percents 62.34% 33.33% N/A
Applications and interpreting tables/graphs 45.75% 65.54% 68.33%
Integers and Rationals 81.48% 78.13% N/A
Algebraic Expressions and Operations 70.85% 89.64% 33.33%
Operations with exponents 63.99% 41.67% N/A
Equations, Inequalities, and Word Problems 61.07% 81.45% 66.67%
Functions 100.00% 91.27% 100.00%
Trigonometry N/A N/A 89.63%
Geometry 71.12% 100.00% 82.98%
Graphing 97.78% 76.93% 100.00%
Applications and Other Algebra Topics 86.86% 100.00% 91.08%
67
Standards with a small number of matches can easily meet the DOK
criteria; but the results are not necessarily meaningful. For instance, if there
is only one content match within a standard, then either 100% or 0% of the
matched items assess the content at the desired level of complexity. Of
course, in this case, there are not enough items testing the standard to
determine that it is being adequately measured. Therefore, in this discussion
we will only explore the DOK level of standards that have also met the
categorical concurrence criteria.
For all three tests, the majority of the standards that achieved
categorical concurrence far exceeded the DOK criteria. The only exception
was with the standard of Applications and Interpreting Tables/Graphs, which
only weakly satisfied the DOK criteria for the General Mathematics test. This
indicates that where high school tests are assessing the same content as
community college tests, they generally do so at a cognitive level at least as
great as that of the community college placement tests.
Range of Knowledge. As with depth of knowledge and for similar
reasons, range of knowledge will only be discussed in areas where the
categorical concurrence criterion has been met. Table 6 provides the
calculated values for this index for all standards.
Within the General Mathematics tests, Equations, Inequalities, and
Word Problems met this criteria only weakly, while Algebraic Expressions
and Operations as well as Applications and Other Algebra Topics failed to
68
meet this criteria even minimally. For the Algebra I test, Algebraic
Expressions and Operations met the required ROK criteria. Additionally,
Equations, Inequalities, and Word Problems weakly met the criteria. The
Geometry test exceeded the minimum requirement in its singular matched
standard of Geometry. The fact that several of the standards failed to meet
the minimum ROK requirements suggests that many of the standards that
meet the benchmark for Categorical Concurrence do so because many high
school test elements map onto only a few community college objectives
within that standard.
TABLE 6: Range of Knowledge Ratings
Criterion value for alignment: 50%
CCC Math Placement Test Content General Math CST Algebra I CST Geometry CST
Whole Numbers and Fractions 58.33% 23.75% N/A
Decimals and Percents 65.28% 14.58% N/A
Applications and interpreting tables/graphs 50.79% 20.41% 9.52%
Integers and Rationals 48.15% 31.25% N/A
Algebraic Expressions and Operations 27.34% 68.06% 6.25%
Operations with exponents 54.17% 52.08% N/A
Equations, Inequalities, and Word Problems 42.42% 49.49% 12.12%
Functions 25.40% 20.99% 11.11%
Trigonometry N/A N/A 38.89%
Geometry 56.94% 18.75% 70.14%
Graphing 18.06% 27.08% 25.00%
Applications and Other Algebra Topics 21.67% 6.67% 16.19%
Balance of Representation. Balance of representation is analyzed
differently than the other three alignment values and is benchmarked at a
69
higher proportion of .70. For all three tests, every standard that met the
criteria for Categorical Concurrence far exceeded this minimum standard.
This is partially explained by the fact that the community college objectives
rarely matched more than one or two high school test elements. Table 7
provides the calculated value for the Balance of Representation index across
all community college objectives for General Mathematics, Algebra I, and
Geometry.
Research Questions
Question 1: How well do the California Standards Tests (CST) in
General Mathematics, Algebra I, and Geometry align with the content of the
most commonly utilized math placement tests in the California Community
TABLE 7: Balance of Representation Ratings
Criterion value for alignment: 70%
CCC Math Placement Test Content General Math CST Algebra I CST Geometry CST
Whole Numbers and Fractions 90.72% 94.73% N/A
Decimals and Percents 90.62% 100.00% N/A
Applications and interpreting tables/graphs 84.74% 84.39% 72.50%
Integers and Rationals 97.69% 98.44% N/A
Algebraic Expressions and Operations 86.31% 87.15% 66.67%
Operations with exponents 86.71% 100.00% N/A
Equations, Inequalities, and Word Problems 90.61% 86.49% 100.00%
Functions 89.05% 89.71% 100.00%
Trigonometry N/A N/A 88.13%
Geometry 94.05% 100.00% 87.38%
Graphing 81.53% 94.14% 100.00%
Applications and Other Algebra Topics 85.97% 100.00% 79.26%
70
College (CCC) System? Overall, the California Standards Tests explored did
not demonstrate a considerable amount of content alignment with the
California Community College placement tests. Fully one-third of all
objectives emphasized on community college placement tests are not tested
on the General Mathematics, Algebra I, or Geometry CSTs. Furthermore, if
the results of the Brown and Niemi study are considered along with those of
the current study only one additional category, Functions, is measured by the
additional tests. Therefore, one-quarter of math objectives emphasized by
community college placement tests are standards that are not tested at the
high school level.
Several standards that matched in terms of categorical concurrence
failed to meet the ROK criteria. This indicates that many standards that meet
the benchmark for categorical concurrence do so because the high school
test elements are mapped onto a minimal number of community college
objectives within that standard. This demonstrates that there are significant
areas of content mismatch between the exams, even beyond the community
college objectives, that are not adequately measured by the high school
standards tests.
In the areas where content match does occur, however, the high
school tests do comparatively well in terms of depth and balance. For
instance, the majority of standards that achieved categorical concurrence far
exceeded the DOK criteria and all standards at least weakly satisfied this
71
criteria. This indicates that where high school tests are assessing the same
content as community college tests, they generally do so at a cognitive level
at least as great as that of the community college placement tests. Balance
of Representation did not add anything to the understanding of alignment, as
all standards meeting categorical concurrence far exceeded the minimum
threshold.
Question 2: Are there areas emphasized on the mathematics
placement tests that are not measured by the California Standards Tests?
As previously indicated, this analysis identified four categories, or one-third of
the CCC placement standards, that failed to achieve a sufficient content
match. According to the Brown and Niemi study, the augmented forms of the
Summative High School and Algebra II standards tests also fail to meet the
minimum criteria in three of these four identified categories, (1) Integers and
Rationals, (2) Trigonometry, and (3) Graphing. Overall, when the results of
these two studies are considered together, it becomes apparent that 25% of
the core objectives being assessed by the California Community College
placement exams are not emphasized in the California high school
assessment systems. Furthermore, given that several standards failed to
meet the minimum benchmark for range of knowledge, it can be determined
that a considerable proportion of standards that have demonstrated content
match place emphasis on objectives that do not appear on the high school
tests, particularly in Algebra.
72
It is possible that the areas of Integers and Rationals or Graphing are
assessed on earlier CSTs. However, considering that the General
Mathematics test measures standards as early as 6
th
grade this is not likely.
It is more plausible these three standards: (1) Integers and Rationals, (2)
Trigonometry, and (3) Graphing measure college preparatory course
elements that are no longer emphasized to the degree they have been in the
past. Since 1997, when California adopted high school mathematics content
standards, the mathematics curriculum has changed considerably. Although
California has specific standards for Trigonometry and Linear Algebra
(California State Board of Education, 2007) which may address some of
theses inconsistencies, these courses are no longer part of the standard
college preparatory math sequence (see University of California, 2008) or the
high school math assessments taken by most California students.
Question 3: To what degrees can performance on the CSTs measure
progress towards meeting the “de facto” college readiness standards in
mathematics, as defined by CCC placement test content? Although, there
are areas of gross inconsistencies between the high school and college level
assessments explored here, many standards did show adequate alignment.
Furthermore, where that alignment did occur, the high school assessments
measured the objectives at a level of cognitive complexity at least as great as
that of the college placement tests. This indicates that the high school
standards tests have the potential to be utilized as benchmarks towards
73
meeting college readiness standards, as they already do a good job at
measuring achievement of the college readiness standards areas where
content match does occur. Therefore, should the articulation between the
two systems be improved in terms of content, the CSTs have the potential to
be utilized to diagnose areas in which students need additional preparation
before matriculating to a postsecondary course of study.
Most of the alignment that did occur was found within the General
Mathematics Test. In fact, neither the Algebra I nor the Geometry test
showed Categorical Concurrence with any standard that was not also
addressed by the General Mathematics Test. The only exception is that the
Algebra I test did a better job at measuring the standard of Algebraic
Expressions and Operations. The General Mathematics Test averaged 5.67
matches with this standards, versus 16.67 average hits for the Algebra I test.
Furthermore, only in this category, Algebraic Expressions and Operations,
did any of the standards that failed to meet the Range of Knowledge criteria
for the General Mathematics Test meet the minimum requirements in either
the Algebra I or Geometry Tests.
The General Mathematics test also demonstrated content match in
several of the categories that are essentially excluded from the augmented
CSTs used to make placement decisions at the UC/CSU level: (1) Whole
Numbers and Fractions, (2) Decimals and Percents, (3) Applications and
Interpreting Tables/Graphs, and (4) Operations with Exponents. Therefore, it
74
would be beneficial to explore the degree to which this test could be utilized,
either in its existing or a modified format, to make placement decisions at the
community college level.
75
CHAPTER 5
DISCUSSION
The primary purpose of this study was to determine the degree that
the high school mathematics content to which most students are exposed is
consistent with California Community College entrance expectations.
Previous studies suggest that on a national level alignment between
secondary and postsecondary educational systems is poor. Studies in
California, of tests taken by the best prepared 11
th
grade students, suggest
that alignment is good in English but poor in mathematics. This leads to the
question of whether the missing topics are covered in earlier tests within the
system.
The current study suggests that the mathematics tests taken by the
majority of California high school students are not well-aligned with the
content of the California Community College (CCC) placement tests. There
are four major areas of misalignment: (1) Integers and Rationals, (2)
Functions, (3) Trigonometry, and (4) Graphing. This indicates that only two-
thirds of the primary objectives in CCC placement tests are based upon
content that is emphasized in the high school curriculum taken by most of the
community college bound students. This suggests that students may only be
prepared to answer approximately 66% of the questions correctly by their
high school preparation.
76
One of the misaligned community college objectives, Functions, was
shown in a previous study to align with the augmented forms of the
Summative High School Math and Algebra II CSTs (Brown & Niemi, 2007).
Therefore, 3 rather than 4 of the 12 CCC placement test standards are not
assessed by examinations in California’s high school level accountability
system. However, this means that only 75% of the core objectives utilized to
make course placement decisions are based upon content that is
emphasized in the high school curriculum designed to prepare students to
place into and succeed in college level courses.
However, where the assessments measure the same content, the
high school tests address the concepts at a cognitive level at least as great
as that of the placement tests. This indicates that the placement tests are
not more cognitively demanding than the high school level tests and
suggests that improving the alignment in terms of content alone may be
sufficient to increase the fairness of the tests.
Implications for Policy and Practice
Findings suggest that California’s K-12 and college level expectations
need to be more explicitly articulated. This is consistent with findings across
the nation that indicate that there is a disconnect between what occurs at the
secondary and postsecondary levels of education. This lack of connection,
in California, means that a student who masters the standards of a college
77
preparatory curriculum during high school is still likely to be denied access to
college level courses at the community colleges. It also means that the high
school standards, in their current form, do not prepare students for the reality
of community college expectations. Nevertheless, students who complete a
college preparatory course sequence have every reason to believe that they
are well prepared. Given that community colleges are designed to serve any
student who can benefit from a higher education, this current situation is not
only inefficient and unfair, it is inconsistent with the philosophy of the
institutions.
This could be addressed through a combination of two main policy
changes. One option is, the high school level standards and assessments
could be revised to reflect the expectations that students will face upon
matriculation to the college campus. Alternatively, the community colleges
could predicate their placement test on the high school level standards of the
prerequisite classes for college level coursework.
The best solution may be a combination of the two. The community
colleges could explicitly define their expectations for beginning college level
work, keeping in mind the high school standards for the pre-requisite
coursework. A common set of placement exams should then be developed
that considers these expectations. Given that the postsecondary education
system may consider a particular objective to be vital, even if it is not part of
the existing standards, the K-12 education system should re-evaluate their
78
standards. This will ensure that all required content knowledge is sufficiently
emphasized during the standard course-taking sequence that most college
bound high school students take.
P-16 Alignment. The disconnected curricula and missions across the
secondary and postsecondary educational systems create barriers to
achievement for students, especially poor and minority students who often
lack the capital to navigate through this complex system without outside help.
The P-16 council initiated by the superintendent of California’s K-12 system
has the potential to dramatically improve the articulation across educational
systems; but, will be largely ineffective if the council is not given the authority
and funds to effect dramatic and lasting changes in the way education is
structured. These systems, which all have independent governing bodies,
will not be motivated to collaborate and cooperate unless they are held
accountable for the results.
First, members of the council must be held accountable for developing
a viable articulation plan. Second, each educational institution and system
needs to be held accountable for implementing the plan. This can be
accomplished by placing stipulations on both funding and accreditation that
require schools, districts and educational systems to participate in developing
the articulation plan and to implement the new mandates. Finally, to ensure
that best practices are followed, this process needs to be supervised by a
79
common governing official as well as leading experts in learning, standard
setting, and related fields.
Implications for the K-12 System. California needs to ensure high
school level standards sufficiently emphasize the topics that are deemed
necessary for college preparation. In the next revision of the California
Content Standards, it is important for the K-12 educational system to take
steps to ensure that these standards reflect college readiness standards.
Once the higher education system ensures that their expectations are made
explicit, the K-12 standards can include these expectations.
To help students, teachers, and counselors diagnose problem areas in
students’ preparation early enough for remediation during high school, the
testing system should include college readiness indicators. Along with their
scores on the regular standards tests, students should receive feedback on
their progress towards meeting college readiness goals. Teachers should be
given a report in the Fall on the areas of strength and weakness of their class
so that they can supplement classroom instruction to strengthen these areas.
Counselors can utilize the individual reports to place students in the proper
classes.
College readiness indicators would allow for students to improve their
math preparation during their senior year of high school. This would allow
them to review the core concepts that are necessary to successfully place
into and perform well in college level math classes. This would give the
80
students time to review content that is emphasized on placement exams but
underemphasized in the high school curriculum.
To truly address this problem effectively will take a great deal of time,
and collaboration across various educational sectors. However, high school
instructors and administrators can take steps to immediately improve the
success of their students at the community colleges. First, educational
leaders should work with the local community college district that the majority
of their graduates attend. Through explicit articulation agreements and clear
and continuous communication, student outcomes can be improved at the
community colleges. For example, high school administrators can negotiate
with community college administrations to allow students to place into college
level courses through the successful completion of pre-approved courses.
Second, high school teachers should be made aware of the content of the
placement exams of the local community colleges. This will allow teachers to
supplement the regular curriculum with content that appears on the
placement exams but is not emphasized in the state standards.
Implications for the Community College System. First, the California
Community College System should explicitly define their expectations for
college level coursework. Incoming college freshmen would also benefit
from a systemwide placement policy so that both students and teachers at
the high school level are clear as to what individuals are expected to know
and demonstrate to be placed into college level classes. A new systemwide
81
placement test based upon the high school standards, with a well-publicized
blueprint, would also help to clarify the expectations of students preparing for
college. Once the testing systems are better aligned, the California
Department of Education should begin to include college readiness indicators
as part of its Standardized Testing and Reporting procedures. This would
allow schools, teachers, parents, and students to become aware of any
deficiencies students may have while there is still time to address them.
Next, the California Community College system may want to consider
the feasibility of utilizing data from the augmented form of the California
Standards Tests for student placement, given that these exams are already
being used for this purpose by the California State University system.
Finally, they may want to consider working with the state department of
education to develop an augmented form of the General Mathematics CST
that includes the CCC placement standards currently not being assessed.
This is important because the majority of community college bound students
do not take the Algebra II or Summative High School math tests. In this way,
all students would have an opportunity to take a math test that could be used
to make placement decisions.
Community colleges have a history of local control over all decisions,
including curricular content. This has led to the current system of a myriad of
different placement tests and differing course content across the state. The
concept of setting standards for community college courses is met with
82
resistance from many community college instructors. They both enjoy and
feel entitled to their autonomy. However, the California systems of higher
education are built around a transfer model; and, community college courses
designed for transfer to the four-year universities should have the same pre-
requisites and content as their counterparts at the university level. In
practice, this means that students will be more successful if the courses are
standardized. The University of California and California State University
systems have both standardized their entrance expectations with the a-g
requirements. The K-12 system has standardized its course specific and exit
expectations with curricular standards. The California Community College
System is the only educational system within the state that has not
standardized its expectations.
Standardizing community college expectations will require a significant
investment initially in terms of both time and money; but, it is better for
students and more fiscally responsible. Standardization will allow the
colleges to develop a common placement exam and facilitate communication
with high schools and their students. Ultimately, standardization has the
potential to significantly reduce the amount of time and money spent on basic
skills level courses.
83
Limitations of the Study
Measuring the alignment between secondary and postsecondary
assessments is an important step in understanding the factors that contribute
to the widespread placement of California students into remedial level, non-
credit bearing courses. Nevertheless, alignment alone will not fix the
problem. Other factors that may contribute to the issue are language
barriers, student motivation, and real lack of preparation for college.
Furthermore, alignment is not the only important issue to consider when
evaluating the appropriateness of college placement tests. Colleges also
need to examine the predictive validity of the cut scores used against
performance in future coursework. Finally, this analysis highlighted
systemwide areas of misalignment across many placement tests. It does not
provide validity evidence for individual assessments within the system.
Suggestions for Future Research
Although there are areas of misalignment, many CCC placement
objectives were adequately measured by the California high school
standards tests. It would be useful to utilize these assessments as one
measure of readiness for college level work. Utilizing high school tests to
make college placement decisions could mitigate many alignment concerns
while saving colleges a considerable amount of money. Therefore, the
84
potential for utilizing CST scores to make placement decisions should be
explored.
However, these tests were designed as summative evaluations not as
placement tests. Therefore, before they can be used for this purpose, an
analysis will need to be performed to verify both their predictive validity, for
performance in future courses, and to determine appropriate cut scores for
this purpose. This could be accomplished by following a sample of high
school graduates and determining how well their high school standards
scores predict future math success.
Furthermore, in order to graduate from California high schools,
students are required to pass an examination in mathematics and English,
the California High School Exit Exam (CAHSEE). Considering that all
students take this exam, it would be informative to determine the degree to
which performance on the CAHSEE can be used to measure progress
towards college readiness or to make placement decisions. To that end, an
alignment study should be performed on the CAHSEE and the core
objectives for the California Community College Placement Tests.
Finally, the timing and subject matter of courses students take in high
school has an impact on future performance. Some work has been done
regarding the impact that taking courses such as Algebra II has on future
college success (ACT Inc., 2007b). California could benefit from a study that
85
looks at the course-taking patterns of high schools students and its effect on
both placement scores and performance in future classes.
Conclusion
In conclusion, this study found that there are several areas of
misalignment between California secondary and postsecondary
expectations. Specifically, community college placement tests assess
students in three main areas that are not part of the key standards that
students are held accountable for in high school. These areas are Integers
and Rationals, Trigonometry, and Graphing. In order for placement tests to
fairly measure students’ capabilities to perform in future math courses, the
high school and community college assessment systems should be better
aligned.
Most students, regardless of ethnic or economic background, want to
go to college. However, the current misaligned system disproportionately
affects poor and minority students. African American and Latino students are
more likely to begin their education at the community college level, more
likely to be placed into noncredit bearing courses and less likely to remediate
successfully. Therefore, misalignment contributes to racial inequities rather
than providing an equal opportunity for advancement. Improving the
alignment across secondary and postsecondary assessments can help to
ensure that students are given a sufficient opportunity to learn and
86
demonstrate the skills and knowledge that are necessary for college
success.
Although improving alignment will not solve all social inequities, it is a
vital step toward providing all students equal opportunities to obtain the
education they want and need. Improving outcomes at the community
college level where students enter the postsecondary educational pipeline,
has the potential to improve students’ motivation and performance in
community college classrooms, leading to increased opportunities for
transfer and ultimate degree attainment. This will provide all students,
especially poor and minority students, the opportunity to improve their social
standing through increased educational attainment.
87
REFERENCES 1
Academic Senate for California Community Colleges. (2004). Issues in basic 2
skills assessment and placement (09.22.04). Retrieved April 12, 2008, 3
from http://www.asccc.org/Events/sessions/fall04/materials/BasicSkills 4
Paper.doc. 5
Achieve Inc. (1997). A review of the 1996 National Education Summit. 6
Washington, DC: Achieve, Inc. 7
Achieve Inc. (2002). Staying on course. Standards-based reform in 8
America's schools: Progress and prospects. . Washington, DC: 9
Achieve, Inc. 10
Achieve Inc. (2004). Ready or not: creating a high school diploma that 11
counts. Washington, DC: Achieve, Inc. 12
Achieve Inc. (2007a). Aligned expectations? A closer look at college 13
admissions and placement tests. Washington, DC: Achieve, Inc. 14
Achieve Inc. (2007b). Closing the expectations gap 2007. An annual 50-state 15
progress report on the alignment of high school policies with the 16
demands of college and work access. Washington, DC: Achieve, Inc. 17
ACT Inc. (2007). ACT National Curriculum Survey®, 2005-2006. Iowa City, 18
IA.: ACT, Inc. 19
Ali, R. (2002). The high school diploma: making It more than an empty 20
promise. Sacramento, CA: Senate Standing Committee on Education. 21
Alpert, D., Alquist, E., & Strom-Martin, V. (2002). The California master plan 22
for education: Joint Committee to Develop a Master Plan for 23
Education. 24
American Educational Research Association, American Psychological 25
Association, & National Council on Measurement in Education. (1999). 26
Standards for educational and psychological testing. Washington, DC: 27
American Educational Research Association. 28
Antonio, A. L., & Bersola, S. H. (2004). Working toward K-16 coherence in 29
California. In M. W. Kirst & A. Venezia (Eds.), From High School to 30
College: Improving Opportunities for Success in Postsecondary 31
Education (pp. 31-76). Indianapolis, IN: Jossey-Bass, An Imprint of 32
Wiley. 33
88
Armstrong, W. B. (1994). Math placement validation study: A summary of the 1
criterion-related validity evidence and multiple measures data for the 2
San Diego Community College District: San Diego Community Coll. 3
District, CA. Research and Planning. 4
Armstrong, W. B. (1999). Explaining community college outcomes by 5
analyzing student data and instructor effects. Unpublished 6
Dissertation, University of California Los Angeles, Los Angeles, CA. 7
Armstrong, W. B. (2000). The Association among Student Success in 8
Courses, Placement Test Scores, Student Background Data, and 9
Instructor Grading Practices. Community College Journal of Research 10
and Practice, 24(8), 681-695. 11
Bahr, P. R. (2004). The rough and rocky road of remediation: Racial 12
inequalities in postsecondary remedial mathematics. Unpublished 13
Ph.D., University of California, Davis, United States -- California. 14
Baker, E. L. (2004). Aligning curriculum, standards, and assessments: 15
Fulfilling the promise of school reform. CSE Report 645. Los Angeles, 16
CA California Univ., Los Angeles. Center for the Study of Evaluation. 17
Bhola, D. S., Impara, J. C., & Buckendahl, C. W. (2003). Aligning tests with 18
states' content standards: Methods and issues. Educational 19
Measurement: Issues and Practice, 22(3), 21-29. 20
Blank, R. K. (2002). Surveys of enacted curriculum: Tools and services to 21
assist educators access. Washington, DC: Council of Chief State 22
School Officers, Washington, DC. 23
Blank, R. K., Porter, A., & Smithson, J. (2001). New tools for analyzing 24
teaching, curriculum and standards in mathematics & science. Results 25
from survey of enacted curriculum project. Final Report. Washington, 26
DC: Council of Chief State School Officers, Washington, DC. State 27
Education Assessment Center. 28
Brennan, R. L. (2001). Generalizability theory. New York: Springer. 29
Brown, R. S., & Conley, D. T. (2003, April 21-25, 2003). Comparing state 30
high school assessments to standards for success in entry-level 31
university courses. Paper presented at the Annual Meeting of the 32
American Educational Research Association, Chicago, IL. 33
89
Brown, R. S., & Conley, D. T. (2007). Comparing state high school 1
assessments to standards for success in entry-level university 2
courses. Educational Assessment, 12(2), 137. 3
Brown, R. S., & Niemi, D. N. (2007). Investigating the alignment of high 4
school and community college assessments in California- a report to 5
the National Center for Public Policy in Higher Education: National 6
Center for Public Policy in Higher Education. 7
Bryant, A. N. (2001). ERIC review: Community college students recent 8
findings and trends. Community College review, 29(3), 77-93. 9
Bueschel, A. C. (2004). The missing link: The role of community colleges in 10
the transition between high school and college. In M. W. Kirst & A. 11
Venezia (Eds.), From High School to College: Improving Opportunities 12
for Success in Postsecondary Education (pp. 252-284). Indianapolis, 13
IN: Jossey-Bass, An Imprint of Wiley. 14
California Community College Assessment Association. (2001). Assessment 15
101: A compilation of information for validating assessment tests. 16
Retrieved April 28, 2008, from http://198.189.144.207/CCCAA/ 17
Assessment%20101.pdf. 18
California Community Colleges. (2006). System strategic plan. Retrieved 19
March 25 2007, from http://strategicplan.cccco.edu. 20
California Department of Education. (1960). A master plan for higher 21
education in California, 1960-1975. Sacramento, CA. 22
California Department of Education. (2008). California Standardized Testing 23
and Reporting (STAR) Program. Retrieved May 5, 2008, from 24
http://star.cde.ca.gov/star2007/Viewreport.asp 25
California Postsecondary Education Commission. (2005). University 26
preparedness of public high school graduates. Sacramento, CA. 27
California State Board of Education. (2007). Content standards - Standards & 28
frameworks (CA Dept of Education). Retrieved Nov 27, 2007, from 29
http://www.cde.ca.gov/be/st/ss/ 30
Callan, P. M., Finney, J. E., Kirst, M. W., Usdan, M. D., & Venezia, A. (2006). 31
Claiming common ground: State policymaking for improving college 32
readiness and success. National Center Report #06-1. San Jose, CA: 33
National Center for Public Policy and Higher Education. 34
90
CCC Chancellor's Office. (2007). California Community Colleges - 1
Chancellor's office website. Retrieved Nov 27, 2007, from 2
http://www.cccco.edu/ 3
Colwell, R. (2006). MENC handbook of research methodologies. Oxford; 4
New York: Oxford University Press. 5
Conley, D. T. (2003a). Mixed messages: what state high school tests 6
communicate about student readiness for college. Portland, OR: 7
University of Oregon. 8
Conley, D. T. (2003b). Understanding University Success. A Report from 9
Standards for Success. Eugene, OR: Center for Educational Policy 10
Research. 11
Conley, D. T. (2003c). Who governs our schools? Changing roles and 12
responsibilities. Critical issues in education leadership. New York, NY: 13
Teachers College Press. 14
Conley, D. T. (2005a). College knowledge: what It really takes for students to 15
succeed and what we can do to get them ready. Indianapolis, IN: 16
Jossey-Bass. 17
Conley, D. T. (2005b). What we must do to create a system that prepares 18
students for college success. Policy perspectives. San Francisco, CA: 19
WestEd. 20
Conley, D. T., & Venezia, A. (2003, April 21-25). High school transitions: 21
State of the art and views of the future. Paper presented at the Annual 22
Meeting of the American Educational Research Association, Chicago, 23
IL. 24
Editorial Projects in Education Research Center. (2007). What It takes to 25
graduate for the class of 2007. Bethesda, MD. 26
Education Trust-West. (2004). The a–g curriculum: College-prep? Work- 27
prep? Life-prep. Understanding and implementing a rigorous core 28
curriculum for all. Oakland, CA. 29
Fall 2006 final regularly admitted first-time freshmen remediation 30
systemwide. Retrieved Mar 19, 2007, from http://www.asd. 31
calstate.edu/remediation/06/Rem_Sys_fall2006.htm 32
91
Flores, G. (2003). Golden state examination (GSE) program: Including, but 1
not limited to, update on the GSE program. Retrieved Jan 02, 2008, 2
from http://www.cde.ca.gov/be/ag/ag/greenapr03item11.pdf 3
Herman, J. L., Webb, N., & Zuniga, S. (2003). Alignment and college 4
admissions: The match of expectations, assessments, and educator 5
perspectives. CSE technical report. . Los Angeles, CA: California 6
Univ., Los Angeles. Center for the Study of Evaluation. 7
Herman, J. L., Webb, N. M., & Zuniga, S. A. (2007). Measurement issues in 8
the alignment of standards and assessments: A case study. Applied 9
Measurement in Education, 20(1), 101-126. 10
Hill, E. G. (1999). A K-12 master plan - starting the process. Sacramento, 11
CA: California Legislative Analyst's Office. 12
Impara, J. C. (2001). Alignment: One element of an assessment's 13
instructional utility. Paper presented at the National Council on 14
Measurement in Education, Seattle, WA. 15
Isonio, S. (1992). Implementation and Initial Validation of the MDTP Tests at 16
Golden West College. 17
Jung, T. (2007). State schools chief O'Connell names members to P-16 18
council. Retrieved Aug 3, 2007, from http://www.cde.ca.gov/nr/ne/ 19
yr05/yr05rel42.asp?print=yes 20
Kendall, J. S., & Snyder, C. (2003). Finding the time to learn: A guide. 21
Washington, D.C.: Institute of Education Sciences: U.S. Department of 22
Education. 23
Kirst, M. W. (1998). Improving and aligning K-16 standards, admissions, and 24
freshman placement policies. Stanford, CA: National Center for 25
Postsecondary Improvement. 26
Kirst, M. W. (2004a). The high school-college disconnect. Association for 27
Supervision and Curriculum Development, 62(3), 51-55. 28
Kirst, M. W. (2004b). Using a K-12 assessment for college placement. In R. 29
Kazis, J. Vargas & N. Hoffman (Eds.), Double the Numbers: 30
Increasing Postsecondary Credentials for Underrepresented Youth. 31
Cambridge, MA: Harvard Education Publishing Group. 32
92
Kirst, M. W. (2007, May 29). Enhancing college completion: secondary 1
schools and colleges must work together. Paper presented at the 2
often: Revisioning the American High School for an Engaged 3
Citizenry, Penn State University. 4
Kirst, M. W., Hayward, G. C., & Fuller, B. (2000). Governance and 5
accountability. In E. Burr, G. Hayward, B. Fuller & M. W. Kirst (Eds.), 6
Crucial issues in California education 2000: Are the reform pieces 7
fitting together? Berkeley, CA: Policy Analysis for California. 8
Kirst, M. W., & Venezia, A. (2001). Bridging the great divide between 9
secondary schools and postsecondary education. Phi Delta Kappan, 10
83(1), 92-97. 11
Kirst, M. W., & Venezia, A. (2004). From high school to college: Improving 12
opportunities for success in postsecondary education. Indianapolis, 13
IN: Jossey-Bass, An Imprint of Wiley. 14
Kirst, M. W., Venezia, A., & Antonio, A. L. (2004). What have we learned, 15
and where do we go next? In M. W. Kirst & A. Venezia (Eds.), From 16
High School to College: Improving Opportunities for Success in 17
Postsecondary Education (pp. 285-319). Indianapolis, IN: Jossey- 18
Bass, An Imprint of Wiley. 19
Koretz, D. M., & Hamilton, L. S. (2006). Testing for accountability in K-12. In 20
R. L. Brennan (Ed.), Educational Measurement. Fourth Edition. 21
ACE/Praeger Series on Higher Education (4 ed.). Portsmouth, NH: 22
Greenwood Publishing Group. 23
Kulm, G., Dager Wilson, L., & Kitchen, R. (2005). Alignment of content and 24
effectiveness of mathematics assessment items. Educational 25
Assessment, 10(4), 333-356. 26
La Marca, P. M. (2001). Alignment of standards and assessments as an 27
accountability criterion: ERIC Clearinghouse on Assessment and 28
Evaluation. 29
La Marca, P. M., Redfield, D., & Winter, P. C. (2000). State standards and 30
state assessment systems: a guide to alignment. Washington, DC: 31
Council of Chief State School Officers. 32
93
Le, V., Hamilton, L., & Robyn, A. (2000). Alignment among secondary and 1
post-secondary assessments in California. In E. Burr, G. Hayward, B. 2
Fuller & M. W. Kirst (Eds.), Crucial issues in California education 3
2000: Are the reform pieces fitting together? Berkeley, CA: Policy 4
Analysis for California. 5
Lundell, D. B., Higbee, J. L., Hipp, S., & Copeland, R. (2004). Building 6
bridges for access and success from high school to college: 7
Proceedings of the Metropolitan Higher Education Consortium's 8
Developmental Education Initiative Minneapolis, MN: Minnesota 9
University Center for Research on Developmental Education and 10
Urban Literacy. 11
Martineau, J., Paek, P., Keene, J., & Hirsch, T. (2007). Integrated, 12
comprehensive alignment as a foundation for measuring student 13
progress. Educational Measurement: Issues and Practice, 26(1), 28- 14
35. 15
Marwick, J. D. (2004). Charting a path to success: The association between 16
institutional placement policies and the academic success of Latino 17
students. Community College Journal of Research and Practice, 18
28(3), 263-280. 19
Marzano, R. J. (1993). How classroom teachers approach the teaching of 20
thinking. Theory into Practice, 32(3), 154-160. 21
Marzano, R. J. (2001). Designing a new taxonomy of educational objectives. 22
Thousand Oaks, Calif.: Corwin Press. 23
Marzano, R. J., & Kendall, J. S. (2007). The new taxonomy of educational 24
objectives. Thousand Oaks, CA: Corwin Press. 25
Moses, Y. T. (1999). Taking a stand on remediation. Presidency, 2(3), 20-25. 26
Mushquash, C., & O'Connor, B. P. (2006). SPSS and SAS programs for 27
generalizability theory analyses. Behavior Research Methods, 38, 28
542-547. 29
No Child Left Behind Act of 2001, Public Law 107-110 (2002). 30
Olson, L. (2006). California high schoolers get preview of college-placement 31
test. Education Week 25(33), 27. 32
94
Perin, D. (2006). Can community colleges protect both access and 1
standards? The problem of remediation. Teachers College Record, 2
108(3), 339-373. 3
Planty, M., Bozick, R., & Ingels, S. J. (2006). Academic pathways, 4
preparation, and performance—A descriptive overview of the 5
transcripts from the high school graduating class of 2003–04 (NCES 6
2007–316). Washington, DC: U.S. Department of Education, National 7
Center for Education Statistics. 8
Porter, A. C., Smithson, J., Blank, R., & Zeidner, T. (2007). Alignment as a 9
teacher variable. Applied Measurement in Education, 20(1), 27-51. 10
Research and Planning Group for California Community Colleges. (2005). 11
Environmental Scan: A summary of Key Issues Facing California 12
Community Colleges Pertinent to the Strategic Planning Process 13
[Electronic Version]. Retrieved March 27, 2007 from www.rpgroup.org. 14
Shavelson, R. J., & Webb, N. M. (1991). Generalizability theory a primer. 15
Newbury Park, Calif.: Sage Publications. 16
Shavelson, R. J., & Webb, N. M. (2006). Generalizability theory. In J. L. 17
Green, G. Camilli, P. B. Elmore & American Educational Research 18
Association (Eds.), Handbook of complementary methods in education 19
research. Washington, D.C.: Lawrence Erlbaum Associates. 20
Shavelson, R. J., Webb, N. M., & Rowley, G. L. (1989). Generalizability 21
theory. American Psychologist, 44(6), 922-932. 22
Shepard, L. A. (1993). Evaluating test validity. Review of Research in 23
Education, 19, 405-450. 24
Sigman, D. (2006). Procedures to award golden state seal merit diploma to 25
graduating seniors. Retrieved Jan 02, 2008, from 26
http://www.cde.ca.gov/ta/tg/sr/gseletter.asp 27
UC Office of the President. (2007). Major features of the California Master 28
Plan for Higher Education. Retrieved April 11, 2007, from 29
http://www.ucop.edu/acadinit/mastplan/mpsummary.htm 30
University of California. (2008). 2008 Guide to “a-g” Requirements and 31
Instructions for Updating Your School’s “a-g” Course List. Retrieved 32
April 20, 2008. from http://www.ucop.edu/a-gGuide/ag/content/Guide 33
toa-gReqs_2008.pdf. 34
95
University of California Academic Senate. (1997). Statement on 1
competencies in mathematics expected of entering college students. 2
Retrieved April 26, 2008, from http://www.universityofcalifornia.edu/ 3
senate/reports/mathcomp.html 4
Venezia, A., Callan, P. M., Finney, J. E., Kirst, M. W., & Usdan, M. D. (2005). 5
The governance divide: a report on a four-state study on improving 6
college readiness and success: The National Center for Public Policy 7
and Higher Education. 8
Venezia, A., Kirst, M. W., & Antonio, A. L. (2003). Betraying the college 9
dream: How disconnected K-12 and postsecondary education 10
systems undermine student aspirations. Retrieved March 27, 2007, 11
from http://www.stanford.edu/group/bridgeproject/betrayingthecollege 12
dream.pdf 13
Webb, N. L. (1997). Criteria for alignment of expectations and assessments 14
in mathematics and science education. Research monograph No. 6. 15
Washington, DC: Council of Chief State School Officers. 16
Webb, N. L. (1999). Alignment of science and mathematics standards and 17
assessments in four states. Research monograph No. 18. Madison, 18
WI: Wisconsin Center for Education Research. 19
Webb, N. L. (2002, April 1-5). An analysis of the alignment between 20
mathematics standards and assessments for three states. Paper 21
presented at the American Educational Research Association Annual 22
Meeting, New Orleans, Louisiana. 23
Webb, N. L. (2007). Issues related to judging the alignment of curriculum 24
standards and assessments. Applied Measurement in Education, 25
20(1), 7-25. 26
Webb, N. M., Herman, J. L., & Webb, N. L. (2007). Alignment of mathematics 27
state-level standards and assessments: The role of reviewer 28
agreement. Educational Measurement: Issues and Practice, 26(2), 17- 29
29. 30
Willett, T. (2007). Research summary 2006-2007. Encinitas, CA: California 31
Partnership for Achieving Student Success. 32
Abstract (if available)
Abstract
More than 60% of all community college students are placed into remedial, non-credit bearing courses. Concerns over the lack of articulation across the K-12 and postsecondary educational systems have led to questions regarding whether students have had the opportunity to learn and demonstrate the skills required for success in college level classes. To measure the degree to which the expectations across these systems are consistent, the degree of alignment between the examinations at these two levels was explored. The California Community College placement test content was compared to the high school level California Standards Tests in General Mathematics, Algebra 1, and Geometry. Only the General Mathematics Test was aligned across a substantial number of standards. Taking into consideration past studies, it appears that the major source of misalignment between the two testing systems occurs within the content areas of Integers and Rationals, Trigonometry, and Graphing.
Linked assets
University of Southern California Dissertations and Theses
Conceptually similar
PDF
A study of alignment between the Common Core State Standards in math and an introductory college textbook
PDF
Reforming developmental education in math: exploring the promise of self-placement and alternative delivery models
PDF
Students in basic skills mathematics: perceptions and experiences of community college progress and help-seeking
PDF
The effect of reading self-efficacy, expectancy-value, and metacognitive self-regulation on the achievement and persistence of community college students enrolled in basic skills reading courses
PDF
The impact of remedial mathematics on the success of African American and Latino male community college students
PDF
High school math reform and the role of policy, practice and instructional leadership on math achievement: a case study of White Oak Tree High School
PDF
Three essays on the high school to community college STEM pathway
PDF
A multi-perspective examination of developmental education: student progression, institutional assessment and placement policies, and statewide regulations
PDF
Persistence among low income community college students
PDF
What motivational factors influence community college students' tendency to seek help through math tutoring?
PDF
Changing the landscape of institutional assessment on transfer: the impact of action research methods on community college faculty and counselors
PDF
Developing adaptive expertise among community college faculty through action inquiry as a form of assessment
PDF
Unprepared for college mathematics: an investigation into the attainment of best practices to increase preparation and persistence for first-time California community college freshmen in remedial...
PDF
Secondary math reform and the role of policy, practice, and instructional leadership on math achievement
PDF
College academic readiness and English placement
PDF
Teaching interpersonal and communication feedback skills to standardized patients: assessment of a cognitive model
PDF
The perceptions and attitudes of “low-router” students in developmental math
PDF
Relationships between a community college student’s sense of belonging and student services engagement with completion of transfer gateway courses and persistence
PDF
Factors affecting the success of older community college students
PDF
Achieving equity in educational outcomes through organizational learning: enhancing the institutional effectiveness of community colleges
Asset Metadata
Creator
Shelton, Alison R.
(author)
Core Title
Measuring the alignment of high school and community college math assessments
School
Rossier School of Education
Degree
Doctor of Education
Degree Program
Education (Psychology)
Publication Date
07/31/2008
Defense Date
06/12/2008
Publisher
University of Southern California
(original),
University of Southern California. Libraries
(digital)
Tag
alignment,basic skills education,California,community college,High School,Higher education,mathematics,OAI-PMH Harvest,placement,standards,testing
Place Name
California
(states)
Language
English
Advisor
Brown, Richard (
committee chair
), Niemi, David (
committee member
), Rueda, Robert (
committee member
)
Creator Email
ahibbard@usc.edu
Permanent Link (DOI)
https://doi.org/10.25549/usctheses-m1474
Unique identifier
UC1438313
Identifier
etd-Shelton-20080731 (filename),usctheses-m40 (legacy collection record id),usctheses-c127-91808 (legacy record id),usctheses-m1474 (legacy record id)
Legacy Identifier
etd-Shelton-20080731.pdf
Dmrecord
91808
Document Type
Dissertation
Rights
Shelton, Alison R.
Type
texts
Source
University of Southern California
(contributing entity),
University of Southern California Dissertations and Theses
(collection)
Repository Name
Libraries, University of Southern California
Repository Location
Los Angeles, California
Repository Email
cisadmin@lib.usc.edu
Tags
alignment
basic skills education
community college
placement
standards
testing