University of Southern California Dissertations and Theses

Systemic multilayered assessment of global awareness in undergraduate students: an innovation study

(USC Thesis Other)

Systemic multilayered assessment of global awareness in undergraduate students: an innovation study

PDF

Download a page range

Download transcript

Copy asset link

Request this asset

Transcript (if available)

Content Running head: MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 1
SYSTEMIC MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS IN
UNDERGRADUATE STUDENTS: AN INNOVATION STUDY

by

Eric Canny

A Dissertation Presented to the
FACULTY OF THE USC ROSSIER SCHOOL OF EDUCATION
UNIVERSITY OF SOUTHERN CALIFORNIA
In Partial Fulfillment of the
Requirements for the Degree
DOCTOR OF EDUCATION

May 2016

Copyright 2016 Eric Canny

MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 2

Acknowledgments
I have been privileged to have wonderful mentors, and would like to acknowledge a few
individuals. Dr. John Fitzgerald Gates, for teaching me about leadership through your actions,
that with authority comes responsibility, and in 1999 asking me to breakfast to discuss a new
amorphous entity called “Global” at New York University. Dr. Earl F. Gibbons for teaching me
the importance of strategy, pushing me into new professional realms beyond my comfort zone
while simultaneously holding me to high standards, and for being there unconditionally at a very
hard time. Dr. Elizabeth “Beth” Paul for telling me not only that I had earned my seat at the
table but that my voice was equal, for hours of conversation about my future, and encouraging
me to apply to USC for my doctoral degree.
My dissertation committee, who provided the guidance and support of my dissertation
and helped shape this writing and research. A special thank you to my chair, Dr. Cathy Krop,
who served as a guide in my academic and professional development through my years at USC.
From meeting in the lobby of the hotel in Hong Kong talking me down from a cliff, to coffee in
Los Angeles reminding me to look forward, but, not to run away, you kept me focused and made
the process human. Your laser focused insight cut through my convoluted text coupled with
theoretical gibberish, found solutions that made my writing better, and pushed me in new
directions. Dr. Rob Filback for motivating me from the very first course. Your insightful
guidance pressed my thinking into new areas I could not have imagined from my initial research
question, and, you taught me to embrace innovation. Dr. Helena Seli for laying KMO
groundwork in a way my brain got, keeping me contained when I pushed back (embrace APA!),
and your careful questioning to refine the complexities around my research and ensure it was
manageable.
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 3

To all Global Ed.D. faculty members, and program staff, who’s dedication to students
makes the program what it is: amazing from design, logistics, support, to engagement and
learning that truly is unique in the field. The compassion and speed with which you all respond
sets a new bar in education. You have created, and are committed to, something so special.
To my Cohort Three Global Ed.D. colleagues thank you. Without each of you, this
journey would have been much less exciting and stimulating. You brought different
perspectives, listened, pressed, and debated with passion and commitment.
To my Mexicana Avocado, for being there without fail. From meeting me at airports
caffeine in hand, movie nights in hotels around the globe, an ambulance ride, editing and
feedback in airports and lounges around the world, thank you. I don’t want to even attempt to
count the emails, text, WhatsApp, and WeChat messages that kept me smiling, as well as going.
To the trainers at Steele Fitness Downtown, for being my stress relief, and reminding as
well as helping to keep my body in shape as I pushed my mind.
To my family and friends for your unconditional love, patience, support, and
understanding during my two-year absence. Thank you for accepting one-line emails, brief
phone calls, missing milestones, and simply disappearing for weeks on end. And, for believing
in me that I could do it this time.
Finally, Marco. Whether words of encouragement or letting me be silent for hours on
end, making me laugh, and keeping me focused. Thank you for helping me to always see the big
picture when I got into the weeds, and pulling me back from the edge countless times.

MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 4

Table of Contents
Acknowledgments 2
List of Tables 7
List of Figures 8
Abstract 10
Chapter One: Overview of the Study 11
Organizational Context and Mission 12
Organizational Performance Need 14
Related Literature 15
Importance of the Organizational Innovation 18
Organizational Performance Goal 19
Stakeholders and Stakeholders’ Performance Goals 19
Stakeholder for the Study and Stakeholder Performance Need 22
Purpose of the Project and Questions 23
Methodological Framework 24
Definitions 24
Organization of the Dissertation 27
Chapter Two: Literature Review 29
Global Education 29
History and Growth of the Field 29
Current Definitions of Global Education in Higher Education Institutions 31
Current Activities of Global Education in Higher Education Institutions 32
Current and Evolving Models of Global Education in Higher Education Institutions 33
Evolution of Definition of Global Education as a Dimension in the Curriculum 35
Expanding Views of Assessment in Higher Education 36
Typical Models of Assessment in Higher Education 36
Recent Movements Toward Broad-Based Assessment 37
Challenges in Assessing Global Education in Higher Education Institutions 45
Academic Programs 46
Student Affairs Programs 48
Other Programs 49
Knowledge, Motivation and Organization Models 49
Organizational Goals and Mission 50
Performance Needs 51
Knowledge and Skills 52
Motivation 52
Organization 52
Administration Knowledge, Motivation and Organizational Needs to Develop Global
Assessment in Higher Education Institutions 54
Knowledge and Skills 54
Motivation 56
Organization 57
Conclusion 59
Chapter Three: Methodology 63
Purpose of the Project and Questions 63
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 5

Methodological Framework 63
Assumed Performance Needs 67
Preliminary Scanning Data 67
Learning and Motivation Theory 71
Summary 74
Data Collection 78
Surveys 78
Interviews 79
Document Analysis 80
Validation of the Performance Needs 80
Summary 81
Data Analysis and Validity 81
Trustworthiness of Data 82
Role of Investigator 83
Limitations and Delimitations 84
Chapter Four: Results and Findings 86
Knowledge of Current Assessment at North College 88
Knowledge of, and Current Success at, Assessment at North College 96
Knowledge of AAC&U Framework Rubrics 99
Knowledge of Global Education Assessment 102
Knowledge of and to Develop Multilayered Assessment 104
Student Knowledge of Learning Outcomes 108
Synthesis of Results and Findings for Knowledge Needs 108
Results and Findings for Motivation Needs 110
Need for Increased Motivation to Serve on Committee(s) and Global
Education Assessment 111
See Importance of Assessment and Global Education to Mission of North College 112
See Value of AAC&U Rubrics in Assessment 116
Need Confidence in Ability to Map Student Learning Outcomes Across Curriculum
and Co-curricular 116
Have Desire to Learn New Types of Assessment Models 116
See Value of Global Education Student Learning Outcomes Assessment 118
See Value in Importance of Assessment between Curricular and Co-curricular 120
Synthesis of Results and Findings for Motivation Needs 121
Results and Findings for Organization Needs 122
Organizational Clarity and Defined Goals 124
Structure of Accountability 128
Newer Stakeholders to Provide Feedback Within Formal Settings 131
Commitment and Resources Devoted to Assessment of Global Education 132
Support of Learning and Motivation in Assessment and Development of New
Types of Assessment 137
Clarity and Resources Due to Prior Financial Crisis 138
Synthesis of Results and Findings for Organization Needs 139
Summary of Validated Knowledge, Motivation and Organization Needs 140
Chapter Five: Solutions, Implementation and Evaluation 145
Validated Needs Selection and Rationale 146
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 6

Solutions 149
Solution Theme One: Institutional Establishment of Measures and Capacity for
Excellence to Reinforce Assessment Accountability 152
Proposed Solution One: Establish Clear Goals, Communication and Oversight of
Assessment at North College 153
Proposed Solution Two: Establish Knowledge and Capacity to Capture SLOs 153
Solution Theme Two: Stakeholder Capacity to Define, Implement and Encourage
Standards of Excellence in Assessment 154
Proposed Solution Three: Establish a Comprehensive Stakeholder Assessment
Development Program 157
Organizational Features Relevant to Implementation 160
Additional Components of Implementation Plan 163
Implementation Plans: Create an Organizational Culture of Learning
Regarding Assessment 164
Theme: Institutional Establishment of Measures and Capacity for Excellence
to Reinforce 164
Theme: Stakeholder Capacity to Define, Implement and Encourage Standards
of Excellence in Assessment 167
Evaluation Plan 169
Level 1: Reactions 169
Level 2: Learning 170
Level 3: Behavior 171
Level 4: Impact 172
Strengths and Weaknesses of the Approach 173
Limitations 174
Future Research 174
Conclusion 176
References 180
Appendix B: Interview Protocol – Sample Group 208
Appendix C: Document Protocol 211
Appendix D: American Association of Colleges And Universities Global Education
and Intercultural Rubrics 212

MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 7

List of Tables
Table 1: Organizational Mission, Organizational Goal and Stakeholder Performance Goals 22
Table 2: Relationships Between Selected High-Impact Activities, Deep Learning, and
Self-Reported Gains 41
Table 3: Relationships Between Selected High-Impact Activities and Clusters of
Effective Educational Practices 42
Table 4: Summary of Assumed Needs for Knowledge, Motivation, and
Organizational Issues 75
Table 5: Summary of Assumed Knowledge Needs and Validation 81
Table 6: Knowledge Needs Validated, Not Validated, and New Needs 87
Table 7: Summary of Validated Assumed Knowledge Needs & New Needs 109
Table 8: Summary of Motivation Needs Validated, Not Validated 110
Table 9: Summary of Motivation Needs Validated, Not Validated 122
Table 10: Summary of Organization Needs Validated & Not Validated 123
Table 11: Summary of Organization Needs Validated & Not Validated 140
Table 12: Summary of Assumed Knowledge, Motivational and Organizational
Needs & New Needs 143
Table 13: Summary of Assumed Knowledge, Motivation and Organizational
Needs & New Needs 147
Table 14: Summary Proposed Solutions 150
Table15: Summary of Solutions and Timeline 158
Table 16: Solution #1 Establish Clear Goals, Communication and Oversight of
Assessment at North College 164
Table 17: Solution #2 Establish Knowledge and Capacity to Capture SLOs 165
Table 18: Solution #3 Establish a Comprehensive Stakeholder Assessment
Development Program 167

MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 8

List of Figures
Figure 1: A Visual Representation of Systemic Multilayered Assessment 26
Figure 2: A Visual Representation of the Stakeholders 27
Figure 3: The Gap Analysis Process. Adapted from Clark and Estes (2008)
and Rueda (2015) 66
Figure 4: Respondent Representation On Committees Charged with Assessment Oversight 78
Figure 5: Respondent View of Entity at North College Responsible for Assessment 90
Figure 6: Respondent View of Who Has Responsibility for Assessment at North College 91
Figure 7: North College Undergraduate Student Learning Outcomes 94
Figure 8: Respondent View of Ability to Conduct and Develop General Assessment
Within and Across Areas 98
Figure 9: Respondent View of AAC&U Rubrics Interconnected with Global Education
Rubric at North College 101
Figure 10: Respondent View of Ability to Conduct and Develop Global Education
Assessment Within and Across Areas 103
Figure 11: Respondent View of Sufficient Faculty and Staff Knowledge to Conduct
Global Education Assessment 107
Figure 12: Respondent View of Importance of Assessment at North College Broadly 113
Figure 13: Respondent View of Why Assessment Is Important to North College
as an Institution 114
Figure 14: Respondent Belief that North College Is Committed to Assessment of
Global Education 115
Figure 15: Respondent Desire to Learn New Kinds of Assessment Models 117
Figure 16: Respondent Desire to Learn New Kinds of Assessment Models for
North College That Include Global Education 117
Figure 17: Respondent Views on Importance, Value and Confidence to Develop
Assessment Models 119
Figure 18: Respondent View of Why Assessment Is Important to Their Specific Work
at North College, Not The College as A Whole 120
Figure 19: North Core Curriculum 124
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 9

Figure 20: Respondent View That the Academic Model at North College Makes
Assessment Difficult 125
Figure 21: Respondent View Assessment at North College Could Improve While
Maintaining the Academic Model 126
Figure 22: Respondent Sense of Clear Goals Related to Assessment 130
Figure 23: Respondent Belief North College Is Committed to Assessment of
Global Education 132
Figure 24: Respondent Sense of Institution Support for Assessment 133
Figure 25: Respondent Sense of Institutional Resources to Implement Multilayered
Assessment of Global Education 135
Figure 26: Respondent Belief North College Is Committed to Assessment of
Global Education 138
Figure 27: Managing Complex Change 146
Figure 28: A Visual Representation of Systemic Multilayered Assessment Revised 148

MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 10

Abstract
This study utilized the Clark and Estes (2008) gap analysis framework to conduct a needs
analysis that examined the knowledge, motivation, and organizational needs to propel North
College toward the innovation goal of the implementation of a systemic multilayered assessment
model for global education linking curricular and co-curricular student learning outcomes.
Assumed knowledge, motivation, and organizational needs were generated from related
literature, learning and motivation theories, and personal knowledge. The analysis of data from
this qualitative case study validated 18 needs that led to three recommended solutions. The
proposed solutions in Chapter Five provide concrete strategies and implementation plans for
addressing the knowledge, motivation and organization needs to achieving this innovative
systemic multilayered assessment model. The three solutions fall under two broad categories:
(A) establish measures and capacity of excellence to reinforce institutional accountability, and
(B) define, establish and encourage standards of excellence. The three solutions are (1) establish
clear goals, communication and oversight of assessment at North College; (2) establish
knowledge and capacity to capture student learning outcomes; and (3) establish a comprehensive
stakeholder assessment development program which includes a system of rewards and incentives
embedded within it. The paper concludes with an evaluation framework for monitoring the
effectiveness of the proposed solutions.

MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 11

CHAPTER ONE: OVERVIEW OF THE STUDY
Higher education institutions are challenged by a shift in public discourse around
assessing student learning, proving to accreditors that assessment measures are robust, and the
need to systemically measure increasingly complex student learning outcomes, such as global
awareness. As Robert Barr and John Tagg (1995) argued, the mission, vision, culture, and
structure of higher education institutions must undertake a paradigm shift from providing
instruction to producing learning. Barr and Tagg’s framework was broad, encompassed all
graduates, and outlined assessment of learning should be done without regard to a particular
curriculum or educational experience. The framework also focused on learning, not teaching,
and, when coupled with the public discourse around student learnings, causes institutions to shift
focus to determine how to reliably assess learning outcomes across all graduates as well as
throughout the institution as a whole.
Higher education institutions are increasingly pressured to internationalize. Articles
appear in the Chronicle of Higher Education, Inside Higher Education and mainstream media
such as CNN, BBC and others, and statements and centers focused on institutional
internationalization emerge from The American Council of Education (ACE), American
Association of Colleges and Universities (AAC&U), and Association of International Educators
(NAFSA). The ACE states that colleges and universities must “… be globally engaged and
prepare students to be citizens of a multicultural community both at home and in a globalized
world. Institutions accomplish this by having a multi-dimensional, comprehensive strategy that
includes internationalization at home and engagement with global issues and partners” (ACE,
2011, para. 5). Indeed, hundreds of U.S. campuses list global learning and intercultural
competence as part of their student learning outcomes, related to the AAC&U (2014).
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 12

Mirroring ACE describing the nature of internationalization as “multi-dimensional”, but
highlighting the challenges of assessment, a report by the Association of International Educators
(NAFSA, 2010) noted that international education happens on multiple levels including student,
program, institutional, regional, national, and international. Additionally, global education
happens in all parts of an institution from student learning to student and faculty research to co-
curricular. Global education is cross-cutting. NAFSA’s latest document, International
Education Professional Competencies (2015), discussed later, highlights these complexities in
the quickly changing phenomenon of internalization. As ACE likewise stipulates, it is the
infusion of this global perspective throughout an institution that is central to the concept of
internationalization. However, infusion of global education throughout an entire institution
suggests that assessment must be multilayered if institutions are to provide the learning outcome
data that the public, accreditors and others now want to see. Systemic accountability of student
learning outcomes relating to global education, or assessment, has increasingly proved
problematic due to the multi-dimensional nature of internationalization. It is, therefore,
imperative that assessments occur at the various levels as well as in the interaction among the
levels (NAFSA, 2010).
Organizational Context and Mission
North College is a private religious institution founded in the mid-1800s with an
undergraduate enrollment of approximately 2,650 and total enrollment of approximately 3,500
students. More than 30% of the students are the first generation of their family in college with
more than 40% identified as non-Caucasian. North offers more than 50 undergraduate majors
and nine graduate degrees and is ranked in the top Regional Universities Midwest by U.S. News
and World Reports. In addition to its main campus, North College has an additional site in
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 13

another city in the same state. Through the Institute for Global Learning, the college maintains
sites in other countries.
North’s mission highlights the institutional focus to educate students to be informed
citizens, thoughtful stewards, critical thinkers, and responsible leaders. Grounded in its religious
tradition and located in an urban setting with campuses in other countries, the experience at
North includes an engaged community committed to intentional diversity. The institution
defines its educational focus as excellence in both the liberal arts as well as professional studies.
The college links the mission directly to the all students’ learning outcomes. It
documents North states that, as responsible leaders, students will be able to explain diverse
positions as well as collaborate effectively across social, cultural and geographic differences.
The college furthermore states that students will demonstrate these skills and knowledge in both
local as well as global contexts.
North has defined, within the Institute for Global Education, global education as a
program that includes empowerment of the participants to work for personal and social
transformation, which is grounded in a direct dialogue that occurs between students and
members of local communities. At North, global education is inspired by Paulo Freire’s
philosophy of popular education and by experiential, feminist, indigenous, and postcolonial
pedagogies that foment transformation. The five pillars of global education in North’s
philosophy are (1) Intercultural, (2) Experiential, (3) Holistic, (4) Analytical, and (5)
Transformative.
Intercultural. Draws upon the diverse backgrounds of participants, students, faculty,
staff, and community members with a goal of raising awareness about one’s own
culture(s) and the complexities of diverse cultures within other countries. Because
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 14

knowledge is culturally constructed, programs broaden the base of what are commonly
considered to be valid sources of information by facilitating encounters with
marginalized people.
Experiential. Believes in reflection upon prior knowledge as one engages in new
experiences. Critical analysis of experiences is a must because experiences alone are not
truly educational. Programs combine a variety of activities such as story-telling, dialogue
with host families and a range of community members, small group work, excursions,
lectures, internships, films, and the use of music and the arts with other hands-on
experiences.
Holistic. Commits to both intellectual and emotional learning, encouraging
people to reflect upon how they learn best and to grow outside of their comfort zone.
Programs are developed with different learning styles in mind.
Analytical. Encourages participants to analyze multiple perspectives and to
develop their own viewpoints. Experiential interactions, combined with background
reading and research to determine the validity of information, provide the basis for
critical analysis of multiple sides of an issue.
Transformative. Believes that the purpose of education is to create a more just
and sustainable world. The goal is for learners to engage both in their own
transformation and to act to bring about the changes they envision for our world.
Transformative education leads to greater personal and social responsibility.
Organizational Performance Need
In order to fulfill its mission and ensure accreditation, North College has specific student
learning outcomes related to global education. However, they are imbedded within the
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 15

undergraduate general education learning outcomes measured individually in a class. As such,
there is no systemic, multilayered process for assessing whether the learning outcomes are
accomplished beyond the general education requirement or with non-undergraduate students.
Despite this, the college’s catalog states it expects students will achieve the general education
student learning outcomes cumulatively. Student learning is to be achieved through students’
general education courses, majors, minors, electives, and participation in co-curricular
experiences. However, there is no systemic, multilayered process for assessing this. The student
learning outcomes have a counterpart statement in the student affairs division that was
previously developed and currently not aligned to the academic student learning outcomes nor
measured alongside them in a systemic, multilayered process. Furthermore, the college states
that assessment of general education outcomes, which include global education and intercultural
competence, will be conducted across institutional divisions.
It is imperative that North College create a systemic and multilayered process to assess
whether global awareness student learning outcomes. Failure to do so can result in a loss of
accreditation as well as declining institutional rank from publically reported documents. Thus,
the organization needs a systemic method for monitoring and maintaining assessment across the
college divisions. Currently, no such system exists.
Related Literature
The ACE asserts that higher educational entities must incorporate global perspectives
into teaching, learning, and research (ACE, 2011). ACE outlines five areas to focus on across an
institution: (1) Internationalizing Student Learning; (2) Student Mobility; (3) Faculty Global
Engagement; (4) Global Partnerships; and (5) Articulating Global Strategy. ACE does not state
how to achieve this incorporation of global perspectives holistically within an institution.
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 16

Indeed, ACE acknowledges that, as the field of global education and engagement continually
develops, there is no single way in which it will manifest itself within U.S. higher education.
ACE, like other educational associations in the field, argues that this phenomenon of a
continually evolving model of global education will require new assessment mapping skills
across an institution. Arguably, this is a level of skills and mapping at a level of complexity that
is either now in development, or needed, at most U.S. higher educational institutions (ACE,
Campus Internalization, 2011).
Assessment has been a focus of U.S. higher education for at least three decades (Aghion,
Dewatripont, Hoxby, Mas-Colell, & Sapir, 2009). The value of assessment, as articulated by
Palomba and Banta (1999), lies in the opportunity to improve the practice of teaching. Further,
through an assessment process, institutions can examine whether their curricular offerings meet
their intended outcomes (Barr & Tagg, 1995). Additionally, it is a way to measure if students,
through all experiences, have the skills and knowledge they are expected to possess (Barr &
Tagg, 1995). Despite the centrality of assessment to higher education, assessment of global
learning is still in its infancy. For example, nowhere in ACE’s Report of the Blue Ribbon Panel
on Global Engagement Strength through Global Leadership and Engagement: U.S. Higher
Education in the 21st Century was the word “assessment” used in conjunction with student
learning and outcomes.
There are rubrics for measuring what students should know (outcomes) in the areas of
global learning and intercultural knowledge and competence, but not for how to measure those
outcomes in a multilayered assessment model (see Definitions at end of chapter). Between 2007
and 2009, the AAC&U engaged faculty and educational professionals from over 100 higher
education institutions to develop rubrics for the LEAP Essential Learning Outcomes. Faculty
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 17

tested the rubrics on over 100 college campuses. Today, more than four hundred U.S. campuses,
or almost 10% of higher educational institutions in the U.S., list global learning and intercultural
knowledge and competence as student learning outcomes implying they utilize the AAC&U
rubrics (AAC&U, 2009). However, systemic accountability or assessment of global education
proved increasingly problematic. As Robert Barr and John Tagg (1995) argued, institutions are
challenged to measure learning outcomes for all graduates in areas such as global education
across all curricula and co-curricular realms. The complexity of the assessment process
increases as institutions ensure all graduates have certain skills and knowledge related to their
major/minor, yet students are additionally expected to achieve broader outcomes regardless of a
particular experience or curricula track. In a review of three institutions that received national
awards for internationalization efforts, all mapped assessment of global learning outcomes to a
handful of majors and exclusively measured them within the academic realm (Green, 2013). As
such, despite winning the awards, and each spending millions of dollars in the development of
assessment, they failed to develop a measure of global learning for all graduates, let alone
attempt to measure the outcomes across various parts of the institution (Green, 2013).
A report released by NAFSA (2010) highlights the challenges multilayered assessment
can present in achieving. While in 2013 NAFSA awarded the national awards for global
education assessment, the institutions chosen had not achieved the level of assessment previously
set by NAFSA in 2010. NAFSA stated, in 2010, that the problem institutions face in assessing
global learning is the complexity of a multilayered approach given the breadth of activities
across an institution. In particular, the report stressed the challenge in measuring interaction
between the levels, not just within academics or to a specific group of students but to all
graduates and across the academic and co-curricular realms of institutions. The report states that
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 18

only such a model, which it argues for but does not provide guidance on, can truly be a holistic
model of institutional assessment of global learning (NAFSA, 2010). To date, the review of the
literature has found no institution with a systemic multilayered model of assessment of student
learning outcomes relating to global education.
Importance of the Organizational Innovation
It is important for North College to implement a systemic method for assessing global
education learning outcomes for a variety of reasons. First, the college needs to measure these
learning outcomes to submit proof of students meeting them to the Higher Learning Commission
of the North Central Association of Colleges and Schools (HLC). HLC requires proof of
achievement of all stated student learning outcomes for both overall accreditation as well as
periodic review of targeted outcomes (e.g., global education). Without HLC accreditation, the
college cannot offer degrees. Second, a systemic method for assessing global education learning
outcomes enables North College to confirm meets the goals stated in its mission statement, and,
to the extent it is not, the college can make informed and needed adjustments in teaching and
learning practices. Third, systemic assessment of global education learning outcomes helps
ensure that students acquire the skills necessary to function in the 21st century. Thus, failure to
assess and report student learning outcomes may lead to a loss in accreditation, and a negative
impact on the organization’s ability to achieve its mission.
North College, like other higher educational institutions, increasingly devotes resources
to measuring outcomes other than student learning. Policy instruments are more politicized, with
accountability measures focusing less on educational and learning outcomes than on other factors
linked to access and equity within the educational system (El-Khawas, 2008). The need,
however, at North College is to ensure that all students graduate having met learning outcomes in
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 19

global education. The institution already tracks other factors such as retention, diversity, gender;
it needs to focus its work on the measurement of student learning outcomes to align with its
strengths in the other non-academic outcome measures of assessment it is undertaking. It is
through this work that the quality of teaching and knowledge acquisition will improve, and it is
possible that other measures such as retention could increase.
Organizational Performance Goal
By September 2017, using the AAC&U Global and Intercultural Rubrics, North College
administrators will implement a multilayered assessment model for global education learning
outcomes in 100% of majors as well as 50% of activities of the division of student affairs.
Stakeholders and Stakeholders’ Performance Goals
North College has multiple stakeholders involved in advancing global education and
assessing global education outcomes. Primary stakeholders include faculty, student government
and college administrators. As global education is embedded within the curriculum, both at the
individual course level and across departments spanning majors and minors, faculty are critical
to advancing global education and to assessing student learning outcomes. A 2005 survey of
faculty and staff suggested there is awareness and understanding of assessment of student
learning; resistance to assessment may no longer be the greatest obstacle. Rather, availability of
information and resources as well as communication from the assessment committee seem to be
the problem, and perceptions and involvement in assessment remain at the individual and
department level rather than at the college level.
Student government at North advances educational goals and well-being of the college
community, supports and advocates for student concerns, needs, and activities and is the primary
student voice as well as liaison between students and the administration, faculty, and staff. The
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 20

student government has an official governing role at North, including representation on the
committees identified within the stakeholder performance goals.
College administrators involved in advancing global education and assessing student
learning outcomes include those who sit on the Academic Affairs Committee (AAC), the
Graduate Academic Affairs Committee (GAAC), the Assessment Committee and the Student
Affairs Sub-Committee. The AAC consists of 11 voting members: the dean of the college or
designee, nine members of the faculty, and a student selected by the student government. In
addition, the registrar, the library director, a person from the student affairs office (chosen by
student affairs), a person representing an international program of the college (chosen by the
dean), the director of academic advising and the director of the center for service, work, and
learning serve as advisory members. AAC is entrusted with the general oversight of the
academic program including matters of curriculum and academic planning, requirements for
admission and graduation, grading, academic calendar, academic policies, international
education, and evaluation of the academic program.
The GAAC consists of the following voting members: the dean of the college or dean’s
academic affairs committee designee, the associate dean of adult programs, one representative
from each graduate degree program, one at large faculty member, and at large graduate student
representing all graduate students. The registrar, the library director, and the director of graduate
recruitment serve as advisory members. GAAC is entrusted with the general oversight of the
graduate academic programs, including matters of curriculum and academic planning,
requirements for admission and graduation, grading, academic calendar, academic policies,
international education, and evaluation of the academic program.
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 21

The Assessment Committee is comprised of 11 members, including dean of arts and
sciences; five faculty representatives, one from each division, one from a graduate program, and
one at large; a faculty member of AAC, appointed by AAC; the director of assessment; the
associate dean for student affairs, or designee; a member representing the center for service,
work and learning, appointed by the director of the center; and a student appointed by the student
senate. A sub-committee will serve as a working team with the dean for the purpose of
analyzing, summarizing, and disseminating assessment reports. The charge of the Assessment
Committee includes oversight of student learning assessment in college academic programs;
coordination of the assessment of student learning in college co-curricular programs; advising on
developing and implementing assessment plans; involving faculty, staff and students in
assessment; identification and recommendation of resources the college needs to successfully
conduct assessment; monitor and implement and changes in assessment plans; and organize
workshops/activities for exchange of assessment ideas and for updates of department assessment
plans.
The Student Affairs Sub-Committee on Co-Curricular Assessment consists of 14
members, spanning the areas of student and academic affairs. The sub-committee is charged,
broadly, with the implementation of an established strategy for co-curricular assessment
supporting campus colleagues in growing a culture of assessment at North.

MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 22

Table 1
Organizational Mission, Organizational Goal and Stakeholder Performance Goals
North College Mission
North College educates students to be informed citizens, thoughtful stewards, critical
thinkers, and responsible leaders. The North experience is supported by an engaged
community that is committed to intentional diversity in its life and work. A North education
is defined by excellence in the liberal arts and professional studies, guided by the faith and
values of the Lutheran church, and shaped by its urban and global settings.
Organizational Assessment Goal
By September 2017, using the AAC&U Global and Intercultural Rubrics, North College
administrators will implement a multilayered assessment model for global education
learning outcomes in 100% of majors as well as 50% of activities of the Division of Student
Affairs.
Faculty Goal
By May 2017, faculty
will construct class-
specific global
education learning
outcome(s) for 100% of
courses.
Student Government
Goal
By January 2017, the
student government will
provide its
recommendations about
the multilayered
assessment model.
College Administrators sitting on
the AC/AAC/GAAC and Student
Affairs Sub-Committee Co-
Curricular Assessment Committees
Goal
By September 2017, North College
administrators will implement a
multilayered assessment model for
global education learning outcomes in
100% of majors, as well as 50% of
activities of the Division of Student
Affairs utilizing the AAC&U Global
and Intercultural Rubrics.

Stakeholder for the Study and Stakeholder Performance Need
The joint efforts of all stakeholders will contribute to the achievement of the overall
organizational goal of implementing a systemic multilayered assessment model for global
education learning outcomes in 100% of majors as well as 50% of activities of the division of
student affairs, utilizing the AAC&U Global and Intercultural Rubrics. It is important to
understand the knowledge, motivation and organizational needs of college administrators sitting
on the AC/AAC/GAAC and Student Affairs Sub-Committee as they attempt to implement a
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 23

model that aligns with institutional and accreditation expectations. Therefore, the stakeholders
of focus for this study were college administrators sitting on the AC/AAC/GAAC and Student
Affairs Sub-Committee.
The stakeholders’ goal, supported by the dean of global education, is that, by September
2017, a model of assessing global education learning outcomes in 100% of majors as well as
50% of activities of the division of student affairs will be completed. Compliance procedures
include activities such as committee attendance, monthly and summary progress reports,
assessment model development review, mapping of the curricular and co-curricular areas to be
assessed, and definition of the multilayered areas that will be focused on in the model.
Currently, no administrator sitting on the AC/AAC/GAAC and Student Affairs Sub-Committee
implemented a systemic multilayered assessment model because no such model exists. The gap
in performance, therefore, is 100%.
Purpose of the Project and Questions
The purpose of this study was to conduct a needs analysis in the areas of knowledge and
skill, motivation, and organizational resources necessary to reach the organizational performance
goal. The analysis began by generating a list of possible needs and then moved to examining
these systemically to focus on actual or validated needs. While a complete needs analysis would
focus on all stakeholders, for practical purposes, the stakeholders focused on in this analysis
were college administrators sitting on the AC/AAC/GAAC and Student Affairs Sub-Committee.
As such, the questions that guided this study were the following:
1. What are the knowledge, motivation, and organizational needs necessary for the
North College administrators as stakeholders to achieve their goal of implementing a
multilayered assessment model for global education learning outcomes in 100% of
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 24

majors, as well as 50% of activities of the division of student affairs, utilizing the
AAC&U Global and Intercultural Rubrics?
2. What are the recommended knowledge, motivation, and organizational solutions to
those needs?
Methodological Framework
Clark and Estes’ (2008) gap analysis, a systemic, analytical method that helps to clarify
organizational performance goals and identify the gap between the actual performance level and
the prescribed performance level within an organization, was adapted for innovation analysis.
Assumed knowledge, motivation and organizational needs were generated based on institutional
knowledge, learning and motivation theory and related literature. Surveys, interviews and
document analysis were used to validate or not validated these assumed needs. Research-based
solutions are recommended and a comprehensive evaluation plan is proposed in Chapter Five.
Definitions
Assessment: A philosophy of measuring student learning that privileges multiple expert
judgments and shared understanding of the quality of student work through the curriculum, co-
curriculum, and beyond over reliance on standardized tests administered to samples of students
disconnected from an intentional course of study (AAC&U, 2009).
Global Learning: AAC&U (2009) defines global learning as a critical analysis of and an
engagement with complex, interdependent global systems and legacies (such as natural, physical,
social, cultural, economic, and political) and their implications for people’s lives and the earth’s
sustainability.
Intercultural Knowledge and Competence: AAC&U (2009) defines intercultural
knowledge and competence as “a set of cognitive, affective, and behavioral skills and
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 25

characteristics that support effective and appropriate interaction in a variety of cultural contexts.”
(Bennett, 2008, p. 97).
Systemic multilayered assessment: An institutionalized process by which assessment is
measured across all students, as well as the institution as a whole, ensuring broad-based student
learning outcomes are met. This process, in particular, aligns assessment across institutional
areas such as academics for undergraduate and graduate students, co-curricular activities, and
assesses not only the individual area(s) but also the learning outcomes where the areas intersect.
A visual representation of systemic multilayered assessment is shown in Figure 1. A
visual representation of the stakeholders is shown in Figure 2.

MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 26

Figure 1. A Visual Representation of Systemic Multilayered Assessment

Multilayered
intersection areas of
Global Education
assessment
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 27

Figure 2. A Visual Representation of the Stakeholders
Organization of the Dissertation
Five chapters are used to organize this dissertation. This chapter provides the key
concepts and terminology commonly found in a discussion about accountability of student
learning outcomes relating to global education, or assessment thereof. The organization’s
mission, goals and stakeholders as well as the concepts of needs analysis adapted to needs
analysis were introduced. Chapter Two provides a review of current literature on the scope of
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 28

the study. Topics of the definition and expansion of global education, broad assessment
frameworks, review of student learning outcomes and specifically AAC&U rubrics, and models
of implementation of global education on a campus are addressed. Chapter Three details the
assumed causes for this study as well as methodology when it comes to choice of participants,
data collection and analysis. In Chapter Four, the data and results are assessed and analyzed.
Chapter Five provides solutions, based on data and literature, for addressing the needs and
closing the performance needs along with recommendations for an implementation and
evaluation plan for the solutions.

MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 29

CHAPTER TWO: LITERATURE REVIEW
Reviewed in this chapter are the history and growth of the field of global education,
current models of global education in higher education institutions, definition of global education
as a dimension in the curriculum, assessment of higher education programs, and both examples
of broad-based assessment as well as challenges in assessing global education. This chapter also
reviews learning and motivation theory, paying special attention to knowledge and skills,
motivational, and organizational issues. Finally, administration knowledge, motivation,
organizational factors, and professional training and development, are reviewed.
Global Education
History and Growth of the Field
The literature outlines broad trends for the continually shifting field of global education
(ACE, 2011, 2012; Knight, 2001, 2003, 2004; McCabe, 2001; NAFSA, 2010; Olsen, Green, &
Hill, 2005, 2006; Olson, Evans, & Shoenberg, 2007; Olson, Green, & Hill, 2006; Rudzki, 2000;
van der Wende, 2007; Wächter, 2003; Woolf, 2002). The shifts cited include complexity, speed
of development, integration into core curricular areas, and the globalization of higher itself.
The speed at which global education affects higher education is accelerating along with
the breadth of activities that embody globalization in higher education. It is, however, not a new
phenomenon. That there were activities prior to World War II is not in doubt, however post-
WWII is when several factors aligned to influence globalization in higher education. First, study
abroad sites reopened and expanded (Desruisseaux, 1999, 1999a, 1999b). Second, there was a
relationship of increased competition of global knowledge as a result of the Cold War broadly,
and Sputnik in particular (Desruisseaux, 1999, 1999a, 1999b, 2000; Goodman, 1999). Third, a
shifting within institutions around the philosophy of area studies, linked to the nationalistic needs
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 30

stemming from the Cold War and funding, was also a factor (Tarrow & Hall, 1998). There is
discussion about when globalization and higher education accelerated as the phenomenon spans
decades of interchanging activities and geographic diversity. However, consensus that the late
1990’s was a period of rapid development, in particular of new models of activities in a boarder
range of countries, coupled with preliminary definitions of the term globalization, are seen
(Altbach & Knight, 2007; Carr, 1999; Desruisseaux, 1999, 1999a, 1999b, 2000; Hudzik, 2011;
Knight, 1993 & 2004; Yang, 2002).
Higher education institutions, indeed, are bound to internationalize in a response to
globalization forces (Coughlin, 1994; Lipsman, 2000; Sassen, 2001; Yang, 2002). An early
definition of internationalization, which has altered and is addressed in the following section, is
“the process of integrating an international, intercultural or global dimension into the purpose,
functions or delivery of postsecondary education (Knight, 2004, p. 11).” Higher education has
been, and continues to, alter and evolve practices to integrate global components into academic
and co-curricular programs, research, and operations; in all three areas in the realms of direct
services, management, and strategy and policy (NAFSA, 2015). There is great variance, based
upon institutional type and other factors, of how globalization affects an institution (Yang, 2002).
A tendency toward business-related practices, including privatization, as well entrepreneurialism
coupled with an increase in breadth of the type of programs is often cited (Altbach & Knight,
2007; Mazzarol & Soutar, 2002; Stromquist, 2002). Another example of the evolution of the
field is a recent professional document (NAFSA, 2015) that highlights five broad areas of global
education: comprehensive internationalization; education abroad; international enrollment
management; international student and scholar services; and cross-cutting competencies.
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 31

Internationalization of institutional activities center around two factors, educational and
economic (Raby, 2009). First, institutions internationalize because they must provide an
education relevant to a changing world (Armstrong, 2007) this is both educational as well as
business in terms of drawing students with a curriculum viewed as applicable. Second, on the
business side, pressures in the form of increased competition from other institutions’
internationalization efforts, coupled with unchanging or declining student enrollment, both of
which force institutions to undertake new initiatives (Armstrong, 2007; Mazzarol & Soutar,
2002; Stromquist, 2002). Additional factors include peace building, national security,
reputational advancement, human capital development, expanded tertiary access, demographic
growth, and research collaboration (Altbach & Knight, 2007; Edwards, 2007; Hser, 2005;
Knight, 2004).
Current Definitions of Global Education in Higher Education Institutions
Current literature refers to global education with a range of definitions echoing the
models, practices, and programming (Altbach & Knight, 2007; Beerkens, 2003; Hudzik, 2011;
Kehm & Teichler, 2007; Knight, 2001, 2004; Matthews, 2002; Melanie, 2013; Olsen et al., 2005;
Olson et al., 2007; Olson et al., 2006; Whitsed & Green, 2014). There is increasing importance
of a broad range of models of global education within the literature. These models also
underscore divergence within the field of global education based upon many factors such as
institutional type, institutional mission, scale of operations, internationalization of the home
campus as well as study abroad, international students, and internationalization of the curriculum
(ACE, 2011, 2012; Burnett, & Huisman, 2010; Childress, 2009; Huang, 2007; Hudzik, 2011;
NAFSA, 2010; Olsen et al., 2005, 2006; Olson et al., 2007).
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 32

Definitions of internationalization of a higher education institution, not activity based like
those above, vary greatly. Olson et al. (2007) define internationalization in context of
multicultural knowledge of peoples and regions outside the borders of the United States. This is
coupled with understanding of the relationships between nation-states and of variety in and
amongst global trends and systems. Hudzik, defines internationalization as connecting higher
educational institutions with changing local and global environments, which, in turn, provides
relevant service to society while meets the needs of students during changing realities (Hudzik,
2011). These two examples are reflective of the great variance of definitions of
internationalization, incorporating the interconnected nature of the changing field, with the
education of students to meet this changing reality. Additional definitions are discussed later, in
particular the one utilized for this innovation study.
Current Activities of Global Education in Higher Education Institutions
Activities of global education include an array of initiatives dependent upon an
institution’s history, culture, resources and priorities (Yang, 2002). Historically, these included
foreign language acquisition, cultural studies, institutional partnerships, faculty exchanges, study
abroad programs and international student enrollment (Altbach & Knight, 2007; Armstrong,
2007; Farnsworth, 2005; Hser, 2005; Knight, 2004, 2011). Further activities that increased in
scope and scale within the last decade include the following. First are twinning or 2+2 programs,
or articulation agreements by which a student completes part of their degree at one institution
and transfers, with all credits, to the other for earning the degree. Second are joint or dual degree
programs where two institutions in partnership offer a program wherein the student earns one or
two degrees. Third are for-profit programs, such as gap year or bridge English programs and
certificate programs. Fourth are collaborative international research (profit and non-profit)
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 33

projects. Fifth are franchising arrangements where institutions allow their name and courses to
be used on a per fee basis. Sixth are education hubs, often policy-driven entities attempting to
bring foreign institutions into a single location resulting in a “hub” of educational options.
Seventh is international recruitment agent usage. Eighth is branch campus establishment, which
includes both full-degree branches of the home campus overseas and campuses which offer a
broad range of studies but require returning to the home campus to earn the degree (Altbach &
Knight, 2007; Armstrong, 2007; Farnsworth, 2005; Hser, 2005; Knight, 2004, 2011).
International branch campuses (IBCs) and offshore collaborations, some but not all with
foreign partners, were hypothesized as the next generation of internationalization activity
(Armstrong, 2007). What makes IBCs unique is that, in some cases, they are transformed
beyond a branch campus into full-degree-granting entities that may simultaneously serve as a
branch campus. IBCs, therefore, may contain some or all of, the globalization activities of the
home campus, as well as position the institution into the local market of students easier than the
home campus would.
Current and Evolving Models of Global Education in Higher Education Institutions
Given the breadth of activities that can be undertaken, in tandem or alone, including
hypothesized next generation, of global education there are various theoretical models which
attempt to articulate a framework for the field. Qiang (2003) provides a model that categorizes
approaches to internationalization. Overarching the framework, and defining the categorization,
is the stance taken by senior leadership regarding institutional internationalization. First is
activity approach focused on student and faculty exchanges as well as international students.
Second is ethos approach including the embrace of, and placement of value in,
internationalization by the organization. Third is process approach advancing infusion of
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 34

service, teaching, and research into the institutional processes. Fourth, is competency approach
when an organization examines skills, knowledge, attitudes, and values of all stakeholders. This
approach acknowledges the need for organizational analysis and professional development of
stakeholders to have skills and knowledge required for both internationalization as well as
achievement of global education student learning outcomes (Qiang, 2003).
Current models of global education broadly incorporate in some manner, the five
previously mentioned domains of activities as defined by NAFSA in 2015 of comprehensive
internationalization with five sub-categories; education abroad with seven sub-categories;
international enrollment management with six sub-categories; international student and scholar
services with five sub-categories; and cross-cutting competencies with eight sub-categories
(NAFSA, 2015). Of note are particular sub-categories that cut across these domains, including
“Internationalization at Home (Curricular and Co-curricular)” and “Intercultural
Communications” (2015). Despite being sub-categories, these each include many historic
activities comprising study abroad, bi-lateral exchange, internationalization of the curriculum,
expanded co-curricular programming with an intercultural and global focus, student and faculty
research, and faculty and staff exchange. Institutions may also select a theoretical framework, as
those outlined above by Qiang, to provide a philosophical and pedagogical value to the activities
undertaken. There is, however, no single model or framework, as such models are linked to
institutional mission and culture.
Simultaneously, a growing body of evidence defines models of best practices in
academic, institutional, risk management, and financial management of global education
programs. The U.S. Department of Justice and the Federal Trade Commission recognized The
Forum on Education Abroad as the standards development organization for the field of education
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 35

abroad (The Forum on Education Abroad, 2009, 2011). The Forum is also deploying the second
versions of a QUIP accreditation process for institutions to achieve certification in best practices
(The Forum on Education Abroad, 2009, 2011, 2015). Most recently, the NAFSA board of
directors tasked its Professional Education Pathways Task Force with identifying the skills,
knowledge, and competencies for professional practice areas within international education.
This document was based upon feedback gathered from NAFSA member focus groups,
knowledge community team reviews, a survey of 2,000 international educators, and an invited
group of reviewers. The result is NAFSA International Education Professional Competencies
(NAFSA, 2015), designed to be utilized by professionals ranging from advisors to senior
international officers focusing on direct services, management of programs and activities, and
strategic and policy development across five domains in international education.
Evolution of Definition of Global Education as a Dimension in the Curriculum
In recent years, global education moved into being focused not solely on overseas
activities or international students, but also embedded into the curriculum as a student learning
outcome (SLO) for all students. As this shift has occurred, it has offered increased complexity as
well as opportunity to ensuring that SLOs are, indeed, embedded into the curriculum. The
complexity arises in ensuring that SLOs are not only achieved but also mapped across an entire
curriculum as well as, ideally, to the co-curricular as well.
For the purpose of this dissertation, which is focused on the assessment of measuring
global education SLOs, the AAC&U definition of global learning is utilized. The AAC&U
definition focuses not just on experiences overseas, but also on the educational and co-curricular
experiences students can have on the home campus. AAC&U defines global learning as a
measurable assessment rubric focused on “the critical analysis of and an engagement with
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 36

complex, interdependent global systems and legacies (such as natural, physical, social, cultural,
economic, and political) and their implications for people’s lives and the earth’s sustainability”
(AAC&U, 2009, para. 2). Such a definition aligns to the cross-cutting nature of the activities
that students can undertake in the previously mentioned domains and activities while allowing
for assessment of SLOs in areas that are both distinctive (e.g., undergraduate, graduate) as well
as holistic in approach institutionally (e.g., curricular, co-curricular, research). As such, the
definition, and rubric, provide an evolving framework that encompasses a holistic, or multi-
dimensional, approach to global education. Additionally, the rubric specifically states that SLOs
in this field are cumulative and cannot be achieved through one assignment or, likely, one course
(2009).
Expanding Views of Assessment in Higher Education
Typical Models of Assessment in Higher Education
A broad review of SLOs assessment is required to provide context. SLOs define
institutionally expected knowledge, skills, and attitudes of students that are developed as a result
of a learning experience (Suskie, 2004). Institutions must determine whether the learning
experiences offer sufficient opportunities to produce the intended learning, select assessment
measures (e.g., rubrics), and then assess if evidence of SLOs data matched the institutional
expectations. Results should, if the SLOs are not being met, result in review and improvement
of existing programs as well as possible creation of new programs and opportunities as needed
(Olson et al., 2005).
Typically, models of assessment are one-dimensional, meaning they assess SLOs across a
major, minor, general education requirement, co-curricular, or similar areas. Such models assess
only a slice of the entire student experience. For example, a standardized test such as the
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 37

Collegiate Learning Assessment (CLA) returns a view of a sample of students at a particular
time, and is voluntary. As such, it assesses some learning of some students. Research shows
that students are not motivated to do their best work on standardized tests such as the CLA
(AAC&U, 2009). Additionally, psychometric practice discards the theory of utilizing any single
measure as a proxy either for individual student competency or for institutional evaluation
(2009). Finally, assessment from a particular test, as it is disconnected from specific curricula,
offers little assistance to identify specific areas in which to focus efforts to achieve higher levels
of mastery for students as well as faculty.
It is, also, typical that such models of assessment measure SLOs either through the
passing of a class and the learning outcomes embedded in it or through an evaluation at the end
of a term or year. The assessment of SLOs across dimensions as well as cumulatively across a
student’s entire higher educational experience is, however, starting to occur in limited cases.
Recent Movements Toward Broad-Based Assessment
Literature highlights a movement toward broad-based assessment, possibly due to
students’ being expected to master knowledge that overlaps disciplines, learning experiences that
are increasingly diverse, and learning experiences that are increasingly occurring outside of
traditional classroom setting (Alexander, 2000; Astin & Antonio, 2012; Filback & Green, 2013;
Linn, Baker, & Dunbar, 1991; Sadler, 2005; York, 2003). Global education, in particular,
touches on all three of these possible impetuses. There is, however, lack of a common definition
of what criteria‐ based broad-based assessment means or what it implies for practice (Alexander,
2000; Astin & Antonio, 2012; De Wit, 2010; Linn et al., 1991; Sadler, 2005; York, 2003). The
significance of formative assessment in measuring SLOs is generally acknowledged, but it
remains in its infancy, not always clearly defined across higher education (Alexander, 2000;
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 38

Astin & Antonio, 2012; Yorke, 2003). In recent years, there has been an increasing emphasis on
assessment results as well as on what use is made of the results. As such, there is increasing
interest in alternative forms of assessments, particularly complex assessments of SLOs
(Alexander, 2000; Astin, & Antonio, 2012; De Wit, 2010; Linn et al.,1991; Yorke, 2003).
For the purposes of this study, a broad-based assessment framework developed by the
AAC&U is explored more deeply. This framework includes rubrics for both Global Learning
and Intercultural Knowledge and Competence, and, the SLOs at North College are based upon
these rubrics.
Association of American Colleges and Universities. The AAC&U is a national
association concerned with the quality, vitality, and public standing of undergraduate liberal
education. Founded in 1915, today AAC&U comprises more than 1,300 member institutions of
all types and sizes, dedicated to extending a 21
st
century definition of liberal education to all
students regardless of academic specialization or intended career. AAC&U coordinates its effort
with member institutions around four broad goals: (1) LEAP: Liberal Education and America’s
Promise, (2) Quality: 21st Century Markers for the Value of U.S. Degrees, (3) Equity:
Innovation, Inclusive Excellence, and Student Success, and (4) Social Responsibility: Integrative
Liberal Learning for the Global Commons (AAC&U, 2014).
This section discusses two initiatives of AAC&U that fall under LEAP, a campus action
and research initiative built around a twenty-first century liberal education. First is VALUE
(Valid Assessment of Learning in Undergraduate Education) or learning assessment that
acknowledges multiple expert judgments and shared understanding of the quality of student work
through the curriculum, co-curriculum and beyond rather than a dependence on completing
standardized tests disconnected from a planned course of study. Second is a rubric development
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 39

project, linked to VALUE, which provides means of assessment for twenty-first century liberal
education.
Liberal Education and America’s Promise (LEAP). LEAP is a campus action and
research initiative built around a twenty-first century liberal education, responding to the
changing demands for more college-educated workers and more engaged and informed citizens
with higher levels of learning and knowledge as well as strong intellectual and practical skills
(AAC&U, 2007). LEAP was renamed to “LEAP: Liberal Education as a Global Necessity”
(AAC&U, 2014). This initiative supports the previously discussed factors guiding
internationalization, educational and economic, and aligns them to a curricular and co-curricular
plan for an institution to meet these factors while addressing students learning outcomes.
LEAP is structured around a set of “Essential Learning Outcomes” which are developed
for students by a 21
st
Century definition of liberal education (AAC&U, 2009). Described in
College Learning for the New Global Century, these outcomes are the first part of a framework
to measure students' cumulative development through college. VALUE, is an initiative of
AAC&U recognizing a growing importance placed upon learning assessment that acknowledges
multiple expert judgments and shared understanding of the quality of student work through the
curriculum, co-curriculum, and beyond rather than a dependence on completing standardized
tests disconnected from a planned course of study (AAC&U 2009; Bloxham & Boyd, 2007;
Nygaard et al., 2009; Palomba & Banta, 1999; Rhodes & Finley, 2014; Shupe & Shupe, 2007).
Building upon this philosophy, VALUE is an institutional-based assessment initiative as
part of the LEAP initiative, that provides tools to assess students' own authentic work to
determine whether and how well they are progressing toward graduation-level achievement in
learning (AAC&U, 2013). The sections below comprise the four learning outcomes.
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 40

The first learning outcome is principles of excellence (AAC&U, 2013). Current literature
refers to an era of educational reform and renewal for higher education (AAC&U, 2007; Palmer,
Zajonc, & Scribner, 2010; Tierney & Hentschke, 2007; Vukasović, 2012). The principles of
excellence are intended to be the second part of a framework that influence practice across the
disciplines as well as in general education programs regardless of institutional type during this
period of innovation (AAC&U, 2014).
The second learning outcome is high-impact educational practices (AAC&U,2013).
There is a growing importance within the literature upon assessing high-impact educational
practices that help students achieve essential learning outcomes within, as well as outside of, the
traditional classroom setting (AAC&U, 2014; Brownell & Swaner, 2012; Kuh, 2012; Kuh &
O'Donnell, 2013). Examples of high-impact educational practices include 1) First-Year
Seminars and Experiences, 2) Common Intellectual Experiences, 3) Learning Communities, 4)
Writing-Intensive Courses, 5) Collaborative Assignments and Projects, 6) Undergraduate
Research, 7) Diversity/Global Learning, 8) Service Learning, Community-Based Learning, 9)
Internships, and 10) Capstone Courses and Projects (AAC&U, 2014). Data are exhibiting, as
seen in Tables 2 and 3, the positive correlation between high-impact practices and student
learning. Additionally, while only study abroad is clearly international, based upon the AAC&U
rubric, it could be imbedded in all the other high-impact activities (i.e., intercultural learning
communities, internships with a global focus, service learning overseas, research with a global
focus).

MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 41

Table 2
Relationships Between Selected High-Impact Activities, Deep Learning, and Self-Reported Gains
Deep Learning Gains:
General
Gains:
Personal
Gains:
Practical
First Year
Learning
Communities
+++ ++ ++ ++
Service
Learning
+++ ++ +++ +++
Senior
Study
Abroad
++ + + ++
Student-
Faculty
Research
+++ ++ ++ ++
Internships ++ ++ ++ ++
Service
Learning
+++ ++ +++ +++
Senior
Culminating
Experience
+++ ++ ++ ++
+ p<0.001, ++ p<0.001 & Unstd B > 0.10, +++ p<0.001 & Unstd B > 0.30
AAC&U 2013

MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 42

Table 3
Relationships Between Selected High-Impact Activities and Clusters of Effective Educational
Practices
Level of
academic
challenge
Active and
collaborative
learning
Student-
faculty
interaction
Supportive
campus
environment
First Year
Learning
Communities
+++ +++ +++ ++
Service
Learning
+++ +++ +++ +++
Senior
Study
Abroad
++ ++ ++ ++
Student-
Faculty
Research
+++ +++ +++ ++
Internships ++ +++ +++ ++
Service
Learning
+++ +++ +++ +++
Senior
Culminating
Experience
++ +++ +++ ++
+ p<0.001, ++ p<0.001 & Unstd B > 0.10, +++ p<0.001 & Unstd B > 0.30
AAC&U 2013
The third learning outcome is authentic assessments (AAC&U, 2013). Current
literature documents the positive outcomes of authentic assessments, which examine whether
students can apply what they learned to complex problems and real-world challenges (AAC&U
2014; Fischer & King, 1995; Janesick, 2006; Hart, 1994; Rhodes & Finley, 2014).
The fourth learning outcome is inclusive excellence (AAC&U, 2013). The action of
making access, student success, and high-quality learning inclusive of all students, or inclusive
excellence, requires that institutions discover inequities in student success, distinguish effective
educational practices, and construct such practices into their missions and institutional operations
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 43

for sustained change in the areas of diversity, equity, and educational quality efforts into their
missions and institutional operations (AAC&U, 2013).
Rubric Development Project. Part of AAC&U’s LEAP initiative, the VALUE rubrics
are central to assessment of college student learning, currently utilized by approximately 10% of
U.S. higher educational institutions (AAC&U, 2015a). The VALUE rubrics were established to
assess the development, and application of, 16 learning competencies. The templates assist in
the assessment of, embodied via a collection of student work (i.e., a paper, a performance, a
project, etc.), each rubric can be modified to also assess individual student work. Each rubric
addresses five to six key criteria for a competency (e.g., quantitative literacy) with the assessor
selecting from amongst the descriptors as a measure of competence demonstrated by the student
work. By assessing students’ work done in their curricula across time, it is believed the
assessment measures a broader sample of how a student’s knowledge and skills grew and
matured during college (AAC&U, 2009).
Since the release of the rubrics in 2009, more than 32,000 first-time individuals visited
the VALUE website between June 2010 and January 2014. Representative of the scope of their
impact, the rubrics were assessed by more than 5,600 discrete institutions, including schools and
higher education associations as well as 3,300 colleges and universities in the United States and
around the world (2015).
This new approach to general education is a dramatic departure from current practice
(2015.) The rubrics are meant to offer a theoretical framework by which each institution can
develop principles to develop general education curricula that focus on core proficiencies,
intentional educational pathways within and across the institution, and that allows assessment of
students’ engagement through demonstrated accomplishments in inquiry- and problem-based
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 44

learning. In particular, the AAC&U rubrics, as in Global Learning and Intercultural Knowledge
and Competence, connect general education to both issues in larger society as well as to
students’ major fields. Through the rubrics, students should demonstrate ability to become
informed, open-minded, and responsible individuals who are attentive to diversity across the
spectrum of differences; understand how their actions affect both local and global communities;
and address the world's most pressing and enduring issues collaboratively and equitably
(AAC&U, 2014). The global education and Intercultural Knowledge and Competence rubrics
are summarized below, and the entire rubrics can be viewed in Appendix D.
Intercultural knowledge and competence. The AAC&U defines Intercultural
Knowledge and Competence in its rubric as “a set of cognitive, affective, and behavioral skills
and characteristics that support effective and appropriate interaction in a variety of cultural
contexts,” drawing from Bennett’s 2008 work Transformative Training: Designing Programs for
Culture Learning (AAC&U, 2014). The rubric, as part of the LEAP and VALUE initiatives, is
meant to be broad in nature focusing on both cognitive as well as behavioral skills that are
measured over a students’ entire academic career.
Global learning. Global learning as a rubric is defined by the AAC&U (2014) as the
“critical analysis of and an engagement with complex, interdependent global systems and
legacies (such as natural, physical, social, cultural, economic, and political) and their
implications for people's lives and the earth's sustainability”. Through the rubric, students should
demonstrate ability to become informed, open-minded, and responsible individuals who are
attentive to diversity across the spectrum of differences; understand how their actions affect both
local and global communities; and address the world's most pressing and enduring issues
collaboratively and equitably (AAC&U, 2014). Again, aligned to the LEAP and VALUE
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 45

projects, this rubric is broad enough to map across all academic disciplines, as well as focus on a
students’ demonstrated assessment of cognitive and behavioral analysis of complex ideas which
are synthesized across disciplines.
Challenges in Assessing Global Education in Higher Education Institutions
If colleges and universities are able to truly assess a student’s demonstrated ability in
global education, as defined by the institution, it may be able to claim with integrity its students
met the institutional SLOs (Braskamp et al., 2009). The literature highlights that differing
models of global education resulted in difficulty assessing the field (ACE, 2011, 2012; Green,
2002, 2012, 2013; NAFSA, 2010).
Despite the movement toward best practices, any institution may call its initiatives and
activities international programs, or global, which are just two of the current terms used.
Armstrong (2007) advocates the terminology used by institutions is less important than the need
to clearly identify the reasons for which to globalize programs. As is discussed in subsequent
sections regarding organizational needs and knowledge, this implies the purpose of global
education should be embedded in, and derived from, institutional mission and SLOs to ensure
assessment of the stated program outcomes. Attainment of the mission of a global experience, or
perspective, can only be assessed by SLOs, not program title.
Given the breadth of activities, and theoretical frameworks, which are coupled with
institutional mission and resources; there is no singular assessment need, or model, to succeed
with the need of multilayered global education assessment. Rather, as individual institutions
develop internationalization initiatives consistent with strategic plans, missions, values, and
resources (Hudzik, 2011) each then attempts to create assessment models to capture outcomes.
With over 4,300 degree-granting institutions in the United States, enrolling more than 18 million
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 46

students (Hudzik, 2011), there is no single process for assessment of SLOs relating to global
education, let alone a multilayered model able to be deployed across multiple institutions.
Compounding this issue is the division of goals into areas of faculty and administrative
control. Institutions must identify specific global education knowledge, skills, and attitudes
students are expected to demonstrate via assessment, which typically falls under the purview of
faculty defining SLOs at an institutional, divisional, and programmatic level. However, the
criteria by which to recognize the achievement of these outcomes could be in the purview of
faculty, and/or administration. How this criterion will be measured (e.g., by rubrics of student
work, career success) will need to be captured through complex assessment software and
administrative support, falling under the purview of administration (Olson et al., 2005).
Academic Programs
There is growing importance within the literature on building models of assessment that
measure global education practices which help students achieve measureable learning outcomes
within, as well as outside of, the traditional classroom setting (AAC&U, 2007 & 2009; Barr &
Tagg, 1995; Gacel-Ávila, 2005; Friedlander & Serban, 2004; Green, 2002, 2012 & 2013; Jones
& Killick, 2013; Leask, 2001; Mestenhauser, 1998; NAFSA, 2010; Nygaard, Holtham &
Courtney, 2009; Palomba & Banta, 1999; Schoorinan, 1999; Shupe & Shupe, 2007).
There are a multitude of ratings and rankings that provide comparative data on inputs,
activities, resource outputs (including articles published), enrollment rates, funding, resources
used, and number of programs mounted in global education. These performance indicators do
not, however, provide evidence as to whether or not students actually develop the knowledge and
skills promised by stated learning outcomes. The Institute for International Education Annual
Open Doors Report (IIE, 2014) is one example of such an indicator, which started reporting data
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 47

on incoming international students to the United States in 1919, and outbound U.S. credit
bearing study abroad students in 1985 (IIE). Another, GLOSSARI: The Georgia Learning
Outcomes of Students Studying Abroad Research Initiative, while looking at SLOs did so only in
relation to study abroad programs (Georgia, 2010). Even reports such as US News and World
Report and others publish numerical indicators relating to global education, but not student
learning. An absence of comparable learning outcome assessment indicators across institutions
leaves only ratings and rankings. However, they do not provide evidence of student learning in
global education holistically (Nusche, 2007).
Student learning assessment goes beyond the definition of SLOs. For global education,
assessment requires gathering evidence that demonstrates student achievement of specific
competencies as a result of educational experiences abroad as well as on the campus, and similar
measures in the co-curricular realms regardless of location. Green and Olson (2003) state
institutions rarely accomplish this. They propose a framework to guide institutional
measurement of the student outcomes. First, identification of indicators to be measured to
provide evidence of student achievement of articulated SLOs. Second, identification of process
that gather evidence at the course, program, and institutional levels. Third, is the identification
of campus experts to formulate assessment strategies (2003). This framework furthermore
requires comprehensive institutional support to ensure program analysis of the measurement
process to assure SLO goals are supported across levels by actual evidence of successful
achievement of the stated learning outcomes.
There is no one specific agreed upon list of global education SLOs. There has, over time,
become some merging of themes amongst various institutions such as AAC&U, NAFSA, Forum
on Education Abroad, and ACE. These rubrics are broadly applicable to a range of academic
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 48

departments, undergraduate as well as graduate education, can be applied in the U.S. and
overseas, and also to co-curricular activities. Application of many of the rubrics is both germane
to specific course level measurement indicators, as well as across a student’s entire academic
career (Green & Shoenberg, 2006).
Learning outcomes should be observable, demonstrable, and measurable (Spady, 1998).
Higher education institutions should articulate expected SLOs as clearly and concisely as
possible, so that assessment can be achieved. Failure to do so, often results in SLOs that are not
measurable, and comparative assessment is not feasible (Nusche, 2007; Melton, 1996).
Student Affairs Programs
A body of evidence documents models for measuring global education learning outcomes
associated with effective student affairs or co-curricular, programs as higher educational
institutions move to globalize their campuses (Green, 2012; Leask, 2008; Lewis & Niesenbaum
2005; Sample, 2013; Soria & Troisi 2013). Likewise, a substantial body of evidence documents
models for measuring learning outcomes, in general, associated with effective student affairs or
co-curricular programs that are not specific to global education (Barr & Tagg, 1995; Blimling,
2013; Lindsay, Stroud, & Tubbs, 2013; Schuh & Upcraft, 2001).
For institutions to assist students in their achievement of global education SLOs,
connecting co-curricular and student life experiences with the academic may well be an
influence. Studies show that the connection between academic practices and the student
experience on campus will have an impact on engagement and learning outcomes as well as on
how these affect the associations students make between in-class theory and out-of-class
experiences (Umbach & Wawrzynski, 2005; Braskamp, Trautvetter, & Ward, 2006).

MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 49

Other Programs
Current literature refers to a need for models of learning assessment that measure global
education that cross within the classical domains of curricular and co-curricular programs as well
as with other types of programming (AAC&U, 2007; Barr & Tagg, 1995; Deardorff, 2006;
Friedlander & Serban, 2004; Green, 2012, 2013; Jones & Killick 2013; Leask, 2008;
Mestenhauser, 1998; NAFSA, 2010; Schoorinan, 1999; Shupe & Shupe, 2007; Soria & Troisi
2013; Stone, 2006).
SLOs may be crafted for measuring cognitive outcomes (e.g., recall or recognition of
knowledge, development of intellectual skills and abilities), ranging from domain-specific
knowledge to the most general of reasoning and problem-solving skills (Shavelson & Huang,
2003). Non-cognitive SLOs are usually crafted for the development of values or changes in
beliefs (Ewell, 2005). The latter can be more complicated to measure skills and knowledge, and
global education SLOs frequently measure both areas. Other factors may also affect the
assessment of a specific program and lead to the achievement of the SLOs, including social
maturation, generational effects, and specific life events (Pascarella & Terenzini, 2005). Hudzik
(2011) also offers institutional level indicators for measurement of SLOs, including the number
of students who achieve identifiable knowledge competency; program impact on students;
capacity to learn from and with others from different cultures; and number of students who earn
international certificates.
Knowledge, Motivation and Organization Models
If needs exist in terms of meeting goals and strategic plans for change within an
institution, effective assessment of SLOs may be at risk along with the development of new
measures of assessing SLOs. Clark and Estes (2008) provide a framework to identify the needs
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 50

in performance leading to successful development of multilayered assessment models for global
education. The needs analysis process model (Clark & Estes, 2008) is based on five stages in the
analysis of performance needs:
Stage 1: Identify measurable organizational goals.
Stage 2: Identify quantity and quality assessments of the needs to achieve the goal.
Stage 3: Develop a strategy for analyzing the contribution of each of three causes (knowledge
and skills, motivation, organizational culture) for each need.
Stage 4: Plan and implement a system-wide and individual needs-based solution(s).
Stage 5: Assess solution(s) for continuous improvement.
An in-depth discussion of the framework is provided in Chapter Three. This quantitative
study, based on formative evaluation, used Clark and Estes (2008) needs analysis process to
examine performance needs leading to successful development of systemic multilayered
assessment models for global education.
Organizational Goals and Mission
Using the Clark and Estes (2008) model, institutions of higher education can identify
global education assessment goals that are specific, challenging, and up-to-date in order to meet
the need of a multilayered assessment model. The performance need(s) must be clearly
identified and analysis undertaken to ascertain where the institution is in relationship to the
need(s). After need(s) are identified, solution(s) may be developed that effectively address the
need(s). Only after such an analysis is complete, should execution of solutions and measurement
of the strategies implemented be initiated.
As discussed in Chapter One, institutional mission is the beginning of the needs analysis
process, as the mission should lead change based upon institutional need (Braskamp et. al.,
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 51

2006). Mission is the underpinning for the articulation of institution-specific SLOs (DeJong,
1992) moved into approved learning outcomes and rubrics assigned to measure them. While
stated learning outcomes are mapped across an institution, and may not even be known to
students, it is institutional mission that leads faculty, administrators, and students to comprehend
the broad institutional goals (Braskamp et al., 2006). Institutional missions vary greatly, based
on religious, ideological, or philosophical values regarding learning, teaching, and service to the
greater good (Kuh, Schuh, Whitt, & Associates, 1991). As such, through the assurance of SLOs,
institutions embed organizational goals in their mission (Kuh et al., 1991)
Organizational needs are quantifiable needs based upon what is important to an
institution. The needs, articulated as goals, provide a foundation for accountability to the entire
institution in myriad areas. With clear goals, all parties in an institution should be able to accept
responsibility for accountability, and through assessment define achievement of the goal(s).
Motivation drives goals. Institutions must determine the motivation for assessment of global
education, and how to assess it, in order to first identify organizational needs, and then determine
goals to attain it. While these needs may be found in areas of SLOs, as previously discussed in
Chapter One, the breadth of global education assessment can also have institutional effects on
revenue and markets, research, scholarship, service, a range of global activities, and on
advancing institutional reputation (Hudzik, 2011).
Performance Needs
A central component of needs analysis process (Clark & Estes, 2008) is assessment of
factors that may be creating the need(s). These factors include three areas: knowledge and skills,
motivation to achieve the need, and addressing possible organization barriers. These three areas,
while interacting, are distinct. In order to successfully attain organizational needs, or learning
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 52

outcomes, the factors must be aligned. To affect change or innovation, or meet institutional
needs, analysis of these three factors must occur to determine the root causes (Clark & Estes,
2008).
Knowledge and Skills
Krathwohl (2002) outlines four fundamental types of knowledge: factual, conceptual,
procedural, and metacognitive. Factual knowledge is the knowledge of facts, details, and
elements. Conceptual knowledge is knowledge of principles, theories, and models. Procedural
knowledge is familiarity with procedures, techniques, and methods. Metacognitive knowledge is
knowledge of cognition and awareness of one’s own actions and thoughts.
Motivation
Clark (1999) categorizes problems using the motivation indexes of persistence, active
choice, and mental effort. Persistence is an individual motivation to start a goal. However, the
individual is possibly distracted by other goals or interests, and, as such, ceases work on the
initial task. Active choice is having intent to act, however, failing to follow through on the action
to do so. Mental effort is individuals failing to utilize new knowledge to resolve a problem.
Organization
An organization’s culture can be analyzed based on the cultural settings and cultural
models that exist in it (Gallimore & Goldenberg, 2001). Cultural settings are concrete and
include the employees, their tasks, how and why tasks are completed, and the social context in
which their work is performed. Cultural models refer to cultural practices and shared mental
schema within an organization (Gallimore & Goldenberg, 2001).
Organizational barriers could block attainment of an institutional goal. A review of
literature in the concept of organizational culture is necessary in order to analyze institutional
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 53

challenges. Culture signifies the collective values, character, mission, and identity of an
organization or group of people (Tisdell, 2003). Within an institution of higher education,
culture is developed through a shared shaping of values, practices, beliefs, and assumptions.
These shared norms provide a frame of reference and guide the institution, both holistically as
well as within groups (e.g., faculty, staff, administrators, students). It is, then, through the lens
of this organizational culture by which meaning and interpretation is attached to events and
actions (Kuh,1993). Organizational culture is a model that is formed through group solving
problems and via their shared experiences. These experiences ultimately lead to communal
assumptions regarding organizational processes for change and innovation. These norms, over
time and with practice, become valid and spread to new individuals who join the organization.
The organizational frame interpreting events and actions become embedded in the organization’s
cultural values and history (Schein, 2004).
The ethos of an institution is portrayed through its culture. Institutional culture
incorporates shared vision for the future and, as such, creates and sustains an organization to
meet its desired goals. Social norms, including daily operations, of the organization likewise
become part of the culture. The institutional mission, however, is the underlying determinant
and focus of the culture of the institution (Braskamp, et al., 2006). In the literature, there is an
implication that an effective organization thrives on strong culture, which tends to be common to
higher educational institutions, as a predictor to effective performance (Schein, 2004).
When institutional change is considered, an understanding of the organization’s culture is
required to understand the needs that are necessary for the change process. An institution’s
group members comprise a culture that has stabilized with a sense of group identity and tasks.
These cultural factors help provide meaning and predictability, which the change will alter, and
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 54

needs assessed to move the culture trough the change. Culture affects all levels of operation
within an organization (Schein, 2004). The change could affect the culture of the entire
organization, or only subgroups, and as such the needs of each should to be assessed separately.
Administration Knowledge, Motivation and Organizational Needs to Develop Global
Assessment in Higher Education Institutions
Knowledge and Skills
Knowledge is categorized as factual, conceptual, procedural or metacognitive dependent
upon the complexity needed to process information (Anderson & Krathwohl, 2001). Knowledge
refers to the ideas or understandings that an entity creates and/or possesses that are used to take
effective action to achieve the entity’s goals (Yeh, 2011).
Applying Krathwohl’s (2002) revision of Bloom’s taxonomy to this needs analysis, a
number of factors might contribute to the organizational innovation. It is unclear if the
stakeholders underwent meaningful training programs on assessment or the AAC&U rubrics
linked to SLOs that connect with prior knowledge or by which factual retention improves.
Literature suggests extensive training as a means of remedying factual knowledge deficiencies,
as well as citing performance improvements after training in conceptual as well as procedural
knowledge (Allwood, Hemlin, Martin, & Muckenhuber, 2007; Miller & Cohen, 2001; Coughlan
et. al., 2002; Schraw & McCrudden, 2013).
The need is problematic, and the global education assessment innovation required, given
the growing importance of assessment in both the academic and co-curricular realms (AAC&U
2009; Gacel-Ávila, 2005; Green, 2012 & 2013; Jones & Killick, 2013; Leask, 2001, 2008; Lewis
& Niesenbaum 2005; Mestenhauser, 1998; NAFSA, 2010; Sample, 2013; Schoorinan, 1999;
Soria & Troisi 2013). Varying knowledge and skills of assessment procedures and practices, as
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 55

with any such deficiencies in knowledge and skills, results in inadequately trained staff and
administrators unable to implement assessment protocols with uniformity.
Identifying knowledge needs in administrators’ assessment of global education allows
one to determine the types of knowledge-based resources that are missing and need to be
enhanced. Knowledge is often considered the most strategically important resource and learning
the most strategically important capability for organizations. However, explicitly articulating the
link between knowledge and strategy has always been challenging (Zack, 1999).
A number of factors contribute to the organizational problems facing administrators
tasked with developing assessment models (Burnett & Huisman 2010; Childress, 2009). From a
factual knowledge perspective, administrators may not have access to the information necessary
to successfully perform the functions of their jobs. The majority of professional training and
development for assessing global learning focus on teaching administrators about measuring
learning outcomes in a single part of the institution (AAC&U, 2015a, AAC&U, 2015b; ACPA,
2008 and 2015; NAFSA 2015; NASPA, 2015a, NASPA 2015b). From a metacognitive
perspective, administrators might not accurately consider the importance of global education
assessment from a multi-dimensional perspective, causing administrators to make decisions that
lead to a failure to implement such a model (AAC&U, 2015a, AAC&U, 2015b; ACPA, 2008 &
2015; NAFSA 2015; NASPA, 2015a, NASPA 2015b). Additionally, administrators might be
unable to reflect on their own actions and the challenges they face in carrying out job tasks.
Elmore’s (2002) research in organization improvement implies professional
development, accompanied by incentives, may help address institutional barriers to change.
Elmore puts forth that to increase quality of student’s educational experience, institutions must
invest in the building of knowledge and skills of the educators. Professional development must
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 56

allow for engagement of stakeholder ideas, values, and energy into the process. Otherwise, the
professional development becomes an exercise in demanding compliance rather than capacity
building.
Motivation
Motivation theory refers to the goals, values, beliefs, and expectations individuals, as
well as organizations, hold that influence organizational performance (Clark, Howard, & Early,
2006; Clark & Estes, 2008). Motivation determines need achievement, directing how one will
learn (Ambrose, Bridges, DiPietro, Lovett, & Norman, 2010; Schunk, Pintrich, & Meece, 2010).
It is comprised of three major indices; active choice, persistence and mental effort, and should be
approached as a controllable component shaped by sociocultural, environmental and internal
factors (Clark & Estes, 2008; Dembo & Seli, 2008).
Utilizing principles of Behavior Theory (Tuckman, 2010), Information Processing
Theory (Schraw & McCrudden, 2013), and Social Cognitive Theory (Smith, 2002) the
motivation issues linked to the innovation can be examined. According to Social Cognitive
Theory, self-efficacy provides the foundation for human motivation and accomplishment. As
highlighted by Pajares (2009) people need to believe their actions can produce the desired
outcomes, or there is not incentive to persevere. Self-regulatory practices, by which individuals
go about the task of self-correcting their actions and cognition, are a critical determinant of self-
efficacy (Pajares, 2009).
Drucker (2002) states that individuals by values and beliefs, social recognition and
power, while Earley and Ang (2003) found that higher interest, leads to higher motivation. As
such, when stakeholders have interest in developing new knowledge they will have higher levels
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 57

of motivation. Motivation of individuals and organizations considering an innovation, as such, is
likely a central component to adaptation of the innovation.
Stakeholders, furthermore, can be motivated intrinsically as well as extrinsically.
Intrinsic motivation is vital when tacit knowledge amongst teams must be transferred (Osterloh
& Frey, 2000). When large amounts of knowledge, such as the development and implementation
of institutional level assessment, need to be transferred between individuals, intrinsic motivation
should not be disregarded. Organizational leaders, identifying the correlations between
individual goals and organizational needs, may increase intrinsic motivation as well as the
overall performance from an individual level to the organizational level.
There is a body of evidence substantiating the value of new models of assessment in this
area (Deardorff, 2006; Green, 2005, 2008, 2012 and 2013; Jones & Killick 2013; Leask, 2008;
Mestenhauser, 1998; NAFSA, 2010; Schoorinan, 1999; Soria & Troisi 2013; Stone, 2006).
However, it is not clear if utilizing behavior theory, information processing theory, and social
cognitive theory as a model for the examination of motivation will yield sufficient motivational
factors within the stakeholders at North College despite this body of evidence substantiating the
value of new models of assessment in global education. It is possible that the workload
regarding assessment is also a factor in motivation. This concern is supported by literature
(AAC&U, 2015a, AAC&U, 2015b; ACPA, 2008, 2015; NAFSA 2015; NASPA, 2015) that
continues to focus on singular area assessment practice while highlighting the importance of
global education across the academic and co-curricular areas.
Organization
Organizational dynamics (Bolman & Deal, 2007; Clark; 2004; Clark, 2005; Senge,
Smith, Kruschwitz, Laur & Schley, 2008) classifies several social patterns common to
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 58

organizations, including organizational culture, values, and goals. Needs should be aligned to an
organizational mission and strategic planning process. Needs must also be clear so individual
stakeholders, as well as teams, assume ownership and comprehend the work they undertake
relating to the need. Individuals react to, and internalize, organizational goals uniquely, which
could result in either insignificant or substantial effect on need achievement.
Organizational culture is a central factor in the work environment. Clark and Estes
(2008) describe culture as the “core values, goals, beliefs, emotions and processes learned as
people develop over time” (p. 108). Furthermore, they suggest the examination of culture
through multiple lenses of environment, groups and individuals.
Based on the principles of expectancy value theory (Eccles, 2010; Soupir-Fremstad,
2013) people are more engaged in an activity when they see that activity has high value. North
College promotes a supportive and collective culture in which feedback occurs freely. However,
formal reporting and goal setting from committees and meetings into a shared archive that is
accessible to the larger community is rare. Additionally, the feedback tends to align to
consensus rather than expressing of conflicting views for open debate. The result is lack of
clarity on where, and how, to transfer decisions reached amongst the groups and on common
value in the outcome. Conversely, the collective organizational culture can result in resistance to
the incorporation of new ideas into the culture as well as decision-making structures. The result
can be an organization that adopts a complacent attitude, social loafing, and with groups and
individuals who have power distinct from their leadership role(s).
An organization is not only an interaction of the individuals within it, but also an
assessment of the external environmental factors within which it operates (Rueda, 2011).
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 59

Performance needs can permeate an entire organization given their complex culture and social
design (Clark & Estes, 2008). Applying expectancy value theory to the innovation need a
solution strategy could be to use data from assessment to demonstrate to the stakeholders that
assessment has value in multiple areas, including achieving the global education SLOs,
accreditation at an institutional as well as divisional level, institutional rank from publically
reported documents, and meeting the institutional mission (Eccles, 2010; Soupir-Fremstad,
2013).
Leadership and culture can be challenging to isolate from each other. Group learning
process is complex and creates organizational culture. When change must occur, for example in
the innovation of a multilayered assessment of global education SLOs, it is a task of the
organizational leadership to recognize the impact it will have on the organizational culture. The
notion of innovation may create a threat to the stability of the organization resulting from fears
by stakeholders, as individuals and groups (Schein, 2004).
Qiang’s research (2003) offers alignment to the organizational culture concepts of Schein
(2004). Institutions must distinguish between the organizational and academic elements in
internationalization, such as in assessment. Activities must be central to the mission of the
institution and comprise integrated activities. Importantly for assessment, Qiang (2003) views
the organizational elements of internationalization in terms of governance processes, operational
processes and support services.
Conclusion
The literature outlines broad trends for the continually shifting field of global education.
The field is likewise accelerating in how it is affecting higher education, in particular in the
breadth of activities that embody internationalization. It is, however, not a new phenomenon.
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 60

Higher education has been, and continues to, alter and evolve practices to integrate global
components into academic and co-curricular programs, research, and operations; in all three
areas in the realms of direct services, management, and strategy and policy. There is great
variance, based upon institutional type and other factors, of how globalization affects and is
implemented at an institution.
SLOs define students’ institutionally expected knowledge, skills, and attitudes developed
as a result of a learning experience. Institutions must determine whether they offer sufficient
opportunities to produce the intended learning, select assessment measures (e.g., rubrics), and
then assess SLOs data match institutional expectations. Results should, if the SLOs are not met,
result in review and improvement of existing programs as well as possible creation of new
programs and opportunities as needed.
Literature highlights lack of a common definition of what criteria‐ based broad-based
assessment means, or what it implies for practice. The significance of formative assessment in
measuring SLOs is generally acknowledged, but it remains in its infancy, not always clearly
defined across higher education. In recent years, there has been an increasing emphasis on
assessment results, as well as what use is made of the results. As such, there is increasing
interest in alternative forms of assessments, particularly complex assessments. Current literature
refers to a need for models of learning assessment that measure global education that cross
within the classical domains of curricular and co-curricular programs, as well as with other types
of programming.
SLOs may be crafted for measuring cognitive outcomes (e.g., recall or recognition of
knowledge, development of intellectual skills and abilities), ranging from domain-specific
knowledge to the most general of reasoning and problem-solving skills. Non-cognitive SLOs are
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 61

usually crafted for the development of values or changes in beliefs (Ewell, 2005). The latter can
be more complicated to measure than are skills and knowledge, and global education SLOs
frequently measure both areas. There is, as such, growing importance within the literature on
building models of assessment that measure global education practices which help students
achieve measureable learning outcomes within, as well as outside of, the traditional classroom
setting.
Student learning assessment goes beyond the definition of SLOs. For global education,
assessment requires gathering evidence that demonstrates student achievement of specific
competencies as a result of educational experiences abroad as well as on the campus and similar
measures in the co-curricular realms regardless of location. While there is no one specific agreed
upon list of global education SLOs, there are some merging of themes amongst various
institutions such as AAC&C, NAFSA, the Forum on Education Abroad, and ACE. Such themes,
and the AAC&U rubrics, are expanding in breadth to assess a wider range of academic
departments, include both undergraduate as well as graduate education, to be applied in the U.S.
and overseas, and to incorporate co-curricular activities into the assessment. This framework,
however, requires comprehensive multi-dimensional institutional program analysis of the
measurement process to assure SLO goals are supported by evidence of successful achievement
of the stated learning outcomes.
A number of factors contribute to the organizational problems facing administrators
tasked with developing assessment models. From a factual knowledge perspective,
administrators may not have access to the information necessary to successfully perform the
functions of their jobs. The majority of professional training and development for assessing
global learning focus on teaching administrators about measuring learning outcomes in a single
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 62

part of the institution. From a metacognitive perspective, administrators might not accurately
consider the importance of global education assessment from a multi-dimensional perspective,
causing administrators to make decisions that lead to a failure to implement such a model.
Additionally, administrators might be unable to reflect on their own actions and the challenges
they are facing in carrying out job tasks. Motivation of individuals and organizations
considering an innovation, such as multi-dimensional assessment, is likely a central component
to adaptation of the innovation. Finally, organizational culture, values, and goals should be
aligned to an organizational mission and strategic planning process, is such an innovation to be
implemented.

MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 63

CHAPTER THREE: METHODOLOGY
Purpose of the Project and Questions
The purpose of this study was to conduct a needs analysis, a form of gap analysis, in the
areas of knowledge and skill, motivation, and organizational resources necessary to reach the
organizational performance innovation. The analysis began by generating a list of possible needs
and moved to examine these systemically to focus on actual or validated needs. While a
complete needs analysis would focus on all stakeholders, for practical purposes, the stakeholder
to be focused on in this analysis were college administrators sitting on the AAC, GAAC,
Assessment Committee, and Student Affairs Committee Co-Curricular Assessment Sub-
Committee.
As such, the questions that guided this study were the following:
1. What are the knowledge, motivation, and organizational needs necessary for the
North College administrators as stakeholders to achieve their goal of implementing a
multilayered assessment model for global education learning outcomes in 100% of
majors, as well as 50% of activities of the division of student affairs, utilizing the
AAC&U Global and Intercultural Rubrics?
2. What are the recommended knowledge, motivation, and organizational solutions to
those needs?
Methodological Framework
The Clark and Estes (2008) gap analysis model is primarily intended to analyze deficits
in performance within organizations. However, this methodology has been used for the
identification of performance gaps in education (xii). For example, Rueda (2011) succinctly
encapsulates Clark and Estes’ methodology in an educational context in his synopsis of the
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 64

explanation for modest success in efforts to address low student achievement despite many
attempts to the contrary. Rueda summarizes the causes as “the fragmentation of approaches, the
misalignment of approaches and goals, and the failure to match solutions to problems” (p. 11).
The Clark and Estes methodology purposefully incorporates these elements, and others, into
measuring a performance gap that address the root causes in a systemic manner. With respect to
innovation studies, or needs analysis, as the identified organizational goal does not yet exist, the
focus is on assumed performance needs to achieve the goal or close the “gap.”
For this needs analysis, Clark and Estes provided a methodology for the examination of
the gap between the current organizational performance level and the desired organizational
performance level, the innovation in assessment. This methodology was particularly beneficial
for examining the possible barriers of the organization through the analysis of three needs:
knowledge, motivation, and organizational capabilities. The process began by determining
measurable organizational performance goals. Once the organizational goal was established, it
was essential to ensure stakeholder alignment (Clark & Estes, 2008).
The second step in the process was to analyze the gap between the current organizational
performance level and the desired performance goal. With respect to innovation studies, or
needs analysis, the gap is expressed as a 100% need. As the identified organizational goal
existed, however there was no current plan to actualize the goal, the organization was operating
at 0% of the identified need. Once the desired performance, or the innovation achieved, the
organizational performance should be 100% of the gap.
The third step in the Clark and Estes (2008) methodology is to identify the causes for the
need in organizational performance. The process is distinct in its deliberate and research-based
method to analyze the core cause(s) for the performance needs, prior to identifying and
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 65

implementing possible solutions. Based upon their research, Clark and Estes identified three
principle causes for performance deficits that result from needs in knowledge, motivation as well
as organizational barriers (2008). Rueda, as previously cited, has identified these causes in his
research within educational settings (2011). In the framework of this needs analysis of North
College, these three potential “causes” were referred to as “needs.”
With respect to the knowledge component of the study, an organization must determine
whether its stakeholders have adequate information to achieve the organizational goal (Clark &
Estes, 2008). If stakeholders do not have the information or skills required to attain the
organizational goal, the innovation cannot be accomplished.
According to Clark and Estes (2008), analysis of motivational factors can be challenging
as it is based on an internal psychological process. Motivation, in this methodology, assesses
whether stakeholders elect to work toward a goal, the needs required to continue working toward
the goal, and necessary mental effort to achieve the goal (p. 44). Clark and Estes found in their
research that needs in organizational performance are often motivational.
The third primary cause for performance needs, per Clark and Estes (2008), is an
organizational barrier. Typically, these barriers indicate insufficient organizational processes or
resources or a negative culture. It is critical to examine the “cultural profile” (p. 111) of the
organization when hypothesizing assumed needs around organizational barriers. Proposed
solutions will only work if they are centered within the “profile context” of the specific
organization examined, as well as aligned to the needs in the areas of knowledge and motivation
within the specific context. The needs analysis process is shown in Figure 3.

MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 66

Figure 3. The Gap Analysis Process. Adapted from Clark and Estes (2008) and Rueda (2015)

Assess:
Knowledge
• Factual
• Conceptual
• Procedural
• Metacognitive

Motivation
• Self-efficacy
• Expectancy
• Outcomes
• Task value
• Attributions
• Goal content
• Goal orientation

Organization

• Models
• Settings
Solutions
• Scanning interviews & observations
• Personal experience
• Research
• Learning, motivation and organizational
theories
• Learning principles
• Motivation principles
• Organizational principles
• Research
Validated
Needs
Goals
Assumed
Needs
Implement Evaluate
Gaps
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 67

Assumed Performance Needs
Clark and Estes’ (2008) gap analysis framework is defined as a systemic, research-based
problem-solving methodology to increase organizational performance. Through needs analysis,
an organization measures short-term as well as long-term goals and defines needs between
current levels of performance and expressed goals, undertaking research to validate the needs.
Organizations frequently assume causes for the existence of needs without validation through
research and analysis (Rueda, 2011). This assumption could result in implementation of
unproductive or unsuitable solutions (Clark & Estes, 2008). The tendency to assume needs may
furthermore result in omission of alternative causes for performance needs or inaction as a
consequence of supposed complexity in solving the problem (Clark & Estes, 2008). A research-
based analysis of North College’s performance needs included four components: (1) informal
scanning interviews with participating stakeholders; (2) examination of learning, motivation, as
well as organization and culture theory; (3) analysis of internal North College documents; and
(4) review of related literature on global education SLOs and assessment. This study reviewed
needs from related literature in Chapter Two and included the needs in Table 4 below. The
following is a discussion of the knowledge, motivation, and organizational assumed needs for the
implementation of a multilayered assessment model for global education SLOs utilizing the
AAC&U Global and Intercultural Rubrics at North College.
Preliminary Scanning Data
Knowledge and skills. At North College, preliminary scanning data suggests
operational knowledge of assessment varied widely, and global education assessment even more.
In informal scanning at North College, similar variance was also suggested to knowledge of
AAC&U Global and Intercultural Rubrics. Literature supported this variance of knowledge
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 68

within the field of higher education regarding assessment (Burnett & Huisman 2010; Childress,
2009).
Preliminary scanning showed that, while many of the stakeholders attended conferences
and training on assessment, few, if any, attended a similar training focused on global education
assessment. Likewise, the scanning suggested that some stakeholders, fewer than for
assessment, attended a training or conference focused on AAC&U Global and Intercultural
Rubrics. Additionally, preliminary scanning data suggested knowledge and skill retention
subsequent to these conferences and trainings was questionable. A focus on conferences and
training is problematic given the growing importance of assessment in both the academic and co-
curricular realms (AAC&U 2009; Gacel-Ávila, 2005; Green, 2012, 2013; Jones & Killick, 2013;
Leask, 2001, 2008; Lewis & Niesenbaum 2005; Mestenhauser, 1998; NAFSA, 2010; Sample,
2013; Schoorinan, 1999; Soria & Troisi, 2013). As observed in meetings, internal memos, and
working groups focused on SLOs, some stakeholders who attended were unable to recall the
information. Furthermore, other stakeholders in the meetings continually asserted that they were
highly proficient in assessment procedures. However, in the last Higher Learning Commission
of the North Central Association of Colleges and Schools (HLC) report, the institution was
criticized for the lack of consistent demonstration of assessment across all departments of the
college. Preliminary scanning suggests varying knowledge and skills of assessment procedures
and practices at North College, as with any such deficiencies in knowledge and skills, could
result in inadequately trained staff and administrators unable to implement assessment protocols
with uniformity.
Motivation. Motivation at North College, from the preliminary scanning data, seemed to
vary depending on the stakeholder position (e.g., staff, administration; senior or junior within the
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 69

prior roles). Preliminary scanning data suggested that, generally, most of the stakeholders were
committed to the organization’s success and, therefore, were eager to discuss new methods to
improve the organization. This commitment appeared to be particularly true of senior
administrators, who seemed to thrive within the discussions of more robust assessment. These
stakeholders, therefore, appeared to be motivated by the social dynamics of the organization,
overlapping with organizational culture.
In contrast, preliminary scanning data suggested that stakeholders who were also faculty
and therefore part of the stakeholder group (e.g., chairs and department heads) seemed less
motivated to work toward the organization’s betterment if innovation is part of the framework, in
particular questioning why innovation around assessment was necessary. There is a body of
evidence substantiating the value of new models of assessment in the area of Global Education
and Intercultural Competence (Deardorff, 2006; Green, 2002, 2005, 2008, 2012 2013; Jones &
Killick, 2013; Leask, 2008; Mestenhauser, 1998; NAFSA, 2010; Schoorinan, 1999; Soria &
Troisi, 2013; Stone, 2006). The reports within committees and open forums on assessment from
stakeholder groups expressed that they were already overloaded, and, while they expressed the
need for more robust assessment simultaneously, they voiced a frustration at more tasks added to
their roles.
Through the preliminary scanning, it was seen that administrators and staff also voiced a
concern about the mapping of SLOs across the curriculum, let alone to the co-curricular, and the
work this would take in relation to their current role and knowledge of other domains. This
concern is supported by literature (AAC&U, 2015a, AAC&U, 2015b; ACPA, 2008 & 2015;
NAFSA 2015; NASPA, 2015a, NASPA 2015b) that continues to focus on singular area
assessment practice while highlighting the importance of global education across the academic
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 70

and co-curricular areas. Non-senior administrators appeared to demonstrate a balance between
the other two stakeholder groups: a voiced need for innovation in SLOs assessment, coupled
with concern around a possible increased workload and the challenges in completing all the daily
work alongside the innovation of a new assessment initiative.
Organization. Preliminary scanning suggests North College promoted a supportive and
collective culture. Stakeholders communicated with each other and offered feedback freely.
However, formal reporting from committees and meetings into a shared accessible archive
appeared to be sporadic. Additionally, preliminary scanning suggested the feedback tended to
align to consensus rather than expressing of conflicting views for open debate. While
stakeholders reported operational needs to each other, informally and formally, through existing
governance structures (e.g., faculty senate, cabinet) there was appeared to be no across
institutional body or archive to examine ideas that result(s) in a clear chain of command to a
decision-making body. As a group, and subgroups, the stakeholders appeared to encourage
collaboration and collective behavior, but the preliminary scanning data suggested there was lack
of clarity where, and how, to transfer decisions reached amongst the groups.
The collective organizational culture, conversely, could result in resistance to the
incorporation of new team members and ideas into the culture as well as decision-making
structures. It is possible that devotion to stakeholders with longevity at the institution may
prevent newer stakeholders from providing feedback within formal settings. Additionally,
preliminary scanning data suggested that the collective nature of decision-making could be
confusing. First, this inclusion of new members and ideas had been observed in preliminary
scanning data in relation to giving of feedback in formal settings that is not perceived as
collective. Second, lack of clarity was suggested through observation regarding the body or
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 71

individual(s) within that body that had the ultimate authority to make a decision. The result
could be an organization that adopts a complacent attitude, social loafing, and with groups and
individuals who have power distinct from their leadership role(s).
Learning and Motivation Theory
Knowledge and skills. Applying Krathwohl’s (2002) revision of Bloom’s taxonomy to
this needs analysis, a number of factors might contribute to the organizational innovation at
North College. Krathwohl (2002) outlines four knowledge categories: factual, conceptual,
procedural, and metacognitive. Factual knowledge requires acquaintance of basic facts
pertaining to a topic. From a factual knowledge perspective, preliminary scanning data
suggested the North College stakeholders might not have known how to develop an assessment
model for a single area. It would be difficult for the stakeholder groups to implement a
multilayered assessment model if they cannot implement one for a single area. In contrast,
conceptual knowledge comprises interrelationship knowledge of factual categories or principles
in a given field, enabling them to work together (Krathwohl, 2002). From a conceptual
knowledge perspective, preliminary scanning suggested North College stakeholders may not
have known the difference between assessment, testing and evaluation. Without this knowledge,
they cannot bring together the principles to implement an assessment model for a single area, let
alone a multilayered model. Knowledge of such concepts would enable stakeholders to derive
meaning from the factual knowledge, improving long-term memory of basic facts for later
retrieval when developing assessment models (Krathwohl, 2002; Schraw & McCudden, 2013).
From a procedural knowledge perspective, the application and criteria of the factual and
conceptual knowledge areas, North College stakeholders might be unfamiliar with the steps
involved in developing an assessment model. The stakeholders cannot develop a model if they
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 72

are not familiar with the components and how they could be deployed within, as well as across,
areas. In this needs analysis metacognitive knowledge has specific meaning defined by
Anderson and Krathwohl (2001) as “knowledge of [one’s own] cognition and about oneself in
relation to various subject matters” (p. 44). From a metacognitive perspective, preliminary
scanning suggested that North College stakeholders might not have accurately assessed the
importance of creating an assessment model that is multilayered. Likewise, they could lack
awareness of their own skills and deficits in assessment, resulting in decisions that led to issues
with accreditation or ensuring SLOs were met. Additionally, preliminary scanning data
suggested stakeholders might have been unable to reflect on how the lack of such a model could
jeopardize accreditation and possibly improve SLOs at North College.
Motivation. As the study questions examine an innovation at North College, active
choice, persistence, and mental effort as defined by Clark and Estes (2008) were not yet
motivational issues. As such, psychological constructs utilizing frameworks from Pintrich
(2003) and Clark (1999) needed to be assessed. These psychological constructs could predict the
stakeholder’s engagement with the innovation in three ways: active choice, or conscious choice
to pursue the innovation; persistence, or continued effort despite distractions; and mental effort,
or expenditure of thought necessary to achieve the innovation.
At North College, preliminary scanning data suggested the motivational variables
appeared to be five possible needs. First was value which is seeing, or not, the importance of
doing a particular task including extrinsic (utility) and intrinsic (interest), as well as attainment
(importance). Second, was self-efficacy which is having, or not, the confidence in one’s ability
for completion of a task. Third, was attributions which is viewing the success, or not, of a task
as based upon external, uncontrollable causes and not the result of one’s own efforts. Fourth,
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 73

was goal orientation which is the level to which one engages in learning to master skills or grow
as learners. Fifth, was interest that is the level to which one has interest, or only situational
interest, in the task to be completed (Clark & Estes, 2008; Pintrich, 2003). Stakeholders at North
College may not have been motivated to develop a multilayered assessment model because they
did not comprehend the value in engaging in this task. Self-efficacy could be a variable, as the
stakeholders did not have confidence in their ability to develop such a model of assessment. The
stakeholders could have been attributing prior assessment issues to faculty, unsupportive
leadership, or lack of resources. Goal orientation could have been a variable as it is not clear
stakeholders want to develop assessment models beyond the general education requirements or
link SLO academic and co-curricular assessment. Prior to the HLC review, stakeholder interest
in assessment may have been low, possibly reinforced by goal orientation.
Organization. This study utilized the sociocultural theory framework of Gallimore and
Goldenberg (2001) in the analysis, focusing on cultural models and cultural settings. Often
automated, cultural models are shared mental schema or normative understandings of how the
world works or ought to work (2001). In contrast, a method of capturing “context” is through
the examination of cultural settings, or observing displayed activity within the specific
environment. In particular, discerning within the environment where behavior is perhaps
undertaken without awareness by those taking it, or, their actions are automatic (2001). In a
needs analysis organizational factors such as cultural models and cultural settings must be
considered as possible causes, how they affect performance and goal achievement. In addition,
possible solutions to the cultural and social context examined must be situated within the
analysis.
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 74

Preliminary scanning data suggested the cultural setting at North College, as with many
universities, was evolving and unpredictable in particular due to the possible shifting of the
institution from a college to university structure as well as the financial crisis that affected North
College in 2012. From a cultural setting perspective, three factors were suggested by
preliminary scanning data that may have been at work to explain the current performance need at
North College. First, stakeholders were possibly overwhelmed by tasks unrelated to assessment,
and this kept them from investing time in developing more robust assessment models. Due to
the financial crisis, (a) conflicting goals around assessment as a priority may have resulted, and
(b) a lack of resources devoted to assessment may have be present. From a cultural model
perspective, lack of accountability and an attitude of helplessness might also have contributed to
the performance need at North College. Since cultural models help contextualize organizational
structure, practices and policies, lack of accountability, and a sense of helplessness are valid
organizational issues that might explain performance needs.
Summary
A summary of the sources of assumed needs categorized as Knowledge, Motivation, and
Organization, by source of assumption, is found in Table 4.

MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 75

Table 4
Summary of Assumed Needs for Knowledge, Motivation, and Organizational Issues
Assumed Needs
Sources Knowledge Motivation Organization
Preliminary
Scanning Data
(informal
knowledge
about the
organization)
• Need knowledge of
AAC&U rubrics.
• Need knowledge of
who is responsible for
assessment at North
College.
• Need knowledge of,
and current success at,
assessment at North
College.
• Need operational
knowledge of Global
Education assessment.
• Need motivation to
serve on Committee(s).
• Need understanding of
importance of Global
Education to Mission of
North College.
• Need to see value of
AAC&U rubrics in their
work.
• Need to have
confidence in their
ability to map student
learning outcomes
across curriculum, and
to the co-curricular.
• Need for more
clarity in and
across committees
in reporting,
openness, chain of
command and
transfer of
decisions.
• Need for newer
stakeholders to
provide feedback
within formal
settings
• Need long-term
commitment to
multilayered
assessment of
Global Education
at North College.
• Need of clarity and
resources due to
prior financial
crisis.
Learning and
Motivation
Theory
• Need knowledge of
Student Learning
Outcomes.
• Need ability to monitor
their skills related to
assessment and make
needed adjustments.
• Need knowledge of
how to develop, or steps
involved in, developing
an assessment model.
• Need knowledge of
links between
assessment in academic
and co-curricular areas.
• Need to see value to put
in effort to learn new
types of assessment
models.
• Need to see value to put
in effort to implement
new assessment models.
• Need clearly
defined goals
related to
multilayered
Global Education
assessment.
• Need resources
devoted to
assessment.
• Need a structure of
accountability.
• Need an
environment that
supports learning
and motivation in
assessment.

MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 76

Table 4, continued
Assumed Needs
Sources Knowledge Motivation Organization
Related
Literature
• Need clear definition of
assessment.
• Need knowledge of
what multilayered
assessment means.
• Need skills to develop
and integrate
assessment within and
across academic and co-
curricular areas.
• Need to see value of
Global Education
student learning
outcomes assessment to
their work.
• Need to see the value in
the importance of
assessment between
academic and co-
curricular areas.
• Need support to
develop new types
of assessment.

Participating Stakeholders
The participating stakeholder population for the needs analysis were all North College
administrators sitting on one or more of the following committees: (1) AAC, (2) GAAC, (3)
Assessment Committee and (4) Student Affairs Committee Co-Curricular Assessment Sub-
Committee. The total number of members per committee, the study population, was as follows:
(1) AAC 22; (2) GAAC 15; (3) Assessment Committee 14; and (4) Student Affairs Committee
Co-Curricular Assessment Sub-Committee 14. Amongst the four committees, comprising 65
total members, there was an overlap of 12 members across one or more committee.
North administrators sitting on the above committee(s) for the purpose of this study were
defined as any member permitted to be on the committee(s) per the appropriate governing
handbook(s) of the college, including voting and non-voting members. For the purposes of this
study, North administrators were defined as administrators and faculty holding a part or full-time
administrative position, serving on any of the above committees as permitted per college
governing handbook(s). Of the individuals who met the above criteria, a sample group of ten
stakeholders across all four committees, as described below, were asked to participate in the
project and the entire sample group queried.
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 77

The sample group of administrators from these committees who were selected to
participate were the formal intermediary between the faculty and administrative governing
bodies of the college while concurrently sitting on formal committee(s) charged with overseeing
college-wide assessment and program evaluation. This group provided formal feedback to the
governing bodies regarding the feasibility of organizational policy decisions and was also
responsible for the implementation of executive policy decisions by the governing bodies.
Additionally, this group interacted directly with other formal and informal bodies of the
institution whose work crosses into assessment (e.g., General Education Committee). The
sample group of ten all met the above rationale for selection in the sample, were administrators
responsible for assessment or program evaluation implementation at North College, in addition
to meeting the definition of a stakeholder for this needs analysis. The sample group is comprised
of members of the committees as follows: (1) AAC 7; (2) GAAC 3; (3) Assessment Committee
4; and (4) Student Affairs Committee Co-Curricular Assessment Sub-Committee 4. There is an
overlap of eight members across one or more committee. The representation of the sample group
to the committees is seen in Figure 4.
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 78

Figure 4. Respondent Representation On Committees Charged with Assessment Oversight
Data Collection
Permission from University of Southern California (USC) and North College Institutional
Review Boards (IRBs) was obtained. To validate assumed needs in knowledge, motivation and
organization, an online survey, in-person interviews and document analysis were undertaken.
Each is discussed next.
Surveys
Surveys were distributed to a subgroup of the stakeholders, the sample group, which
included ten administrators directly responsible for assessment implementation and program
evaluation at North College. The survey was disseminated via online software in October and
November 2015, once approval from the USC and North College IRBs was received. The online
4
7
3
4
0
1
2
3
4
5
6
7
8
Assessment
Committee
Academic Affairs
Committee
Graduate Academic
Affairs Committee
Student Affairs
Committee Co-
Curricular Assessment
Sub-Committee
Number of responses.
Respondent Representation on Committees Charged with Assessment
Oversight.
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 79

survey option was a more viable option due to the nature of the sample structure that limited the
researcher’s ability to gather all participants together. The survey was administered in English
and distributed to the sample group. The survey consisted of 20 items and included Likert-type
items, multiple choice, and open-ended questions. Survey items assessing knowledge,
motivation and organization were grounded on instruments pulling from the work of Krathwohl
(2002), Clark and Estes (2008), Pintrich (2003), Clark (1999), Mayer (2011), Gallimore and
Goldenberg (2001), Schein (2004) as well as Rueda (2015).
Responses were collected anonymously and tabulated through software. Two copies, of
both responses and tabulated results, maintained for backup purposes were stored on a password-
protected laptop as well as in a secured cloud-based location. No individual identifiable
demographic information was collected during the survey. Two years after the study has been
completed, all copies of data, responses, and tabulated results are scheduled to be destroyed. The
survey instrument can be found in Appendix A.
Interviews
North College stakeholders who comprised the survey sample group were also
individually interviewed in person. Interviews consisted of approximately 21 questions and
lasted approximately 45 minutes. Interviews were conducted in English and took place at North
College in a common use meeting room of the institution. Each interview began with a standard
interview protocol that included obtaining permission to record the interview for later
transcription and coding. Two copies of the interview, and coded transcriptions, are maintained
for backup purposes and stored in a password-protected laptop as well as a secured cloud-based
location. Additionally, two copies of identifiable information are stored separately from the
interview and coded transcripts in a password-protected laptop as well as a secured cloud-based
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 80

location. Two years after the study has been completed, all copies of the interview, transcripts
and identifiable information are scheduled to be destroyed.
For the first two interviews, member checks were conducted to confirm the interpretation
was accurate. The interview protocol is included in Appendix B.
Document Analysis
Multiple documents were analyzed for the purposes of this dissertation, including reports
between North College and HLC; internal North College correspondence regarding HLC; faculty
and staff handbooks; notes from stakeholder group meetings; reports and correspondence to the
North College community from stakeholder groups regarding assessment and SLOs; prior
surveys of stakeholder groups on assessment; and reports and internal North College
communication regarding design and implementation of institutional assessment relating to
SLOs. Documents were gathered and reviewed to triangulate results. Documents listed above
were from academic and student affairs divisions to more thoroughly understand the issue of
global education assessment at North College.
Validation of the Performance Needs
The remaining sections of Chapter Three outline how assumed needs were validated in
order to ascertain which required solutions and which turned out not to be needs and, as such, did
not require solutions. A mixed-methods approach was taken for the validation of performance
needs in this study. Data validation methods included analysis of a survey and individual
interviews of the sample group. Documents were analyzed, along with observations, to further
triangulate survey and interview results. The mixed methods resulted in an in-depth validation
of the performance needs, substantiated by the analysis output. A summary of how each need
was be validated is shown in Table 5.
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 81

Table 5
Summary of Assumed Knowledge Needs and Validation
Methods of Validation

Survey Interviews Document
Analysis
Assumed knowledge needs X X X
Assumed motivation needs X X
Assumed organizational needs X X X

Summary
These needs were assessed, validated or not validated, to inform solution generation
regarding the needs of stakeholders in the innovation of development of a systemic multilayered
assessment model for global education at North College. A qualitative case study with
descriptive statistics was used. Responses were collected and statistical analyses conducted
based on survey results, and select interview findings incorporated for further qualitative
analysis.
Data Analysis and Validity
A qualitative case study with descriptive statistics approach to this study allowed for
quantitative and qualitative assessment of survey and interview results. Descriptive statistical
analysis (e.g., calculating frequencies) was undertaken after all survey data were submitted.
Interviews were transcribed, and coded into themes that aligned with the knowledge, motivation
and organization categories. As the population of this dissertation included fewer than 20
individuals, report frequencies (i.e., what percentage of stakeholders strongly agreed or agreed
versus the percentage who strongly disagreed or disagreed) were the mode of analysis.
Documents were assessed to further triangulate survey and interview results, resulting in an in-
depth evaluation that aligned to the statistical output.
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 82

Document analysis occurred from November 2014 to January 2016. The survey was
completed by the sample group between October and November 2015 and closed. An Excel data
entry template was developed for capturing and calculating frequency of survey responses. A
preliminary review of survey data was conducted in November 2015 after the survey was closed,
following the sequence of questions and KMO coding. The interview protocol was modified
based upon the preliminary review of survey data to support triangulation, and validate or
exclude trends in the data. Interviews were conducted in December 2015 to probe further into
assumed needs in KMO categories.
Between December 21, 2015, and January 30, 2016, data analysis was conducted. This
review of data consisted of several activities. The researcher recorded raw notes from data
analysis as well as interviews in a Word document following the sequence of questions and
KMO coding. The Excel data entry template was expanded for capturing and calculating
frequency of survey as well as interview responses. All data was entered and crosschecked for
consistency. Missing responses were highlighted. After the initial coding the researcher
performed triangulation of results and comparison of findings with assumed causes to determine
degree of validation, and organized the data into visual graphics for reporting results.
Trustworthiness of Data
Data for this needs analysis were rendered trustworthy through four means. First,
triangulation of data between surveys, interviews, and document analysis was undertaken
(Merriam, 2009). Second, survey items were based on existing valid and reliable instruments
pulling from the work of Krathwohl (2002), Clark and Estes (2008), Pintrich (2003), Clark
(1999), Mayer (2011) Gallimore and Goldenberg (2001), Schein (2004) as well as Rueda (2015).
Third, assurance of anonymity (survey) and confidentiality (interviews) was undertaken
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 83

throughout the collection and data analysis (Merriam, 2009). Finally, member checks were
conducted to confirm the interpretation of both the survey and interview protocols were accurate.
Role of Investigator
I was the dean of global education at North College. I oversaw all staff within the IGE as
well as indirectly staff of international student and scholar services, and I was responsible for all
aspects of IGE programs operating in the US and overseas. As dean, I was responsible for
engaging diverse constituents in strengthening global education at North in the 21st century;
leading and managing international initiatives, partnerships, programs, off-campus sites, study
abroad/away, and customized programs; providing visionary, strategic, and collaborative
leadership for IGE; on and off campus, overseeing international and study away operations,
human resources, and budgets; exploring new partnerships and opportunities for North College
in the global arena; promoting intercultural understanding and off-campus learning; working
with faculty to deepen global education in the graduate and undergraduate curriculum; with the
appropriate governance bodies ensuring assessment of global education is occurring in the
curriculum as well as through co-curricular opportunities; in collaboration with other
departments. I reported directly to the Provost and Chief Academic Officer.
As principal investigator in this project, my role was to conduct a needs analysis of the
innovation and propose solutions to help North College implement a systemic multilayered
assessment model for global education learning outcomes in 100% of majors, as well as 50% of
activities of the division of student affairs, utilizing the AAC&U Global and Intercultural
Rubrics. This innovation would help the organization establish and maintain a robust assessment
model both for accreditation, as well as ensuring SLOs in global education.
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 84

For this needs analysis, members serving on stakeholder committees were made aware of
my role as principal investigator. Steps were taken at various points to ensure the anonymity of
survey respondents. As part of the interview and survey protocols, the sample group was
informed that the main purpose of the study was to gather information for the purposes of
improving the organization and that any information obtained during this process would be
anonymous and utilized for that purpose only. Sample group members, as well as the larger
stakeholder committees, were informed that no individual identifiers were collected to ensure the
highest level of participant confidentiality. Sample group members were also reassured that
participation was voluntary and that there were to be no consequences for electing not to
participate. North College community members within IGE, as well as formal faculty and
administrative governance structures, were made aware that I was conducting this study as a
doctoral candidate and that findings would be presented to the Provost of the organization.
Stakeholders were informed that it is the Provost who has the authority to decide what to do with
the findings of this project.
Limitations and Delimitations
Some limitations resulted from the design of this project. First, the sample size of this
study was only ten individuals. Second, the needs analysis was limited by the honesty and
possible bias resulting from the sample group providing answers they considered to be socially
desirable, not a true presentation of their experience, or selective in memory. This bias was
possible in both the interview and survey contexts, including assessing knowledge with Likert
scale in the survey though it was triangulated to questions in the interview. Third, the needs
analysis was potentially limited by the fact that it assumed all participants understood and
interpreted the interview questions and survey items in the manner intended. Sample group
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 85

participants could also have someone else complete the survey since it was administered
anonymously online. Fourth, document analysis was based upon those documents that were
currently available; as such, the sample analyzed could be biased either in missing historical
documents or those that were not stored in formal institutional repositories.
The focus of this study was to conduct a needs analysis of the innovation and propose
solutions to help North College implement a systemic multilayered assessment model for global
education learning outcomes in 100% of majors, as well as 50% of activities of the division of
student affairs, utilizing the AAC&U Global and Intercultural Rubrics. The primary delimitation
of the project is that it was context specific to North College and addressed this specific
organization’s mission and innovation goal; neither of which can be generalized. However,
other institutions may benefit from the application of this project’s use of Clark and Estes (2008)
needs analysis process to bring about innovation application in a similar manner.
The project was also delimited to examining a sample of the key stakeholder group’s
experience that may or may not be representative of that of other stakeholder groups. While
other stakeholder groups’ experiences and contributions are important to the organization, an in-
depth investigation of those was outside the scope of the current project.

MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 86

CHAPTER FOUR: RESULTS AND FINDINGS
The purpose of this study was to identify the root needs that confront North College in
innovating a systemic multilayered assessment model for global education. Utilizing the needs
analysis framework, research identified unmet underlying knowledge, motivation, and
organizational needs that may be inhibiting North College from innovating a new model of
assessment. In particular, assumed needs in KMO categories were produced from a
comprehensive review of literature, learning and motivation theories, organizational theory, and
personal knowledge. The needs analysis framework allowed for assumed needs to be
systemically narrowed and validated based on findings from a survey, interviews, and document
analysis. Additionally, while conducting the research, new needs were identified. This chapter
presents the results, with key findings synthesized, in the categories of knowledge needs,
motivational needs and organizational needs. Each section highlights the assumed needs that
were validated, those that were not validated, and any new needs identified during research.
Data were analyzed and triangulated to understand the knowledge, motivation, and
organizational needs. Results were then compared with the assumed knowledge, motivation, and
organizational needs described in Chapter Three to determine if they were valid and, as such,
required solutions, or were invalid and should be eliminated as a root need of the innovation.
The chapter concludes with a summary of the validated needs that informed the development of
solutions in Chapter Five.
The researcher determined that an assumed need was validated if either (A) the findings
were consistent across more than one source (e.g., document analysis, survey, interview) at a
threshold of 40%; or (B) if more than 60% of the sample group confirmed the assumed needs in
either the interview or the survey. The threshold of 40% across instruments was determined as
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 87

the validity point as this lower threshold compensates for differences between the survey and
interview protocols, but would equal 80% threshold in either instrument if such differences did
not exist. The threshold of 60% was determined as the validity point for a single instrument was
utilized as it was higher than a majority of respondents. In most cases, utilizing the threshold of
60%, regardless of the source of data, would not alter the findings.
Results and Findings for Knowledge Needs

A survey, interviews, and document analyses were used to validate or not validate the
assumed knowledge needs. Utilizing Anderson and Krathwohl’s (2001) framework based on
Bloom’s taxonomy, knowledge is categorized into four dimensions, including factual,
conceptual, procedural, and metacognitive. Knowledge needs were identified and classified per
each knowledge dimension. The assumed knowledge needs for this study are shown in Table 6.
Table 6
Knowledge Needs Validated, Not Validated, and New Needs
Assumed Need Validated Not Validated New Need
Need knowledge of who is
responsible for assessment at North
College.
√
Need clear definition of assessment. √
Need knowledge of student learning
outcomes.
√
Need knowledge of, and current
success at, assessment at North
College.
√
Need ability to monitor their skills
related to assessment and make
needed adjustments.
√
Need knowledge of how to develop,
or steps involved in, developing an
assessment model.
√
Need knowledge of links between
Assessment in curricular and co-
curricular areas.
√

MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 88

Table 6, continued
Assumed Need Validated Not Validated New Need
Need knowledge of AAC&U
rubrics.
√
Need operational knowledge of
Global Education assessment.

√

Need knowledge of what
multilayered assessment means.
√
Need skills to develop and integrate
assessment within and across
academic and co-curricular areas.
√

Student need of knowledge of
learning outcomes.
√

To generate a synthesized approach to the findings, results were grouped by assumed
needs into larger groupings as categorized in Table 6, as utilizing Anderson and Krathwohl’s
(2001) framework dimensions more than one type of knowledge could be needed within each
broad need category (e.g., knowledge of current assessment at North College includes factual
knowledge of who is responsible for assessment, a definition of assessment which while factual,
also correlates to SLOs in factual, procedural and conceptual framework dimensions when using
the AAC&U framework upon which the model is based). This approach allows for findings to
be triangulated across the results of the survey, interviews, and document analysis. The
categories are Knowledge of Current Assessment at North College, Knowledge of, and Current
Success at, Assessment at North College, Knowledge of AAC&U Framework and Rubrics,
Knowledge of global education Assessment, Knowledge of and to Develop Multilayered
Assessment, and Student Knowledge of Learning Outcomes.
Knowledge of Current Assessment at North College
The findings validated the factual knowledge need of respondents to know who has
responsibility for assessment at North College, a key need for any effort to innovate a new
systemic, multilayered assessment model. This knowledge need was, furthermore, validated in
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 89

both procedural and conceptual knowledge areas, as well as connected to organizational needs
that is discussed later in this chapter.
Within faculty governance documents of North College, responsibility for assessment is
formally in the charge of three different committees: Assessment Committee, Academic Affairs
(AAC) and GAAC. While the Assessment Committee has assessment as its primary focus, AAC
and GAAC are responsible for overall program evaluation at the institution. Additionally, the
Assessment Committee is formally charged with the coordination of assessment of co-curricular
SLOs. Outside of this faculty governance structure exists the Student Affairs Sub-Committee on
Co-Curricular Assessment, charged with assessment of co-curricular learning at North College
by the Vice President for Student Affairs. Representation on the Student Affairs Sub-Committee
on Co-Curricular Assessment, and the Assessment Committee, is achieved by the chairs of each
committee named to the other.
While 100% of respondents identified the Assessment Committee has oversight of
assessment at North College, less than a majority identified either AAC or GAAC as having a
role in assessment (Figure 5). This figure, like others in this chapter, is a radar chart displaying
multivariate data in the form of a two-dimensional chart of three or more quantitative variables
represented on axes starting from the same point. Each axis represents one of the variables, with
the findings of the variable for the data point relative to the maximum magnitude of the variable
across all data points. Radar charts are utilized in this chapter as they can assist in highlighting
clusters of similarities, and outliers, as well as displaying multivariate findings with a number of
variables relative to each other.
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 90

Figure 5. Respondent View of Entity at North College Responsible for Assessment
This figure shows the contrast between the number of respondents who identified the
Assessment Committee and Student Affairs Committee, 10 and 8, respectively, as having
oversight of assessment with the number who identified the AAC and the GAAC, 5 and 5
respectively, as having oversight of assessment. This finding on the lack of clarity around who
has oversight on assessment was supported by interviews. The need finding of who has
responsibility for assessment at North College was triangulated in interview data where more
than 60% expressed lack of clarity on AAC and GAAC as having any role in assessment. A
respondent who sits on AAC stated “I wasn’t aware we had any role in assessment.” Despite
having it formally in their charges, other respondents who sit on AAC and GAAC stated, “We do
nothing with assessment” and “I don’t see AAC doing anything with assessment.” As is
discussed later, there was some variance within the other 40% in seeing how the work of these
committees does, indeed, link to assessment, despite lack of knowledge regarding oversight
responsibility.
0
2
4
6
8
10
Assessment Committee
Academic Affairs
Committee
Graduate Academic
Affairs Committee
Student Affairs Committee
Co-Curricular Assessment
Sub-Committee
Respondent View of Entity at North College Responsible for
Assessment.
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 91

This factual knowledge need was triangulated to other data in both the survey as well as
interview. Respondents were asked in the survey to identify who has responsibility across the
realms of faculty, staff and administration (Figure 6) and in the interview through an open-ended
question “Describe who has responsibility for assessment at North and your role it in.” While
100% of respondents correctly indicated faculty had some to great extent in responsibility for
assessment in the survey, which they do per governance documents, all also saw administration
and staff as likewise having some to great extent of the responsibility. On the formal faculty
committees charged with curricular assessment within the governance structure, administration
and staff have no voting role.

Figure 6. Respondent View of Who Has Responsibility for Assessment at North College
This need for knowledge around who has responsibility for assessment at North College
was corroborated in the interview where 100% of respondents identified faculty as the primary
8
2
6
4
4
6
0
0
2
4
6
8
10
12
14
16
18
To great extent To some extent To no extent
Number of responses.
Respondent View of Who has Responsibility for Assessment
at North College.
Faculty Administration Staff
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 92

body with responsibility of assessment and, a majority also identified the newly appointed Chair
of the Assessment Committee as having primary responsibility. However, as in the survey,
100% of respondents also voiced that assessment was the responsibility of everyone at the
college. In particular, mentioning both staff and administration in addition to faculty when co-
curricular was included. Furthermore, it was articulated by 50% of the respondents that while
assessment oversight rests with faculty, that the allocation of resources required collaboration, as
well as oversight, from the administration. The findings show it is not clear to the respondents
who is in charge of assessment, in particular when co-curricular assessment in included. As
discussed in the organizational needs findings section, this is not entirely surprising due to mixed
messages, unclear, as well as changing messages around who has responsibility for assessment in
addition to the resource allocation issue.
In addition to respondents demonstrating a need for additional knowledge around who is
responsible for assessment, they also lack a clear definition of assessment. North College
utilizes the AAC&U rubrics for assessment, as such, AAC&U’s definition of assessment was
triangulated against respondents’ answers. AAC&U (2009) defines assessment as
A philosophy of measuring student learning that privileges multiple expert judgments and
shared understanding of the quality of student work through the curriculum, co-
curriculum, and beyond over reliance on standardized tests administered to samples of
students disconnected from an intentional course of study. (para. 3).
In the interview respondents were asked to define assessment, evaluation and testing.
There was no consistent definition of assessment given, though there were commonalities in
answers, such as measuring student learning, and measuring of student learning through the
curriculum and co-curricular. There was variance in definitions whether assessment or
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 93

evaluation was the actual measure of student learning, or if either or both were rather
institutional measures of how as an entity such as North College was achieving (or not) its
intended SLOs. For example, one stated for assessment, “it’s student progress through certain
programs and whether these programs are achieving the outcomes they hope” focusing on
program level evaluation and not the overall student experience distinct from a course of study as
AAC&U defines assessment. Another two interviewees focused on the institutional knowledge
and motivation in stating “assessment is gathering evidence to something we claim” and it is to
determine “whether or not we are meeting the learning objectives of the college.” There was a
recurrent need, 60% of respondents, to possess the knowledge to differentiate assessment and
evaluation. This need was most clearly articulated by one stakeholder who stated “I am
struggling to define them in ways that are distinct from each other.”
While respondents were able to articulate that assessment measures student learning,
more than 60% were not able to define key aspects of the AAC&U definition, including (A)
utilizing multiple expert judgments, (B) a shared understanding of the quality of the student
work, and (C) reliance on samples of student work disconnected to an intentional course of
study. In fact, more than 60% of respondents stated that there was not a shared common
understanding of measuring student work at North College, which one respondent encapsulated
in stating “if we could come to an agreement on three or four things…I just don’t know that we
as a community have had conversations about this in terms of ‘so what does this mean.’”
Additionally, only 20% of respondents indicated assessment having any disconnect from an
intentional course of study, as defined by AAC&U. One respondent voiced it as “departmentally
driven.” As such, the factual need of a definition of assessment for North College was validated.
This root need must be addressed, without a clear common factual definition of assessment, the
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 94

institution cannot move into the procedural and conceptual realms of innovating a systemic
multilayered assessment model for global education.
Building from a need for knowledge on a definition of assessment, there was likewise
seen a need for knowledge around institutional SLOs. North College’s institutional
undergraduate curricular SLOs, approved by the faculty in December 2014, are directly linked to
the AAC&U rubrics to create an assessment model. Appendix E presents the fully approved
document. Below, Figure 7, is a visual showing the mission based outcomes mapped to specific
AAC&U rubrics.

Figure 7. North College Undergraduate Student Learning Outcomes
A total of eight learning outcomes, connected to one of four institutional mission
educational values (informed citizens, thoughtful stewards, critical thinkers, and responsible
Critical Thinkers
Information Literacy /
Critical Thinking /
Writing / Oral
Communication
Thoughtful Stewards
V ocation / Religious
Literacy / Health and
Wellness /
Sustainability
Responsible Leaders
Quantitative Literacy /
Problem Solving /
Integrative Learning
Informed Citizens
Civic Engagement /
Ethical Reasoning /
Global Awareness /
Intercultural
Competence
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 95

leaders) are then connected to one or more AAC&U rubrics for assessment. Additionally, there
are course, major and minor, graduation skills, and program/disciplinary assessment measures
currently in place at the college. While there is the overarching model of AAC&U, when layered
down into programs, majors and minors, as well classes there is need for knowledge around the
SLOs, as well as procedural and conceptual knowledge of the entire model linked to the
institutional outcomes.
Administrators’ needs for knowledge of SLOs were validated across interview findings
where less than 40% were able to answer questions around them correctly. One respondent
described them as “vague and strewn throughout” and another described the lack of clarity
around a coherent layering as “a source of frustration.” Another respondent, when asked to
specify their answer to North, implied conceptual knowledge in stating, “So I cannot give you
the answer, I was thinking of somewhere else.” Another respondent also demonstrated
conceptual knowledge in stating they are “those hoped for achievements in cognitive, affective
and behavioral realms.” Thirty percent of respondents mentioned cognitive and affective in
relation to student learning, and 30% referenced Bloom’s taxonomy.
Interview data findings regarding student learning outcomes at North College included
30% articulated knowledge of all types of SLO assessment occurring at the college; and while
100% knew that institutional SLOs existed broadly, and that they were directly linked to the
mission, only 30% stated the correct number of them. These data validate root needs in factual,
procedural and conceptual knowledge that must be addressed, without knowledge of how these
SLOs scaffold and the conceptual framework upon which they are built, the college cannot
innovate a systemic multilayered assessment model for global education.

MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 96

Knowledge of, and Current Success at, Assessment at North College
To innovate a new model of multilayered global assessment at North College, individuals
and the institution will need to build off of their current ability to and skills and success at
assessment. When asked in the interview to articulate their knowledge and success at assessment
broadly, both individually as well as an institution, 30% of respondents used the word “failure”
and another 30% implied it. At the same time, there was a theme of increasing institutional
effort put toward assessment, as one respondent stated “up until this point I don’t know, not
really successful, in that we are just getting intentional about it.” Another echoed this in stating,
“it still feels like we are rolling the boulder up the hill” and a third articulated that “building the
institutional capacity to be more successful” was occurring.
There was, however, metacognitive knowledge data that despite these efforts needs were
validated. One respondent stated that “the Committee (Assessment Committee) knows there is a
problem and the administration knows.” Another more clearly articulated that, given where the
college is at with assessment success would be hard to measure, when they stated, “We are at the
very early stages of doing this, even those first steps haven’t yet been proven.” Furthermore, all
respondents indicated that additional training was needed by faculty, staff and administrators at
the institution, including for themselves. Specific assessment training needs articulated by
respondents included definitions; procedural and conceptual methods and models; how to align
conceptual models across curricular and co-curricular for faculty, administrators and staff
including the use of AAC&U rubrics; common assessment cycles; and how to share information
within and across institutional divisions.
The knowledge need of how committees engage in the success of the institutional model
of building a multilayered assessment framework, even if there was not clarity around who had
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 97

direct responsibility, was seen in both procedural as well as conceptual needs realms in interview
data. Less than 50% of respondents articulated that the work of AAC or GAAC linked to
assessment, though both of these committees are charged with approving new courses and the
SLOs attached to them. Thirty percent articulated that when approving courses SLOs were
looked at. Other respondents indicated efforts by these committees to attempt to increase
procedural and conceptual knowledge, one stated they felt the committee was “actually asking
them [faculty] about the SLOs and what they connect to.” These three respondents in particular
articulated the work of AAC as shifting to question more how course changes and approvals, and
their SLOs, fit into as one stated a “broad look across the curriculum” and “how the pieces fit
together.”
One respondent reinforced these findings of the direct link between course approval,
sequencing, and SLOs in stating, “I don’t think we have a collective understanding of what that
means as an institution.” Reflecting the finding that effort goes toward assessment from a
procedural and conceptual knowledge need, another expressed that as an institution they felt
North was “gaining traction, last year I couldn’t have told you.” As such, data corroborated the
knowledge need around SLOs in factual, procedural as well as conceptual realms.
This finding of procedural and conceptual knowledge need was correlated to survey
questions asking respondents to assess their knowledge to conduct assessment within their area,
develop new models of assessment in their area, and to develop new models of assessment across
areas (Figure 8). The survey data corroborated the interview findings. Ninety percent strongly
agreed or agreed they have the knowledge to conduct assessment in their area currently while
10% disagreed. When asked if they have the knowledge to develop new models of assessment in
their area, 70% stated they do while 30% said they do not. When asked if they have the
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 98

knowledge to develop new models of assessment across areas 40% said they do, while 60%
stated they do not.
These data illustrate a decreasing knowledge as the procedural and conceptual model
increases in complexity, moving from simply conducting assessment in its current form, to
development of a new assessment models in one area, to a minority stating they had the
knowledge to develop a multilayered model across areas.

Figure 8. Respondent View of Ability to Conduct and Develop General Assessment Within and
Across Areas
4
5
1
3
4
3
3 1 6
0
0
1
2
3
4
5
6
7
8
9
10
Strongly agree Agree Disagree Strongly disagree
Number of responses.
Respondent View of Ability to Conduct and Develop General Assessment within
and Across Areas.
I have the knowledge to develop new comprehensive models of assessment across areas.
I have the knowledge to develop new comprehensive models of assessment in my area.
I have the knowledge to conduct assessment within my area in a comprehensive manner.
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 99

Adding to the need of assessment models broadly, when asked to articulate how familiar
they were with the college’s recent change in accreditation frameworks, which requires changes
in assessment cycles, data gathering, as well as more institutional responsibility to gather
assessment data, the respondent data reflected a similar lack of knowledge to the new assessment
framework and requirements. While 80% were aware that a change had occurred, fewer than
20% could articulate that the change was significant or express any specifics of what the change
entailed in terms of assessment at the institution. One respondent stated, “I know enough to say
that we have changed”, while another showed metacognitive knowledge in stating, “I don’t know
a lot about the differences.” Only 20% of respondents indicated being very aware of the change
or what it entailed for the institution.
As such, the needs in the knowledge area of, and current success at, assessment at North
are validated. First, there is metacognitive knowledge that the institution is not currently
successful at assessment, as well as factual, procedural and conceptual knowledge needs around
assessment to be addressed across the institution. Second, there is lack of factual, procedural and
conceptual knowledge of the college’s new HLC assessment cycle and requirements. Third,
there is lack of procedural and conceptual knowledge to develop new comprehensive models of
assessment across areas of the institution, as AAC&U stipulates should occur within its
definition of assessment. Utilizing Anderson and Krathwohl’s (2001) framework dimensions all
four types of knowledge are needed within each, as well as across, need category to be able to
systemically innovate around assessment.
Knowledge of AAC&U Framework Rubrics
Knowledge of AAC&U Framework Rubrics are key to innovating a new model of
assessment as North College’s institutional undergraduate curricular SLOs are directly linked to
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 100

the AAC&U rubrics (Appendix D). Data between the survey and the interview conflicted in the
knowledge need regarding the AAC&U rubrics. In the survey, respondents were asked to
articulate where they saw global education intersecting with other AAC&U rubrics (Figure 9).
One hundred percent of respondents answered the survey question asking them to indicate how
they saw global education interconnecting with other AAC&U rubrics. The question allowed
respondents to select any of the other AAC&U rubrics they saw an interconnection with.
However, in the interview 80% of respondents expressed “limited” to “no knowledge” of the
global education rubric. This discrepancy validates a lack of factual knowledge around the
rubrics. It also validates both procedural and conceptual knowledge needs around the rubrics as
they scaffold across building an assessment framework.
If appropriate knowledge was had, when mapping an interaction between global
education rubric to other AAC&U rubrics a standard distribution should be seen. As shown in
Figure 9, a standard distribution is not seen. Figure 9 shows how many respondents selected the
other rubrics as intersecting with global education. When accounting for the total number of
responses, as respondents could select any across the three AAC&U domains that encompass the
rubrics, the respondents in 38% of instances mapped global education to the AAC&U Personal
and Social Responsibility area; in 9% of instances to Integrative and Applied Learning; and in
53% of instances to Intellectual and Practical Skills. The latter, Intellectual and Practical Skills
makes up 50% of the entire AAC&U rubric mode. The mapping validates both (A) an
overemphasis on the Personal and Social Responsibility area at 38% when they are only 25% of
the rubrics reflecting a lack of procedural and conceptual knowledge of how the rubrics intersect
broadly, and (B) a lack of procedural and conceptual knowledge of how utilizing the Intellectual
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 101

and Practical Skills across the curriculum can be a means of imparting skills to build knowledge
in students around global education.

Figure 9. Respondent View of AAC&U Rubrics Interconnected with Global Education Rubric at
North College

0
2
4
6
8
Inquiry and analysis
Critical thinking
Creative thinking
Written communication
Oral communication
Reading
Quantitative literacy
Information literacy Teamwork
Problem solving
Civic engagement local
and global
Intercultural knowledge
and competence
Ethical reasoning
Foundations and skills for
lifelong learning
Integrative learning
Respondent View of AAC&U Rubrics Interconnected with Global
Education Rubric at North College
Interconnects with Global Education at Augsburg College
Personal and Social Responsibility
Integrative and Applied Learning
Intellectual and Practical
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 102

Knowledge of Global Education Assessment
Innovating a systemic multilayered assessment model for global education requires all
the knowledge areas previously discussed as well as specific needs related to multiple types of
knowledge particular to global education. When asked in the interview to articulate their
knowledge and success at assessment of global education, both individually as well as an
institution, 80% either used the word “failure” or implied it. One respondent described the
institution’s attempt to secure data in this realm as “we are flailing.” Another stated that the
institution had “moved from zero to one on a ten-point scale” in the past year. One hundred
percent of respondents indicated that additional training was needed by faculty, staff and
administrators at the institution. Specific to global education, as was found for assessment
broadly, this training needed to include definitions, procedural and conceptual methods, and
aligning conceptual models across curricular and co-curricular for faculty, administrators and
staff including the use of AAC&U rubrics, common assessment cycles, and how to share and
access information. One respondent stated that “if the college is using it and framing various
things around it, I want to be using it.” However, this individual also acknowledged confusion
around what the term global education even meant. Another respondent described this need as
“an attempt to try to glean what we think this means at the college level” as the respondent had
not seen the college clearly define it, or put forth a conceptual or procedural framework for
assessing global education across the institution. This sentiment looking to the college for clarity
and training, as well as information, was expressed by 80% of respondents. It correlates to the
prior findings around who has responsibility for assessment, and is discussed later in relation to
organizational needs.
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 103

As with knowledge and success of assessment broadly, this interview question was
correlated to survey questions asking respondents to assess their knowledge specific to global
education. Specifically, the question ranked knowledge to conduct global education assessment
within their area, develop new models of global education assessment in their area, and to
develop new models of global education assessment across areas (Figure 10).

Figure 10. Respondent View of Ability to Conduct and Develop Global Education Assessment
Within and Across Areas
4
6
2
8
0
2
8
0
0
2
4
6
8
10
12
14
16
18
20
22
Strongly agree Agree Disagree Strongly disagree
Number of respomses.
Respondent View of Ability to Conduct and Develop Global Education
Assessment Within and Across Areas.
I have the knowledge to develop new comprehensive models of assessment of Global Education
across areas.
I have the knowledge to develop new comprehensive models of assessment of Global Education in
my area.
I have the knowledge to conduct assessment of Global Education within my area in a
comprehensive manner.
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 104

When the question was specific to global education, the data shifted significantly from
the prior data asking respondents to state their ability to conduct and develop assessment broadly
(Figures 8 and 10). Not a single respondent stated they strongly agreed they had the knowledge
to conduct or develop comprehensive assessment of global education. Those who felt they had
the knowledge to conduct assessment within their existing area dropped from 80% for
assessment broadly to 40% for global education assessment. When asked if they had the
knowledge to develop new models of assessment either within, or across areas, the percentage
also shifted downwards as well as paralleled each other: 80% disagreed they had this knowledge,
and only 20% agreed.
As such, the knowledge need area of global education assessment at North is validated.
There is metacognitive knowledge that the institution is not currently successful at assessment in
this realm, as well metacognitive need for additional training in factual, procedural and
conceptual knowledge to be addressed across the institution specific to global education
assessment.
Knowledge of and to Develop Multilayered Assessment
In the interviews when asked “Describe what the term multilayered assessment means to
you” 100% of respondents were able to articulate, to varying degrees, the components of
multilayered assessment: covering the entire student educational experience, linking curricular
and co-curricular in particular high-impact practices, securing of data (artifacts) at multiple
points. These findings validate the conceptual knowledge of what the components of such a
model include. A majority, 80%, also raised two issues that were not visually represented in the
initial model of systemic multilayered assessment in Chapter One. First, that such an assessment
model required multiple artifacts, and, second, that they be gathered over time longitudinally. As
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 105

one respondent articulated of such a model’s ability to collect data and compile a picture of
SLOs:
Well, I am collecting my tiles, and then putting them into my mosaic. And, then, I am
stepping back to see what the picture is. Which is we have all of these tiles things all
over the place, can we get them somewhere together to step back and say we are
anywhere near on delivering on this promise of global education.
While the respondents articulated the conceptual components of such a model in the
interview, 90% of them stated they did not have the skills to implement such a model. This data
was triangulated against survey data (Figure 10) where 80% of the sample group indicated they
did not have the knowledge to either develop new models of assessment within their area, or
across areas. Only 40% of respondents indicated having the current knowledge to conduct
assessment of global education within their area, and 60% indicated they disagreed they
currently possessed such knowledge.
There were two primary knowledge needs identified in linking curricular assessment with
co-curricular in a systemic way. First, this need was validated through a lack of definition of
what global education actually is. While, as in other knowledge needs, respondents were able to
articulate components of global education linked to the AAC&U definition, there was great
variance. One respondent articulated global education as specific to a study abroad experience
only. Forty percent articulated global education is now, or should be, integrated in the
curriculum. However, 70% of respondents clearly articulated that no common definition exists
between those charged with assessing global education between the two areas. Second, the need
was validated by 80% acknowledging that SLOs were not mapped between the curricular and co-
curricular with a similar knowledge framework directly linked to the AAC&U model. One
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 106

respondent defined SLOs at North as “the things the faculty have decided are important for
students to secure with their education.” Another stated they were “attaching learning outcomes,
whether they were the faculty approved ones or the co-curricular ones” to their programs.
Additionally, the need to align curricular and co-curricular assessment on the same cycle was
stated by 50% of respondents as creating procedural knowledge confusion of who was measuring
what, when, and how to correlate any data across the curricular and co-curricular areas.
A core factual need for North College to implement a multilayered assessment model is
knowing what SLOs are linked to which courses so that data can be secured and elevated across
specific fields of study. Seventy percent of respondents raised this issue in the interview as an
obstacle. In the last general education revision at North College SLOs were embedded into
courses, a precise tracking of SLOs since that time has not occurred. As one respondent put the
need when linked to the college’s new assessment needs on the Open Pathways requirement, as
well as collecting data systemically and in a multilayered manner:
We built a house, our curriculum, and we’re now retrofitting it like you would retrofit a
Victorian for electrical, which is assessment. We did not build a house where we had
easy places to collect assessment data. So, we have a beginning and an end, but there is
no middle where we collect.
Triangulating this data to the sample group’s ability to monitor their assessment skills as
well as the current success of assessment at North College, metacognitive knowledge is validated
as high. There is awareness that additional training and knowledge is needed both individually
as well as within the institution to implement systemic multilayered assessment of global
education. This was also corroborated in the survey (Figure 11) where, when asked if there was
knowledge within the faculty to conduct global education assessment, 10% strongly disagreed,
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 107

60% disagreed, and 30% agreed. When the same question was mapped across staff knowledge,
the respondents were evenly split with 50% agreeing there was sufficient knowledge and 50%
disagreeing. Without clarity around definitions, processes and knowledge of where SLOs exist it
will be impossible to implement a multilayered assessment model for global education. That
metacognitive knowledge was high, as will be seen later in motivation, will possibly assist North
College in developing solutions to meet these knowledge needs so that it can innovate a
multilayered assessment model.

Figure 11. Respondent View of Sufficient Faculty and Staff Knowledge to Conduct Global
Education Assessment

5 5
0
3
6
1
0
1
2
3
4
5
6
7
8
9
10
11
Strongly agree Agree Disagree Strongly disagree
Number of responses.
Respondent View of Sufficient Faculty and Staff Knowledge to Conduct Global
Education Assessment.
North College, today, has faculty with the necessary knowledge to implement a multilayered assessment model
that includes Global Education.
North College, today, has staff with the necessary knowledge to implement a multilayered assessment model
that includes Global Education.
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 108

Student Knowledge of Learning Outcomes
A new need was identified during the research: ensuring students are aware of the
learning outcomes expected of them across their educational experience. Eighty percent of
respondents raised this need, stating that it would improve student learning by bringing clarity to
student awareness of global education as knowledge was accumulated over time through, but not
necessarily directly linked to, one class. The same percentage stated that, as it was one of the
institutional outcome goals for a student’s experience through both their curricular and co-
curricular experience, the link between the two realms, and ensuring students are aware of
acquiring skills and knowledge from both, was important. As one respondent articulated about
ensuring students have this knowledge, “it is not just about making sure people are learning what
they are learning, but it actually improves the learning and teaching.”
Synthesis of Results and Findings for Knowledge Needs
All 11 of the identified knowledge needs were validated (Table 7). These needs ranged
across Anderson and Krathwohl’s (2001) framework based on Bloom’s taxonomy, including
factual, conceptual, procedural, and metacognitive. Unmet needs were found in having common
definitions, procedural and conceptual methods, and aligning conceptual models of student
learning within the realms of, as well as across, curricular and co-curricular for faculty,
administrators and staff including the use of AAC&U rubrics. Additionally, the need for
knowledge of common assessment cycles across the institution, and how to share and access
assessment information and data was validated in relation to measuring success of assessment
and ability to monitor it. Metacognitive knowledge was validated as respondents had it. In this
realm, respondents expressed knowledge of both individual as well as institutional needs, and
limited success at, assessment broadly and more specifically of global education. The validation
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 109

of these knowledge needs suggests that myriad knowledge needs in the realms of factual,
conceptual, procedural, and metacognitive must be addressed within North College including
faculty, staff and administrators for the institution innovating a systemic multilayered assessment
model for global education.
Table 7
Summary of Validated Assumed Knowledge Needs & New Needs
Assumed Need Validated Not Validated New Need
Need knowledge of who is
responsible for assessment at North
College
√
Need clear definition of assessment √
Need knowledge of student learning
outcomes
√
Need knowledge of, and current
success at, assessment at North
College
√
Need ability to monitor their skills
related to assessment and make
needed adjustments
√
Need knowledge of how to develop,
or steps involved in, developing an
assessment model
√
Need knowledge of links between
assessment in curricular and co-
curricular areas
√
Need knowledge of AAC&U rubrics √
Need operational knowledge of
Global Education assessment
√
Need knowledge of what
multilayered assessment means
√
Need skills to develop and integrate
assessment within and across
academic and co-curricular areas.
√

Student need of knowledge of
learning outcomes
√

MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 110

Results and Findings for Motivation Needs
According to Clark and Estes (2008), motivation drives individuals to actively choose to
engage in an activity, or not, as well as persistence in that task. Motivation also serves to signal
the amount of effort to utilize in the task. Motivation may be a need if gaps in achievement are
related to choice, persistence, or effort (Clark & Estes, 2008). Underlying motivation indices of
choice, persistence, and effort are factors, which are beliefs around self-efficacy, attributions,
interests, values, and goal orientation (Pintrich, 2003). Within these five social-constructors
categories are more specific levels of beliefs, all of which can affect the motivation indicators of
choice, persistence, and effort. The assumed motivation needs in this study are shown,
transcending the five social-constructors where applicable, in Table 8.
Findings suggest there is the motivation to innovate a systemic multilayered assessment model
for global education at North College.
Table 8
Summary of Motivation Needs Validated, Not Validated
Assumed Need Validated Not Validated
Need motivation to serve on committee(s). √
Need understanding of importance of Global
Education assessment to Mission of North
College.
X
Need to see value of AAC&U Rubrics in their
work.
X
Need confidence in their ability to map student
learning outcomes across curriculum, and to the
co-curricular.
√
Need desire to put in effort to learn new types of
assessment models.
X

MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 111

Table 8, continued
Assumed Need Validated Not Validated
Need to see value to put in effort to design new
assessment models.
X
Need to see value of Global Education student
learning outcomes assessment to their work.
X
Need to see value in the importance of assessment
between academic and co-curricular areas.
(Interests, values, and goal orientation).
X

To generate a synthesized approach to the findings, results were grouped by assumed
needs into larger categories, as utilizing the framework of Pintrich (2003) more than one type of
motivational need could be within each broad need category (e.g., seeing value in the importance
of assessment between academic and co-curricular areas could be interest, value and goal
orientation). This approach allows for findings to be triangulated across the results of the survey,
interviews, and document analysis. The categories are Motivation to Serve on Committee(s) and
Global Education Assessment, Understanding Motivation of Importance of Assessment and
Global Education to Mission of North College, See Value of AAC&U Rubrics in Their Work,
Confidence in Ability to Map Student Learning Outcomes Across Curriculum and Co-curricular,
Desire to Learn New Types of Assessment Models, See Value of Global Education Student
Learning Outcomes Assessment to their Work, and See Value in Importance of Assessment
between Curricular and Co-curricular.
Need for Increased Motivation to Serve on Committee(s) and Global Education Assessment
Motivational need to serve on committees charged with assessment was validated. When
questioned in the interview about their motivation to serve on the committee(s), 70% of
respondents said they were “appointed” to the committee(s) and did not mention any interests
related to or desire to serve on the committee(s); the appointment was the reason articulated for
their sitting on the committee, whether in a voting or advisory capacity. The other 30%
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 112

specifically mentioned motivation linked to assessment of SLOs as the particular reason they
joined the committee(s). In particular, these 30% indicated a direct link to professional interests,
value in assessing their work as well as student learning, and goal orientation in improving their
work and programs, as well as those of the college. Given these findings, having motivation to
serve on a committee as an assumed need was validated. Motivation drives individuals to
actively choose to engage in an activity, or not, as well as persist in that task. Motivation also
serves to signal the amount of effort to utilize in the task, all of which are needed to develop a
multilayered assessment model for global education.
See Importance of Assessment and Global Education to Mission of North College
At the same time, respondent data showed clear understanding of the importance of
global education to the mission of North College, understanding of importance of global
education assessment to the mission of North College, and value of global education SLOs
assessment to the institution (Figure 12). Motivational needs in these areas were not validated.
One hundred percent of respondents stated assessment was of extreme importance to ensuring
SLOs. Ninety percent saw assessment of extreme importance to the mission of North College,
with the other 10% seeing it as important. Additionally, 90% saw the direct link between
institutional accreditation and assessment, with the other 10% seeing it as important. These all
correlate to the motivational areas of value and goal orientation. Divergent from data in the
knowledge section as well as other areas to be discussed in motivation needs, the respondents
were evenly split (50%) on the importance of assessment to the institution in the area of co-
curricular programs.

MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 113

Figure 12. Respondent View of Importance of Assessment at North College Broadly
In the survey, when asked a slightly different question, “Why Assessment is Important to
North College as an Institution” and not given a Likert scale option (Figure 13), data triangulated
to the prior motivational needs areas. One hundred saw it of importance to the college mission,
90% to institutional accreditation, 90% saw it of importance to academic and co-curricular
program review, and 80% saw it as having direct importance on their knowledge that their work
has an effect on student learning.
10
9 9
6
2
6
5
1 1
2 6
3
5 2 2
0
2
4
6
8
10
Student learning
outcomes
North College
Mission
Accreditation by
Higher Learning
Commission
Faculty Review Staff Review Academic Program
Review
Co-curricular
Program review
Number of responses.
Respondent View of Importance of Assessment at North College Broadly.
Extremely Important Important Unimportant
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 114

Figure 13. Respondent View of Why Assessment Is Important to North College as an Institution
Interview data triangulated to survey data in showing high motivation linked to the same
areas. One hundred percent of respondents indicated placing value, as both an institution as well
as in their own work, on the assessment of SLOs. One hundred percent of respondents saw it as
directly linked to the institutional mission. As one said, “I mean, we are driven by the mission.
We are a very, very mission driven institution, I truly believe that.” Likewise, and higher than in
the survey, 100% of respondents articulated in the interview that this work was of importance to
ensuring of SLOs. One respondent stated, linking the topic to goal orientation, that assessment
was the means by which “we go back and say how we ensure that every student hits that, right?
So, is it happening in every major, in every part of the core?” Furthermore, 100% saw the
importance of it to the accreditation of the institution.
In the survey, 100% of respondents either agreed or strongly agreed that North College is
committed to assessment of global education (Figure 14). However, as will be discussed later,
0
2
4
6
8
10
Knowing my work has an effect on
student learning outcomes
North College Mission
Accreditation by Higher Learning
Commission
Faculty Review Staff Review
Academic Program Review
Co-curricular Program review
Respondent View of Why Assessment is Important to North College
as an Institution.
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 115

there is divergence from this belief when asked specifically about resources and other types of
intuitional support.

Figure 14. Respondent Belief that North College Is Committed to Assessment of Global
Education
This data was triangulated in the interviews, where 100% also stated that the assessment
of global education was important to the institution. Specifics of why articulated by one
responded included ensuring the institution knows “what are the learning outcomes, how are they
being assessed, how are they being evaluated, how are they being tested.” Another articulated
the need to have a “concrete definition and understanding of the places where this is happening,
or, we expect it to happen.” A third respondent linked assessment of global education to
employability and that assessment needed to occur for “internal and external reasons” to ensure
students had the skills and knowledge to be a “citizen in global society these days.”

2
8
0 0
0
2
4
6
8
Strongly agree Agree Disagree Strongly disagree
Number of respones.
Respondent Belief that North College is Committed to Assessment
of Global Education.
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 116

See Value of AAC&U Rubrics in Assessment
As addressed in the knowledge section, despite a knowledge need around the AAC&U
rubrics, 100% of the respondents indicated that the rubrics would be of value in their work on
assessment. As such, additional motivational need to see the value of AAC&U rubrics was not
validated. One respondent indicated that they would “help clarify the thinking about what are
the components of global education.” Another, who did have knowledge of the rubrics, stated
how they were of particular value specifically as they “were designed without a particular
discipline, or type of study, or type of learning environment in mind.” There was a theme from
respondents that the rubrics would assist with “better assessment” as they were seen as a “tool”
upon which a “common framework” could be constructed.
Need Confidence in Ability to Map Student Learning Outcomes Across Curriculum and
Co-curricular
While there was variance within the interview answer asking respondents about their
monitoring and making adjustments to assessment if needed, linked to the knowledge need, 80%
indicated a lack of confidence to assessment. The lack of confidence was in all cases correlated
to needing more training, a shared understanding of what assessment and global education was,
and, to alignment of goals between assessment of the curricular and co-curricular. As such, the
motivational needs around self-efficacy and goal orientation related to confidence were
validated.
Have Desire to Learn New Types of Assessment Models
When questioned around motivation to learn new types of assessment models, 70%
expressed high interest, and 30% moderate, with no respondents expressing low interest (Figure
15).
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 117

Figure 15. Respondent Desire to Learn New Kinds of Assessment Models
This motivational need was also seen in the data when respondents were questioned
specifically in regards to learning new kind of assessment that included global education.
Seventy percent expressed high interest, and 30% moderate, with no respondents expressing low
interest (Figure 16).

Figure 16. Respondent Desire to Learn New Kinds of Assessment Models for North College
That Include Global Education

0
3
7
0
1
2
3
4
5
6
7
Low Moderate High
Number of responses.
Respondent Desire to Learn New Kinds of Assessment Models.
Low Moderate High
0
3
7
0
1
2
3
4
5
6
7
Low Moderate High
Number of responses.
Respondent Desire to Learn New Kinds of Assessment Models for North
College that Include Global Education.
Low Moderate High
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 118

Interview data, again, mirrored the survey data with 100% of respondents expressing
interest in learning more about assessment generally, and in particular both new models of
assessment and global education in particular. As such, this motivational need was not validated.
See Value of Global Education Student Learning Outcomes Assessment
When asked about importance of conducting assessment, specifically to their work
(Figure 17), 80% of respondents strongly agreed it was important, and 20% agreed. Sixty
percent stated they strongly agreed in seeing value in developing new ways of comprehensive
assessment within their area, and 40% agreed. When asked a similar question regarding seeing
value in developing new assessment models across areas, the percentage of respondents who
strongly agreed increased to 70% with the remaining 30% agreeing. As such, this need is not
validated. When asked about confidence in their ability to develop more comprehensive models
of assessment within their area, however, the first sentiments of disagreement appeared, with
20% not feeling confident. 50% agreed they had this ability, and, the percentage who strongly
agreed lowered to 30%.
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 119

Figure 17. Respondent Views on Importance, Value and Confidence to Develop Assessment
Models
When questioned specifically why assessment is important to their specific work at North
College, not to the college as a whole, the respondents’ answers differed from the similar
question on why assessment is important to the institution (Figure 18). One hundred percent saw
it of value to knowing their work had an effect on SLOs, 80% saw the importance to institutional
accreditation, and 70% saw it of importance to the institutional mission as well as academic and
co-curricular program review. Interestingly, the largest differences in how respondents viewed
importance of assessment between their work and for the institution was in both faculty and staff
review. In both cases these areas ranked more highly of importance to the institution, and less
important to individual work.
8
6
7
3
2 4 3
5
2
0
2
4
6
8
10
Conducting assessment within
my area is important to me.
I see value in developing new
ways of comprehensive
assessment in my area.
I see value in developing more
comprehensive assessment
across areas of North College.
I feel confident in my ability to
develop more comprehensive
assessment in my area.
Number of responses.
Respondent Views on Importance, Value and Confidence to Develop Assessment
Models.
Strongly Agree Agree Disagree Strongly disagree
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 120

Figure 18. Respondent View of Why Assessment Is Important to Their Specific Work at North
College, Not The College as A Whole
As one respondent stated, “North has laid claim to being a place about experiential
learning, it’s about learning grounded in your experience, learning grounded in the environment
in which you live, then I ought to be able to assess that.”
See Value in Importance of Assessment Between Curricular and Co-curricular
In the interview, respondents were specifically asked if assessment that linked the
curricular and co-curricular areas was of importance to North College. One hundred percent
answered positively that such work was of value. As such, this need was not validated. Of note,
80% used the word “critical” in relation to this work and the college. As one respondent stated
“I think some of the most profound experiences students have are in the co-curricular, the most
high-impact experiences. And, let’s learn all we can about where those are happening and what
0
2
4
6
8
10
Knowing my work has an effect on
student learning outcomes
North College Mission
Accreditation by Higher Learning
Commission
Faculty Review Staff Review
Academic program review
Co-curricular Program review
Respondent View of Why Assessment is Important to Their Specific Work
at North College, not the College as a Whole.
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 121

students gain from them.” Another stated that assessment of this area, linked, was a priority as
“there is an absolute fundamental value in the co-curricular” in the educational experience.
Synthesis of Results and Findings for Motivation Needs
Of the eight identified motivational needs, two were validated (Table 9). The
stakeholder group already has many of the assumed motivation needs around developing a new
model of assessment, and as such, these are not needs to be addressed further. Findings suggest
there is the motivation to innovate a systemic multilayered assessment model for global
education at North College. These needs ranged across self-efficacy, interests, values, and goal
orientation (Pintrich, 2003). Within these five underlying motivational factors are specific levels
of beliefs, all of which can affect the motivation indicators of choice, persistence, and effort. As
only two needs were validated, there is seen a high level of motivation at North College to
conduct assessment, and in particular to develop new types of assessment that include global
education. As was highlighted in the knowledge needs and this section, and will be shown in the
organizational needs, the needs impeding North from innovating a multilayered assessment
model appear to be knowledge and organizational, not motivation. At the moment, the data
shows active choice and persistence in conducting assessment work, despite needs in other areas.

MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 122

Table 9
Summary of Motivation Needs Validated, Not Validated
Assumed Need Validated Not Validated
Need motivation to serve on committee(s). √
Need understanding of importance of Global
Education assessment to Mission of North
College.
X
Need to see value of AAC&U Rubrics in their
work.
X
Need confidence in their ability to map student
learning outcomes across curriculum, and to the
co-curricular.
√
Need desire to put in effort to learn new types of
assessment models.
X
Need to see value to put in effort to design new
assessment models.
X
Need to see value of Global Education student
learning outcomes assessment to their work.
X
Need to see value in the importance of assessment
between academic and co-curricular areas.
X

Results and Findings for Organization Needs
This study explored organizational needs that are imbedded in the culture and context of
the organization. The framework of Gallimore and Goldenberg (2001) provided the structure for
the research, in particular cultural settings and models, that facilitated the examination of
organizational needs in one of these two primary categories. Cultural settings are concrete and
include the employees, their tasks, how and why tasks are completed, and the social context in
which their work is performed. Cultural models refer to cultural practices and shared mental
schema within an organization (2001). Within these categories, needs are classified as
organizational resistance, non-participation, resources, autonomy, and strategic planning. Often
automated, cultural models are shared mental schema or normative understandings of how the
world works or ought to work (2001). In contrast, a method of capturing context is through the
examination of cultural settings, or observing displayed activity within the specific environment.
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 123

In particular, discerning within the environment where behavior is perhaps undertaken without
awareness by those taking it, or, their actions are automatic (2001). In a needs analysis
organizational factors such as those associated with cultural models and cultural settings must be
considered in terms of how they affect performance and goal achievement. The assumed
organization needs, most of which transcend cultural models and cultural settings, are shown in
Table 10.
Table 10
Summary of Organization Needs Validated & Not Validated
Assumed Need Validated Not Validated
Need clarity in and across committees in reporting, openness,
chain of command and transfer of decisions.
√
Need a structure of accountability. √
Need clearly defined goals related to multilayered Global
Education assessment.
√
Need newer stakeholders to provide feedback within formal
settings.
X
Need long-term commitment to multilayered assessment of
Global Education at North College.
X
Need resources devoted to assessment. √
Need environment that supports learning and motivation in
assessment.
√
Need support to develop new types of assessment. √
Need of clarity and resources due to prior financial crisis. X

Each of these assumed needs is discussed in turn, drawing on data generated from the
survey, interviews and document analysis. Results were grouped by assumed needs into larger
categories. This approach allows for findings to be triangulated across the results of the survey,
interviews, and document analysis. The categories are Organizational Clarity and Defined
Goals, Structure of Accountability, Newer Stakeholders to Provide Feedback Within Formal
Settings, Commitment and Resources Devoted to Assessment of Global Education, Support of
Learning and Motivation in Assessment, and Develop New Types of Assessment and Clarity and
Resources Due to Prior Financial Crisis.
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 124

Organizational Clarity and Defined Goals
North College has a core curriculum, a visual of which is seen in Figure 19. The
institution states this curriculum is “…designed to prepare students to be effective, informed, and
ethical citizens in the 21st century. Grounded in a strong liberal arts foundation, our curriculum
promotes the development of intellectual and professional skills” (North, 2014). However, the
core curriculum is intentionally designed to go beyond traditional classroom learning, the
institutional description continues, “…But it goes deeper, engaging students in an inquiry about
faith and calling in life, and supporting them in meaningful experiences outside of the classroom
(2014).” Of note, there is no clarity within the institutional statement if outside of the classroom
experiences include co-curricular as well as curricular learning within the core curriculum.

Figure 19. North Core Curriculum

North College Core Curriculum
Capstone
Search for
Meaning I
Search for
Meaning II
Electives
Major Coursework
North
Experience

Liberal Arts Foundation
Engage the City
North Seminar
Core Skills
Entry Skills
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 125

As was discussed in the knowledge section, this curriculum included a revision of the
general education requirements, during which time SLOs were embedded into the general
education core. As the core curriculum had been articulated as complex within the institution,
one organizational need that was examined was whether the respondents’ thought the academic
model made assessment difficult.

Figure 20. Respondent View That the Academic Model at North College Makes Assessment
Difficult
Sixty percent of the respondents indicated they agreed or strongly agreed that the
academic model made assessment more difficult (Figure 20). At the same time, 90% felt that
assessment could be improved, agreed or strongly agreed, while maintaining the academic model
(Figure 21). As such, low resistance and strategic planning were validated both within cultural
models as well as settings. Reinforcing this, a respondent in the interview stated “there is no
data, that I can find, that tells me if in the 20 plus years’ history” of their program that SLOs
were achieved. Another respondent stated the need of “identifying our priorities for assessment
1
5
3
1
0
1
2
3
4
5
Strongly agree Agree Disagree Strongly disagree
Number of responses.
Respondent View that the Academic Model at North College Makes
Assessment Difficult.
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 126

is one good place to start, and ensuring we could state those priorities publicly” linked to the
existing academic structure.

Figure 21. Respondent View Assessment at North College Could Improve While Maintaining
the Academic Model
Within the interviews respondents confirmed these data. One hundred percent of
respondents agreed that assessment at North College could be improved, broadly. When
discussing co-curricular assessment, both organizational clarity and defined goals were raised.
As one respondent voiced, there is a clear necessity “tying together the co-curricular and
curricular learning.” Echoing the respondent who could not find assessment data for their
curricular program, another respondent in the interview described an eight-year period when
baseline assessment was not undertaken due to a lack of clarity and approval of learning
outcomes.
Reinforcing this cultural model and setting need, this work is now just beginning at the
institution, and yet, still fundamental issues such as organizational clarity, accountability and
defined goals around alignment between co-curricular and curricular SLOs assessment continue
1
8
1
0
0
2
4
6
8
Strongly agree Agree Disagree Strongly disagree
Number of responses.
Respondent View Assessment at North College Could Improve While
Maintaining the Academic Model.
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 127

to exist. Sixty percent of respondents in interviews raised the fact that there is no alignment in
SLOs between co-curricular and curricular, with one of the two, curricular, clearly linked to
AAC&U and the other, co-curricular, moving in that direction. Likewise, that the cycle of what
data was collected on what outcome in a particular year between the two areas was not currently
aligned, creating confusion. There was, therefore, reinforced lack of organizational clarity
around what of the outside of the classroom learning is part of the educational experience, and to
whom responsibility for assessment of learning outcomes in it falls. This showed needs for
clarity in and across committees in reporting, openness, chain of command and transfer of
decisions. All of these reinforced the strategic needs across multiple areas of the organization, as
well as hinting at stakeholder participation which will be discussed later.
In relation to the need for clarity in and across committees in reporting, openness, chain
of command and transfer of decisions across the institution, 90% of respondents in the interviews
stated these areas needed to be improved. One cited the need to address it at the institutional
level, bringing together different groups to undertake “organizational planning” around
assessment. Another stated that there was a clear necessity to “develop protocols” around
communication, and that at the core it was “about processes and expectations.” Corroborating
the organizational needs in the strategic planning and participation realms, another respondent
voiced the requisite for “plans for assessment that do truly include all the units and departments,
a structure around how, and when it happens, and ensuring that folks are on a cycle.” Another
addressed the need for the institution in this realm, stating “having a model and implementing
one is crucial for our future success.” This reinforced the high metacognitive knowledge areas
previously discussed, and yet, illustrate how the cultural model allows these needs to continue
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 128

unaddressed. As such the needs of clarity in and across committees in reporting, openness, chain
of command and transfer of decisions are validated.
Structure of Accountability
Of importance to organizational need for clarity and goals, and directly related to
accountability of assessment, North College’s Core Curriculum distinctly specifies that the
college will support students in significant experiences external of the traditional classroom
setting. While the phrasing fails to identify if this is specifically curricular or co-curricular in
nature, the Core Curriculum language implies that experiential learning is occurring, in particular
in “Engaging Minneapolis”, “North Experience” and “Keystone.” As was shown in the
knowledge section (Figures 5 and 6) there is lack of knowledge as to which committee(s)
oversee assessment as well as the role of faculty, staff and administrators in this; this need is seen
in a structure of organizational accountability and autonomy as well. As one respondent stated,
implying a cultural model need, the institution needed to shift to “formal rather than informal,
because frankly the whole thing has felt very informal so far.”
In the interviews 80% of respondents reported confusion around who had ultimate
accountability for assessment at the institution. As such this need for a structure of
accountability that includes participation and strategic planning is validated across both cultural
model and setting realms. One respondent stated, “at that level of responsibility, someone kind
of needs to take the lead on making this happen.” This sense of accountability is correlated to a
lack of clear goals around assessment broadly, as well as specific to global education. One
respondent stated,
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 129

So having some consistent level of accountability, to those, those kinds of pieces of work,
would be a start so that this is not just a conversation with the Assessment Committee and
the faculty over here, and nobody else is an accountable part of the work.
Another respondent raised the topic of accountability within faculty who do have formal
oversight of the curricular learning outcomes, hinting at non-participation in highlighting the
focus at faculty meetings is on governance and faculty handbook issues, and that “up until a
couple of years ago, I could count on one hand when we actually talked about something that
was directly related to teaching and learning.”
When moving into organizational clarity around assessment goals, as was discussed in
the knowledge needs section, there is a lack of clear goals related to assessment. This need was
validated, again directly linked to strategic planning as well as resources, which will be discussed
later. As with other topics, respondents were asked to answer a Likert series of questions that
moved from assessment in one area, to across areas, to including global education. The answers
mirror prior needs data in the organization providing less clarity as assessment gets more
complex (Figure 22). Sixty percent of respondents disagreed that North College provides clear
goals around assessment within their area. On this issue, one respondent in the interview stated
that “people would send us what they had, but the artifacts folks were sending were not
measurable. Like, we couldn’t determine whether students were meeting learning outcomes.”
This statement shows a willingness to participate but lack of knowledge as well as planning on
assessment. When moving to assessment across areas, while overall the percentage remained at
60%, within that 10% increased to strongly disagree. As one respondent stated in the interview,
“having one set of SLOs, not two. And having one system into which data is kept, and
maintained” would be a starting point. Likewise, when asked about assessment of global
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 130

education, the percentage remained consistent at 60% who disagreed there are clear goals,
however, the percentage who strongly agreed decreased by 10%. Another stated the need for
“goals that are manageable and meaningful.” As such, the need for clearly defined goals related
to multilayered global education assessment is validated.

Figure 22. Respondent Sense of Clear Goals Related to Assessment
This need was corroborated in the interviews. While 100% of respondents identified the
Assessment Committee has oversight of assessment at North College, less than a majority
identified either AAC or GAAC as having oversight of assessment in any way, and 60%
expressed lack of clarity on AAC and GAAC as having any role in assessment. When bringing
the co-curricular into overall organizational structure, goal definition and strategic planning
2 2
1
2
3
2 2 3
3
1
6
5
6 5 6 1
0
2
4
6
8
10
I feel there are clear goals at
North College to conduct
assessment within my area.
I feel there are clear goals at
North College to conduct
assessment across areas.
North College, today, has
clear goals to implement a
multilayered assessment
model that includes Global
Education.
I feel there are clear
assessment goals on the
Committee(s) I sit on.
I feel there are clear
assessment goals on the
Committee(s) I sit on which
include Global Education.
Number of responses.
Respondent Sense of Clear Goals Related to Assessment.
Strongly agree Agree Disagree Strongly disagree
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 131

needs were validated. While respondents articulated that the institution was assessing the co-
curricular, other than communication between the Student Affairs Sub-Committee and the
Assessment Committee there was no answer of who had responsibility overall for assessment.
As one respondent stated “but if you don’t have somebody who keeps their eye on it, how do you
ever make progress on it? If it is just a one off about it.” In particular relating to the co-
curricular, it was further articulated that while the sub-committee existed that the Academic
Committee does not “vote on anything…I have no idea what sort of structure of governance we
have regarding different roles of faculty and staff.”
Newer Stakeholders to Provide Feedback Within Formal Settings
Within interview data 70% of respondents indicated due to North College’s focus on
assessment newer stakeholders were engaged in this work. In particular, it was voiced that this
representation rests mainly within AAC and the Student Affairs Sub-Committee. Within AAC
respondents voiced the number of new employees to the institution who were now a part of this
committee, most joining the institution within the past three years. Forty percent of respondents
specifically mentioned the new employees on AAC and their bringing additional feedback on
assessment. The creation of the Student Affairs Sub-Committee, within the last year, is equal in
members to that of the Assessment Committee. The Student Affairs Sub-Committee brought 12
new stakeholders into the formal setting, accounting for overlap of two members who also serve
on the Assessment Committee. As it relates to the organizational setting and model, there were
however, negative issues raised in relation to this, though by a minority of respondents. As one
respondent articulated highlighting autonomy as a challenge, “many folks do their own thing too,
at the same time. There is so many different ways that the same thing is done there, that I,
sometimes it boggles my mind.” Despite this, it was also stated there was work being done
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 132

“making sure we are on the same page, because even though we are a small institution
sometimes we are not, you know, coalescing around similar language, similar frameworks.” As
such the needs of new stakeholders in formal settings was not validated.
Commitment and Resources Devoted to Assessment of Global Education
One hundred percent of respondents stated that they believe North College is committed
to the assessment of global education, with 20% feeling strongly agree and 80% agree (Figure
23).
When asked specifically if “I feel supported by North College to conduct assessment
within my area” the percentage decreased 10%, with 90% of respondents answering that they
agreed or strongly agreed (Figure 24.)

Figure 23. Respondent Belief North College Is Committed to Assessment of Global Education

2
8
0 0
0
2
4
6
8
Strongly agree Agree Disagree Strongly disagree
Number of responses.
Respondent Belief North College is Committed to Assessment of Global
Education.
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 133

However, when the question shifted to “I feel supported by North College to conduct
assessment across areas on the institution” the percentage who responded positively decreased to
60% and 40% disagreed (Figure 24).

Figure 24. Respondent Sense of Institution Support for Assessment
Corresponding to prior organizational findings, despite a sense of institutional
commitment to assessment, respondents did not feel there were resources within the institution
across multiple areas to implement a multilayered model that includes global education (Figure
25). One respondent addressed this issue in stating,
I think there is an attempt to say here’s this great sort of structure that we want to use for
assessment, but I don’t know that people feel like I can plug into that…. it’s hard, there is
some generally good thoughts about how this could go, but…
1 1
8
5
1 4
0
2
4
6
8
10
I feel supported by North College to conduct
assessment within my area.
I feel supported by North College to conduct
assessment across areas of the institution.
Number of resposnes.
Respondent Sense of Institution Support for Assessment.
Strongly agree Agree Disagree Strongly disagree
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 134

This highlights that resistance is likely low, but that how to participate in assessment
strategically is a need. Despite the belief that the college is committed to assessment of global
education and feeling support from the college, 70% percent of the sample group disagreed that
North College has the staffing resources to implement assessment. Specific to staff, one
respondent in the interview stated, “I also think that staff need to feel supported in that work, but
again, shifting a culture where assessment isn’t done, or it’s in an ad-hoc manner, to one where it
is consistently done” takes time.
When asked about infrastructural resources, the percentage who disagreed increased to
90%, including 10% who strongly disagreed. One respondent stated, “I would guess we are
probably putting in enough to satisfy the letter of the external expectations, because we have to.”
Respondents were evenly split on whether North College staff has the necessary knowledge.
When questioned on faculty with the necessary knowledge, the disagree rate increased to 70%,
including 10% who strongly disagreed.
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 135

Figure 25. Respondent Sense of Institutional Resources to Implement Multilayered Assessment
of Global Education
The organizational needs of resources and commitment by the institution to support
assessment were corroborated in the interview. One hundred percent of respondents articulated
that the institution could do more to support assessment. One respondent described it as the
“chicken and egg” as assessment had to be done, and was worked on, but there was not a
strategic organizational framework they could align to. Another respondent, when discussing
resources, stated, “I know we are always supposed to look at ourselves as operating in not a
scarce resource environment, but it’s difficult to do anything really well here…and this would be
no exception.” While 60% did articulate an awareness of resources were limited and that the
3
1
5
3
7
8
5
6
1 1
0
2
4
6
8
10
North College, today, has the
staffing resources to implement a
multilayered assessment model that
includes Global Education.
North College, today, has
infrastructural resources to
implement a multilayered
assessment model that includes
Global Education.
North College, today, has staff with
the necessary knowledge to
implement a multilayered
assessment model that includes
Global Education.
North College, today, has faculty
with the necessary knowledge to
implement a multilayered
assessment model that includes
Global Education.
Number of resposnes.
Respondent Sense of Institutional Resources to Implement Multilayered
Assessment of Global Education.
Strongly agree Agree Disagree Strongly disagree
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 136

institution was likely doing what it could, this did not cause the need not be validated. One
respondent articulated the following about how the institution is approaching assessment:
I think, that North is right now doing exactly the way that fits with North’s culture, for
better and for worse….so there is a tension investing the resources that are probably
actually needed to give us that kind of assessment that could really be useful, but that
isn’t happening.
Resources that were listed in the interviews as needed included more staff time (faculty
or other) devoted to assessment work, including 40% of respondents giving examples of other
institutions where full-time employees were devoted to this work. Another example, cited by
80% of respondents in the interview, was increased public resources for assessment for the
institutional community, including information on the work being done, data, training, and
resources. Interestingly, while North College has a website discussing assessment, more than
70% of respondents specifically mentioned there was no such website they could go to for
information. Likewise, 60% of respondents highlighted that North College had, two years’ prior,
committed to the development of an area of institutional planning and effectiveness and had seen
no work from this initiative, including a website where common data could be accessed. This
commitment that was not fulfilled was articulated by this same 60% as directly linked as a
resource need for assessment. Several respondents highlighted the need to “invest in institutional
research” and “clarity about information…it’s been a problem of information flow, I believe.” In
summing up how the institution approached assessment, underscoring both the cultural model
and setting, one respondent said North College is “building our assessment program in that sort
of scrappy, entrepreneurial way, that seems to fit the culture here.” As such, while the need for
long-term commitment to multilayered assessment of global education at North College is not
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 137

validated, however, the needs of insufficient resources devoted to assessment as well as
insufficient support to develop new types of assessment are validated.
Support of Learning and Motivation in Assessment and Development of New Types of
Assessment
Linked to the question of resources, in the interviews, 80% of respondents stated that the
college needed to do more training around assessment, as one respondent reinforced in the
interview to simply “understand what is going on.” Sixty percent of respondents in the interview
highlighted the institutional culture of learning and training in other areas, showing low
resistance to undertaking more learning to develop new types of assessment. This theme was,
however, linked back to knowledge in that almost all employees are somehow engaged in
assessment work. The need for training, as one respondent stated, must “bring faculty and staff
together.” Other respondents went further with this, with one stating that “I think that here we
are cultural, very much, I think we like having faculty in a driving role of assessment” and yet
another added that if “faculty take leadership, then we need to be helping people have enough
time in their schedule to support them.”
As discussed in the motivational needs, there was unanimous data in terms of motivation
to partake in training. As one respondent stated, I think many would “be more proactive learners
if there was a certain amount of release time.” Linked back to the need for organizational clarity
and defined goals, another respondent noted how training could “help faculty to understand the
reasons behind it.”

MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 138

Clarity and Resources Due to Prior Financial Crisis

Figure 26. Respondent Belief North College Is Committed to Assessment of Global Education
Of the sample group, 70% were at North College during the financial crisis in fiscal year
2012-2013. To determine if there was a root organizational need stemming from this crisis, a
series of Likert questions were asked only of these 70% (Figure 26) in the survey. The answers,
across a spectrum of possible factors, including staffing, non-staffing resources, goal clarity, task
time, showed consistently this possible root organizational need was not validated. Respondents
did not feel that the prior financial crisis negatively affected current efforts at assessment. Given
these consistent data from the survey, this need was removed as a question from the interview.

1 1 1 1
2
3
1 1
5 3
3
4
5
4
1 1 2 1 2
0
1
2
3
4
5
6
7
Decreased importance I
placed upon it.
Decreased non-staffing
resources dedicated to it.
Decreased staffing
resources dedicated to it.
I was unclear if it
remained a goal of the
University.
I had less time to
undertake assessment due
to other tasks I had to
assume specifically
because of the financial
crisis.
The financial crisis of
2012 continues to affect
assessment for me today.
Respondent Sense Financial Crisis of 2012 Affected Assessment.
Strongly agree Agree Disagree Strongly disagree
Number of responses. 3 sample group excluded as were
not at the institution at the time of financial crisis.
I was unclear if
it remained a
goal of the
institution.
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 139

Synthesis of Results and Findings for Organization Needs
Of the nine identified organizational needs, six were validated (Table 11). These needs
ranged across the framework of Gallimore and Goldenberg (2001) cultural settings and models.
As was seen in the data, often automated, cultural models are shared mental schema or normative
understandings of how the world works or ought to work (2001) such as a clear structure of
accountability as well as clearly defined goals relating to assessment are identified organizational
needs. Likewise, through the data collection, cultural context was examined and highlighted by
several respondents in the interview. In particular, data were able to discern how within the
environment of North behavior is undertaken without awareness by those taking it. For example,
moving forward with assessment work across the institution while simultaneously
acknowledging that additional training and resources are needed, but not pressing for these to
occur despite a stated belief that the culture is one that supports learning and motivation.
Organizational needs were validated in terms of the requirement for more clarity in and
across committees in reporting, openness, chain of command and transfer of decisions, which
linked to and reinforced the organizational need of a structure for greater accountability. This
was seen, in particular, when bridging the curricular and co-curricular SLOs assessment.
Likewise, these two areas also reinforced the need for clearly defined goals related to
multilayered global education assessment, a need which was also validated in the knowledge
realm given the differences between the knowledge to align assessment across the curricular and
co-curricular. The sense of a long-term commitment to multilayered assessment of global
education at North College, a positive need, was not validated. However, the need for resources
devoted to assessment was validated. This resource need linked to that of the support to develop
new types of assessment which was likewise validated. The environment of North College does
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 140

support an environment of learning and motivation, a positive need, which was validated. The
two needs not validated were newer stakeholders to provide feedback in formal settings, and the
prior financial crisis having an effect on assessment currently.
The validation of these needs suggests that myriad organizational needs in strategic
planning and resources across the institution in the realms of cultural settings and models must
be addressed. The data showed less need for addressing non-participation and resistance, but
that autonomy between North College faculty, staff and administrators in relation to goal clarity
and accountability must be addressed if the institution is to achieve the goal of innovating a
systemic multilayered assessment model for global education.
Table 11
Summary of Organization Needs Validated & Not Validated
Assumed Need Validated Not Validated
Need clarity in and across committees in reporting, openness,
chain of command and transfer of decisions.
√
Need a structure of accountability. √
Need clearly defined goals related to multilayered
Global Education assessment.
√
Need newer stakeholders to provide feedback
within formal settings.
X
Need long-term commitment to multilayered
assessment of Global Education at North College.
X
Need resources devoted to assessment. √
Need environment that supports learning and motivation in
assessment.
√
Need support to develop new types of assessment. √
Need of clarity and resources due to prior financial crisis. X

Summary of Validated Knowledge, Motivation and Organization Needs
The research confirmed needs identified in the literature review. The survey results,
interview findings, and document analysis from the respondents validated some of the assumed
needs in the dimensions of knowledge and skills, motivation and organization based on the needs
analysis framework. Amongst the 27 assumed needs, 19 were validated and eight were not
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 141

validated. One additional need was uncovered during the interviews. Table 12 summarizes the
findings.
On the dimension of knowledge and skills, all of the 11 assumed needs were validated
and one additional need was uncovered during the interviews (Table 12). These needs ranged
across Anderson and Krathwohl’s (2001) framework based on Bloom’s taxonomy, including
factual, conceptual, procedural, and metacognitive. Unmet needs were found in having common
definitions, procedural and conceptual methods, and aligning conceptual models of student
learning within the realms of, as well as across, curricular and co-curricular for faculty,
administrators and staff including the use of AAC&U rubrics. Additionally, knowledge of
common assessment cycles across the institution, and how to share and access assessment
information and data was validated in relation to measuring success of assessment and ability to
monitor it. Metacognitive knowledge was validated. In this realm, respondents expressed
knowledge of both individual as well as institutional needs, and limited success at, assessment
broadly and more specifically of global education. The validation of these knowledge needs
suggests that myriad knowledge needs in the realms of factual, conceptual, procedural, and
metacognitive must be addressed within North College including faculty, staff and administrators
to move forward with developing systemic multilayered assessment of global awareness.
On the dimension of motivation, two of the eight assumed needs were validated (Table
12). These needs ranged across self-efficacy, interests, values, and goal orientation (Pintrich,
2003). Within these five social-constructors categories are more specific levels of beliefs, all of
which can affect the motivation indicators of choice, persistence, and effort. As only two needs
were validated, there is seen a high level of motivation at North College to conduct assessment,
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 142

and in particular to develop new types of assessment that include global education. Data shows
active choice and persistence in conducting assessment work, despite needs in other areas.
On the dimension of organization, six of the nine assumed needs were validated and three
were not validated (Table 12). These needs ranged across the framework of Gallimore and
Goldenberg (2001) cultural settings and models. As was seen in the data, a clear structure of
accountability as well as clearly defined goals relating to assessment are organizational needs.
Likewise, data were able to discern how within the environment of North behavior is undertaken
without awareness by those taking it. For example, moving forward with assessment work
across the institution while simultaneously acknowledging that additional training and resources
are needed. Organizational needs were validated in terms of the requirement for more clarity in
and across committees in reporting, openness, chain of command and transfer of decisions,
which linked to and reinforced the organizational need of a structure for greater accountability.
This was seen, in particular, when bridging the curricular and co-curricular SLOs assessment.
Likewise, these two areas also reinforced the need for clearly defined goals related to
multilayered global education assessment, a need which was also validated separately. The
sense of a long-term commitment to multilayered assessment of global education at North
College, a positive need, was not validated. However, the need for resources devoted to
assessment was likewise validated. This resource need linked to that of the support to develop
new types of assessment. The environment of North College does support an environment of
learning and motivation, a positive need.
The validation of these needs suggests that myriad organizational needs in strategic
planning and resources across the institution in the realms of cultural settings and models must
be addressed. The data showed less need for addressing non-participation and resistance, but
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 143

that autonomy between North College faculty, staff and administrators in relation to goal clarity
and accountability must be addressed if the institution is to achieve the goal of innovating a
systemic multilayered assessment model for global education.
The next chapter presents potential solutions for the validated knowledge, motivation and
organizational needs. All of the solutions that will be offered in the next chapter are evidence-
based and derived from relevant research and case studies.
Table 12
Summary of Assumed Knowledge, Motivational and Organizational Needs & New Needs
Assumed Need Validated
Not
Validated New Need
Knowledge
Need knowledge of who is responsible for assessment
at North College.
√

Need clear definition of assessment. √
Need knowledge of student learning outcomes. √
Need knowledge of, and current success at, assessment
at North College.
√

Need ability to monitor their skills related to
assessment and make needed adjustments.
√

Need knowledge of how to develop, or steps involved
in, developing an assessment model.
√

Need knowledge of links between assessment in
curricular and co-curricular areas.
√

Need knowledge of AAC&U rubrics. √
Need operational knowledge of Global Education
assessment.
√

Need knowledge of what multilayered assessment
means.
√

Need skills to develop and integrate assessment within
and across academic and co-curricular areas.
√

Student need knowledge of learning outcomes. √
Motivation
Need motivation to serve on committee(s). √
Need understanding of importance of Global Education
to Mission of North College.
X

Need to see value of AAC&U Rubrics in their work. X
Need confidence in their ability to map student learning
outcomes across curriculum, and to the co-curricular.
√

MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 144

Table 12, continued
Assumed Need Validated
Not
Validated New Need
Knowledge
Need desire to put in effort to learn new types of
assessment models.
X

Need to see value to put in effort to design new
assessment models.
X

Need to see value of Global Education student learning
outcomes assessment to their work.
X

Need to see value in the importance of assessment
between academic and co-curricular areas.
X

Organization
Need clarity in and across committees in reporting,
openness, chain of command and transfer of decisions.
√

Need of a structure of accountability. √
Need clearly defined goals related to multilayered
Global Education assessment.
√

Need newer stakeholders to provide feedback within
formal settings.
X

Need long-term commitment to multilayered
assessment of Global Education at North College.
X

Need resources devoted to assessment. √
Need environment that supports learning and
motivation in assessment.
√

Need support to develop new types of assessment. √
Need clarity and resources due to prior financial crisis. X

MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 145

CHAPTER FIVE: SOLUTIONS, IMPLEMENTATION AND EVALUATION
The innovation goal for this study was by September 2017, using the AAC&U Global
and Intercultural Rubrics, North College administrators will implement a multilayered
assessment model for global education learning outcomes in 100% of majors as well as 50% of
activities of the division of student affairs. A total of 27 assumed needs thought to be
contributing to the challenges the institution faces in the implementation of a multilayered
assessment model for global education learning outcomes were identified and investigated
through a needs analysis framework (Clark & Estes, 2008). The previous chapter presented
findings that validated 18 of these needs and identified one additional need. The purpose of this
chapter is to offer solutions to address 18 of these validated needs.
The needs analysis framework separates the categories of knowledge, motivation, and
organization in order to clearly identify root needs that may be inhibiting the innovation.
However, as was highlighted in findings in Chapter Four, these categories do not operate in
isolation. As Figure 27 shows, adapted from Knoster, Villa, and Thousand (2000) to align to
Clark and Estes (2009), complex change is the collective result of managing all categories to
enact innovation. The interconnected nature of the categories permits solutions that address
multiple causes of root needs in a systemic and holistic manner, which must be accounted for in
developing successful solutions.
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 146

Figure 27. Managing Complex Change
The proposed solutions in this chapter are organized in cascading order of impact and
importance, with each proposed solution reinforced by the next. While each solution has
specific, anticipated outcomes, overlap in deployment is actually required. Following the
discussion of proposed solutions, an implementation and evaluation plan is proposed.
Validated Needs Selection and Rationale
The data confirmed that knowledge and organizational needs exist, impeding
stakeholders’ ability to innovate a systemic multilayered model of assessment for global
education, which are summarized in Table 13. Three solutions proposed in this chapter address,
to some degree, all 18 needs.
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 147

Table 13
Summary of Assumed Knowledge, Motivation and Organizational Needs & New Needs
Knowledge
Need knowledge of who is responsible for assessment at North College.
Need clear definition of assessment.
Need knowledge of student learning outcomes.
Need knowledge of, and current success at, assessment at North College.
Need ability to monitor their skills related to assessment and make needed adjustments.
Need knowledge of how to develop, or steps involved in, developing an assessment model.
Need knowledge of links between assessment in curricular and co-curricular areas.
Need knowledge of AAC&U rubrics.
Need operational knowledge of Global Education assessment.
Need knowledge of what multilayered assessment means.
Need skills to develop and integrate assessment within and across academic and co-curricular
areas.
Student need of knowledge of learning outcomes.
Motivation
Need motivation to serve on committee(s).
Need confidence in their ability to map student learning outcomes across curriculum, and to
the co-curricular.
Organization
Need clarity in and across committees in reporting, openness, chain of command and transfer
of decisions.
Need a structure of Accountability.
Need clearly defined goals related to multilayered Global Education assessment.
Need resources devoted to assessment.
Need environment that supports learning and motivation in assessment.
Need support to develop new types of assessment.

From the data, two key elements were identified that relate to the validated needs that
should be illustrated in the visual representation of systemic multilayered assessment. These are
incorporated into Figure 28. First is the modes of assessment of global education, or ensuring
there is clarity around the various means by which assessment is undertaken as well as the ways
in which this data is captured so as to be examined across the numerous areas of an institution
(e.g., course outcomes, departmental assessment, accreditation.) Second, and linked to the first,
is the procedural and conceptual framework to capture this data longitudinally, which crosses
into both the knowledge and organizational realms.
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 148

Figure 28. A Visual Representation of Systemic Multilayered Assessment Revised
Modes of assessment
of Global Education
Assessment of Global Education over time
Multilayered intersection
areas of Global
Education assessment
Canny, E. (2016)
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 149

These two additions are central to the selection of, and rationale for, the proposed
solutions. Both, mirroring the data, are embedded within the knowledge and organizational
needs. As illustrated in Figure 27, without addressing knowledge needs the innovation will
likely result in anxiety. This was seen in the data through the awareness of North College’s
current issues around assessment. Second, without the organizational structure to support such a
model of assessment, frustration will also likely result. This, again, was seen in the data
findings. As motivation at North College was high, the focus on both knowledge and
organizational needs are critical, as at the moment, there is little resistance to assessment and
there was seen high desire to improve assessment. Without balancing the three areas, motivation
could quickly move to resistance.
Solutions
The proposed solutions are all, in some way, related to a facet of a learning organization.
There are six activities that provide the structure for a learning organization. First is seeking to
explore new knowledge by systemic problem solving. Second is utilizing personal experience
for learning. Third is learning from others. Fourth is trying new processes. Fifth is sharing
knowledge within the organization. Sixth is assessing learning (Dill, 1999). Though the
solutions do not follow the exact framework, they encompass aspects of all categories. The three
solutions which fall under two broad themes are summarized in Table 14:
A. Institutional establishment of measures and capacity for excellence to reinforce
assessment accountability;
B. Stakeholder capacity to define, implement and encourage standards of excellence in
assessment.

MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 150

The solutions are
1. Establish clear goals, communication and oversight of assessment at North College;
2. Establish knowledge and capacity to capture student learning outcomes;
3. Establish a comprehensive stakeholder assessment development program which includes
a system of rewards and incentives embedded within it.
Table 14
Summary Proposed Solutions
Theme Proposed Solution
Institutional Establishment of
Measures and Capacity for
Excellence to Reinforce
Assessment Accountability
(1) Establish clear goals, communication and oversight of
assessment at North College.
(2) Establish knowledge and capacity to capture student learning
outcomes.
Stakeholder Capacity to
Define, Implement and
Encourage Standards of
Excellence in Assessment
(3) Establish a comprehensive stakeholder assessment
development program.

The underlying theme of the proposed solutions is that North College should incorporate
activities of a learning organization that supports professional learning with regard to
assessment. In the case of North College, as validated by the data, this professional learning
would encompass faculty (full-time and adjunct), staff and administration as all play a role in
assessment. A learning organization is defined as one that systemically problem solves through
engagement in activities of learning and reflection for the purpose of continuous improvement
(Dill, 1999). North College has a history of working with continuous improvement models
under the prior HLC accreditation framework as well as professional learning in other areas such
as retention, pedagogy and intercultural development.
Hoekstra and Crocker (2015) provide a framework for professional learning that will be
applied as the broad solution model. First, professional learning should, through engagement in
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 151

focused activities, result in both improved practices as well as the continued ability to perform in
improved behaviors. Second, Hoekstra and Crocker (2015) highlight that such learning can take
different forms, being intentional and conscious or beyond the learner’s consciousness as well as
include both formal and informal activities. Both of these factors would be central to the
innovation of a system multilayered assessment model as the training will need to be engaging in
both formal and informal ways (e.g., building assessment protocols and testing them) as well as
theoretical. Saroyan and Trigwell (2015) highlight the importance of professional learning,
including the dynamic nature it must take. This is central to the solutions as well given the
changes North College is undertaking that will be discussed as well as the evolving nature of
both assessment and global education. Through integration of practices that support the
innovation of a systemic multilayered assessment model for global education at North College
into a learning organization framework, the institution would make a public commitment to
support all stakeholders in both achieving excellence in as well as reinforcing the importance of
assessment (Dill, 1999; Saroyan & Trigwell, 2015).
Therefore, at the center of the solutions is the role assumed by the institution across
myriad areas and ensuing clarity and alignment across these areas. Though faculty are formally
charged with oversight of assessment, the cultural settings and models within the organization
need to support the ability of all stakeholders to attain the knowledge they need to assess student
learning. Additionally, the organizational conditions need to exist to motivate stakeholders to
actively choose, expend the effort, and persist in continually improving their assessment models
and plans. Though only two of the assumed motivation causes were validated, the proposed
solutions will impact motivation as well because of the interconnected nature of the three
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 152

categories. The three proposed solutions are multifaceted. Each affects the other in some way
and directly addresses some causes and indirectly addresses others.
Solution Theme One: Institutional Establishment of Measures and Capacity for Excellence
to Reinforce Assessment Accountability
Linking the models of (Dill, 1999), Hoekstra and Crocker (2015) and Barr and Tagg
(1995), North College should systemically establish measures and capacity to assess SLOs. To
this end, North College needs to construct a culture that shifts the institution as a whole to a
learning organization regarding assessment. Linking the focus on student learning from Barr and
Tagg (1995), Dill describes measuring learning as “developing indicators of organizational
performance, which can be used to evaluate whether learning is actually occurring” (p. 143).
North College has no central system for the measuring of SLOs, and, there is lack of clarity
around who has oversight of assessment connected to confusion on where data the institution
does have is located. The results and findings from the survey and interview showed that,
though stakeholders have high motivation, they a lack knowledge on definitions, processes,
procedures, and where to find current data on assessment. Additionally, the findings showed the
need for greater institutional commitment to resources to enable assessment. As respondents
highlighted, the lack of data, coupled with unclear expectations, fails to provide stakeholders
with the needed information on whether SLOs are achieved and, if so, where. Lack of
institutional consequences causes unclear expectations, and lack of additional measures does not
give stakeholders, or the institution, benchmarks to compare to and work from. In order to
bridge the organizational need of accountability, the institution should emphasize the already
high value of assessment amongst stakeholders through the implementation of processes that
reiterate and reinforce the expectations and value of assessment at multiple levels (e.g.,
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 153

curricular, co-curricular, class outcomes, institutional outcomes) so that assessment becomes
embedded into the organizational culture.
Proposed Solution One: Establish Clear Goals, Communication and Oversight of
Assessment at North College
Without this first solution to address organizational needs and clarity the other solutions,
while possibly overcoming identified needs within knowledge, will not ultimately overcome the
institutional barriers to innovation of a systemic multilayered model of assessment for global
education.
The implementation plan of the first proposed solution, Establish clear goals,
communication and oversight of assessment at North College, may be seen in Table 16. There
are three key action steps to this solution:
1. Establish clear goals and roles for each committee and major stakeholder groups.
2. Re-draft existing handbooks and policies.
3. Develop clear mechanism of communication.
Proposed Solution Two: Establish Knowledge and Capacity to Capture SLOs
An investment by North College establishing more vigorous assessment tools and
systems for data collection as well as more professional training opportunities will help to signal
the institution’s commitment to assessment. Unless all three components are addressed by the
institution, there will continue to be barriers for development of a systemic multilayered
assessment model. Combined, these three areas will offer stakeholders opportunities to
continuously improve assessment with actual student assessment data, provide enhanced tools
that the institution does not currently have to collect SLOs and conduct assessment, while
simultaneously signaling clearer oversight and expected measures to be achieved. By
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 154

establishing clear measures, the institution develops a method to determine the effectiveness of
professional training opportunities as well as giving stakeholders a clearer understanding of the
key performance indicators they should be working toward. With specific goals, and clear
oversight, stakeholders know what the institution expects, feel supported to continually improve,
and can be held accountable to those measures.
The implementation plan of the second proposed solution, Establish knowledge and
capacity to capture SLOs, may be seen in Table 17. There are three key action steps to this
solution:
1. Determine what data currently exists on the general education SLOs.
2. Determine what software should be utilized to track SLOs.
3. Conduct ongoing trainings, including pre-post assessment to set benchmark data of
stakeholders’ knowledge as well as trainings.
Solution Theme Two: Stakeholder Capacity to Define, Implement and Encourage
Standards of Excellence in Assessment
“Excellence” is an abstract term without an agreed upon definition in higher education,
and the terms of excellence will vary depending on the purpose and context of the specific
situation (Hill et al., 2003; Krause, 2012; Little & Locke, 2011). In order to establish standards
of excellence, North College will need to consider its stakeholders and overall purposes for
assessment when determining what excellence means for the institution. The finding from the
survey and interviews demonstrated that the stakeholders see the importance of assessment
ranging from their own work to the needs of the institution. However, there aren’t clear
definitions and understanding of what assessment is, should mean, and how it can be utilized to
show excellence of student learning that is believed to be occurring. Within the proposed
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 155

solutions, there are mechanisms built into the implementation plan, which are meant to ensure
that this overarching definition and establishment of a definition of “excellence” is achieved.
In order to achieve the innovation of implementing a systemic multilayered process for
assessing student learning in global education, North College needs to prioritize the value of
excellence in assessment by investing the necessary resources to support professional
development in assessment broadly. Findings from the survey and interviews affirmed that the
limited and inconsistent resources for assessment affect stakeholders’ abilities to consistently
conduct, let alone create or develop, new models of assessment that meet institutional needs, let
alone excellence. The limited and inconsistent investment in assessment support does not signal
to the stakeholders that the institution values excellence in assessment of SLOs (Austin &
Sorcinelli, 2013; Dill, 1999; Tarr, 2010). While co-curricular assessment was found to be of
importance to all in the findings, it also not a requirement for accreditation of the institution by
HLC. Likewise, as the faculty oversee assessment of the curricular realm, utilizing solutions
focused on their development is critical.
Austin and Sorcinelli (2013), in their examination of faculty development found that a
significant element for excellence is faculty development as faculty directly deliver education to
students and must be able to teach and adapt to the expectations of the complex postsecondary
environment. Though the establishment of a more comprehensive assessment development
program North College can demonstrate its commitment to instructional excellence measured
through assessment, providing opportunities for all to close the knowledge needs.
In order to fully support stakeholder continual development, and address knowledge
needs, North College must go beyond offering a handful of workshops. Additional professional
development opportunities should be developed to continually offer stakeholders opportunities to
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 156

advance their assessment skills (Lyons, 2007; Tarr, 2010). Ambrose et al. (2010) suggest that
one of the principles of learning is the acquiring of component skills, practice integrating them,
and application of what they learned. As such, the assessment development program should be
informed by adult learning theories, such as andragogy, and be continual opportunities to
demonstrate and showcase the strategies that are most effective (Ambrose et al., 2010; Merriam
& Bierema, 2013).
In their study of professional learning, Saroyan and Trigwell (2015) found that
stakeholders must have internal or external motivation to prompt them to choose to participate,
exert effort and persist in professional learning. With clear measures of excellence and enforced
accountability, North College should establish a system of rewards and incentives to encourage
stakeholders to continually improve their assessment work. While findings showed that
motivation was high, the implementation plan to develop a systemic multilayered
assessment model will take time. Simultaneously, the stakeholders are engaging in assessment
work that findings showed they recognize as inadequate. As such, a system of rewards and
incentives would provide motivation during the period of implementation that would undercut
any frustration with assessment effort that is currently undertaken, and increase continual effort
and persistence in the assessment development program. Having a clear rewards and incentive
system that recognizes excellence in assessment, and reinforces continual improvement in it, will
be central to sustained motivation as the institution moves forward with the innovation (Lyons,
2007; Tarr 2010). Given the breadth of stakeholder (e.g., faculty, staff, administrators; including
new stakeholders at the institution to those with decades of employment; positionality ranging
from mid-level to senior leadership) the system of rewards and incentives will be multifaceted.
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 157

Such a system would also need to account for motivational factors (e.g., intrinsic or external) as
was found in the motivational need data in Chapter Four.
Proposed Solution Three: Establish a Comprehensive Stakeholder Assessment
Development Program
As the data in Chapter Four highlighted there is high motivation amongst stakeholders in
regards to assessment. This motivation should be utilized as a tool to further proposed solutions
for knowledge and organizational needs. To this end, embedded in the three solutions is an
overarching definition of setting of standards of excellence with regard to assessment. First, to
ensure that the stakeholders, as well as the institution, are moving toward a systemic
multilayered model of assessment, North College must define “excellence” for the institution and
stakeholders in this context, and then set standards of excellence. Second, stakeholders then
need the capacity to implement and be continually encouraged to meet these standards of
excellence regarding assessment.
The implementation plan of the third proposed solution, Establish a comprehensive
stakeholder assessment development program, may be seen in Table 18. There are four key
action steps within this solution:
1. Determine stakeholder groups engaged in assessment at various levels.
2. Evaluate the training needed by each stakeholder group.
3. Develop specific training plan for each group as well as broad training on assessment.
4. Establish a system of rewards and incentives to encourage stakeholders to continually
improve assessment efforts aligned to training plan.

MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 158

Implementation Plan
The solutions provided combine to address the validated needs, are connected by two
overarching themes, and all support the development of North College into a learning
organization in the realm of assessment. As there is already cultural context for institutional
learning, such a plan draws upon existing cultural models and context, but in a new realm. With
the implementation of each solution, North College will develop as an institution that instills into
its culture a practice of learning for continuous improvement around assessment, which is
required for both improvement of existing assessment as well as the innovation. Table 15
provides a summary of solutions with their implementation timeline.
Table15
Summary of Solutions and Timeline
Solution Start Date End Date
Some aspects ongoing
1. Establish clear goals, communication
and oversight of assessment at North
College.
September, 2016 May, 2018
2. Establish knowledge and capacity to
capture student learning outcomes.
September, 2016 January, 2018
3. Establish a comprehensive
stakeholder assessment development
program.
September, 2016 November, 2017

The solutions to be implemented is a component of the North College’s overall
assessment strategy, in particular a training plan creating an organizational culture of learning
regarding assessment and focused on what is assessment, what models of assessment are the
institution utilizing and why (e.g., course, major, institutional, co-curricular), and how to develop
assessment protocols within as well as across areas, and time, to assess global education in a
multilayered and systemic model. It is important for North to implement a systemic method for
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 159

assessing global education learning outcomes for three reasons. First, the college needs to
measure these learning outcomes to submit proof of students meeting these stated outcomes to
the Higher Learning Commission of the North Central Association of Colleges and Schools
(HLC). Second, a systemic method for assessing global education learning outcomes enables the
college to confirm that it is meeting the stated goals in its mission statement, and to the extent it
is not, the college can make informed and needed adjustments in teaching and learning practices.
Third, systemic assessment of global education learning outcomes helps ensure that students are
acquiring the skills necessary to function in the 21st century.
Higher education institutions are challenged not only by a shift toward increased
assessment, but also by the need to systemically measure increasingly complex learning
outcomes. As Barr and Tagg (1995) argued, higher education institutions must shift mission,
vision, culture, and structure from providing instruction to producing learning and knowledge.
Barr and Tagg’s broad framework encompassed all graduates and outlined learning assessment
should be done without regard to a particular curriculum or educational experience. As the
framework also focused on learning and knowledge, not teaching, institutions are increasingly
required to reliably assess learning outcomes across graduates and the institution as a whole. As
infusion of global education occurs throughout an entire institution, assessment must be
multilayered, both in location as well as time. Systemic accountability of SLOs relating to
global education, or assessment, proved increasingly problematic due to the multi-dimensional
nature of internationalization. It is, therefore, imperative that assessments occur at the various
levels, in the interaction between the levels, and over time (NAFSA, 2010).

MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 160

Organizational Features Relevant to Implementation
In terms of implementing these three solutions, the institution has particular
organizational characteristics to consider. First, it is highly mission driven, with learning
outcomes as well as motivation of faculty and staff tied to the institutional mission. Second, it
has a heritage linked to both a specific Northern European culture as well as a religious
affiliation. Despite these links now being tenuous (i.e., less than 20% of the student population
is of the same religious affiliation as the institution), they combine to create a culture of
individuality, expected open dialogue around all issues, and a culture of equality on decision
making, all of which continue today. These are amplified by the institutional mission, as well as
the diversity of the institution, creating a highly egalitarian culture where the concept and
application of “power” without all parties in agreement is consistently questioned. As such,
there is lack of clarity on who has decision-making authority, an expectation of discussion
around all decisions or issues, and while highly collaborative, a highly egalitarian culture in this
cultural model also is not strategic in foci or actions (e.g., two groups will focus on the same
work and not communicate with each other). In particular, limitations placed by authority can be
seen as contradictory to the ethos of the institution. As such, linked to the solutions being
implemented, there is an assumed knowledge of assessment at the college as well as
organizational structure to undertake the work, neither of which exist. The training program will
be the first step to ensuring those engaged in assessment work have the factual, procedural, and
conceptual knowledge to conduct assessment. While the organizational issues must be addressed,
these are complicated by the internal cultural model of the institution as outlined above in broad
themes. Given this, the solution focusing on training will both ensure knowledge and be met
with less resistance.
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 161

Moving the training solution forward at North College requires specific action steps,
which are outlined in Tables 16 to 18. These include the three primary solutions, in the
particular order outlined in Table 15. The solutions are organized in a cascading order of impact,
overlapping in timing to reinforce particular needs as well as building capacity in key areas that
are required for subsequent areas. As such, all three solutions will be implemented
simultaneously.
First, is to establish clear goals, communication and oversight of assessment at North
College. Without this first step to address organizational needs and clarity the other solutions,
while possibly overcoming identified needs within knowledge, will not ultimately overcome the
institutional barriers to innovation of a systemic multilayered model of assessment for global
education. Second is the creation of organizational knowledge and capacity training to capture
SLOs assessment data including; tools to track and capture this data; and training around these
tools and processes. Third is the development of a training program around assessment. Within
all three implementation solutions is a coordinated communication plan that addresses
assessment broadly as well as the training program. All three solutions include assignment of
teams with great alignment between them, as well as timing of implementation to scaffold upon
each other, to address root needs to enable North College to implement a systemic method for
assessing global education learning outcomes.
Carrying out these action steps rests on the development of working groups within North
College as well as the use of an external organization. A series of working groups will be
created that, based upon the findings, should enable the solutions to move forward within the
existing organizational structure and cultural model highlighted above. As the innovation of a
systemic multilayered assessment model for global education bridges curricular and co-curricular
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 162

and includes faculty, staff, and administrators, the working groups pull from individuals across
the areas currently charged with assessment as well as heads of academic and co-curricular areas
in an attempt to bring coherence to the training across areas, as well as utilize the high
motivation data showed amongst stakeholders. As the primary working groups include
individuals linked to the entities directly charged with assessment, most of whom are faculty
driven, the head of the Center for Teaching and Learning will be central to implementation of
training since this center already conducts training for faculty. Information technology will also
be included within the teams because data capture is critical to systemic assessment.
Communication and data flow were also raised as issues in knowledge and organization, as such
some groups include marketing and human resources as the training and communication aspects
will need to bridge into the staff and communication realms of the institution to reach a diverse
audience.
An external organization(s) will be identified for specific actions, in particular training,
where outside expertise is needed, something which the institution is familiar with and has
success around on similar initiatives (e.g., intercultural training using the Intercultural
Development Inventory). While the members of the working groups will be central to the
implementation of a plan that aligns to North’s calendar and culture, they also include
individuals who the data showed need training as well. As the training also will need to cover
various knowledge domains (factual, procedural, conceptual) and will be assessed on success in
these realms with possible repetition and/or adjustments needed, the external entity will be
charged with both conducting the training as well as assessing success and needs, allowing the
working groups to focus on other realms of the plan.
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 163

North College is currently undertaking a review of the general education part of the Core
Curriculum, and this review will be leveraged and linked to the knowledge and organizational
changes necessary to implement the solutions, shown in Tables 16 to 18. Relating to knowledge
needs in particular, the review allows for the institution to collect data on where existing SLOs
were embedded, develop a system for ensuring they are continually tracked as the next general
education revision is implemented, and to create as well as train on a systemic system to both
manage and capture data needed for assessment moving forward. As noted above and in the
findings, clarity around goals, data, and working across distinct realms of the institution are
needed. The review and implementation of a new general education core provides a unique
opportunity when aligned to training to both allow the institution to improve assessment now as
well as address the knowledge and organizational needs to innovate moving forward.
Additional Components of Implementation Plan
There are two other factors that were considered when developing the implementation of
the solutions. In fall, 2016 the Board of Regents of North College will vote to explore the
institution shifting from a college to a university. First, while such a shift would not officially
occur until fall 2018, the work of exploring what this alteration would mean in terms of the
structure of the institution provides an opportunity to address the barriers around goals,
communication and oversight of assessment. For example, this work would include an
examination of the faculty handbook, governance structure, as well as how classes and
departments are organized (e.g., College of Arts and Sciences and College of Graduate Studies,
or colleges based upon disciplines.) Both of these will, hopefully, allow the discussions around
the barriers of goals, communication and oversight of assessment to be linked to larger
conversations. This, it is hoped, will enable these conversations to be coupled to other barriers,
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 164

which the institution has acknowledge need to be addressed, as the institution moves through this
exploration of shifting from a college to a university. Second, the exploration shift will also
result in the examination of new systems for managing the institution, communication, and how
students and stakeholders access data (e.g., websites, majors and minors, degree completion). As
such, it is also hoped that the costs associated with the proposed solutions could be considered in
the broader context of the possible institutional shift, as many of these systems will have to be
examined regardless. In particular, these areas provide an opportunity to highlight the new
knowledge need of student awareness of SLOs given the possible restructuring of so many
systems and processes where this information could be placed for students to access.
Implementation Plans: Create an Organizational Culture of Learning Regarding
Assessment
Theme: Institutional Establishment of Measures and Capacity for Excellence to Reinforce
Assessment Accountability
Table 16
Solution #1 Establish Clear Goals, Communication and Oversight of Assessment at North
College
Action Steps Human Resource Roles/
Responsibilities/Capacity
Timeframe
Establish clear goals and roles for each
committee and major stakeholder
group, starting by reviewing the
existing structure and making
recommendations for changes.
Working group consisting of Deans,
Chairs of Assessment Committee,
AAC, GAAC, and Co-Curricular Sub-
Committee, Director of Academic
Administration, Director of General
Education (Working Group 1).
9/16 – 1/17
(5 months)
Re-draft existing handbooks and
policies.
Working Group 1 1/17 – 5/17
(5 months)
Develop clear mechanism of
communication between committees
and major stakeholders. This includes
means, timeline, and links to the
knowledge capture.
Working Group 1 1/17 – 5/17
(5 months)
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 165

Table 16, continued
Action Steps Human Resource Roles/
Responsibilities/Capacity
Timeframe
Take redrafted handbooks and policies
to governing bodies (e.g., Faculty
Senate and University Council) for
approval.
Working group consisting of Faculty
Senate, members of University
Council, Deans, Chairs of Assessment
Committee, AAC, GAAC, and Co-
Curricular Sub-Committee, Director of
Academic Administration, Director of
General Education (Working Group 2).
5/17 – 12/17
(8 months)
Implement new processes and
communication flow Spring 2018 in
alignment with knowledge capture
solution and General Education review
timing.
Working Group 1 1/18 – 5/18
(5 months)

Table 17
Solution #2 Establish Knowledge and Capacity to Capture SLOs
Action Steps Human Resource Roles/
Responsibilities/Capacity
Timeframe
Determine what data currently exists on
the General Education SLOs, where it is
tracked, and how it is linked to any
assessment that is occurring. Determine
how, if possible, this data can be
extracted into new system(s).
Working group consisting of Deans,
Chairs of Assessment Committee and
Co-Curricular Sub-Committee,
Director Center for Teaching and
Learning, Director Academic
Administration, Director of Human
Resources, Registrar (Working Group
1).
9/16 – 2/17
(6 months)

Determine what software should be
utilized to track SLOs, which enables
their update and extraction, across the
institution on a continual basis.
Additionally, that the data links to the
software which is being utilized by the
institution to conduct assessment (e.g.,
export capabilities, artifact upload, etc.)
Working group consisting of Deans,
Chairs of Assessment Committee and
Co-Curricular Sub-Committee,
Director Center for Teaching and
Learning, Director of Academic
Administration, Director of Human
Resources, Director of IT (Working
Group 2).
9/16 – 3/17
(7 months)

Secure all existing SLOs, linked to all
courses at the institution, and load them
into the software.
Working Group 1 3/17 – 5/17
(3 months)

Determine location to share training
materials and information on resources
and software, etc. that is accessible to
participants and linked to the other
locations on assessment and training.
Working Group 2 9/16 – 2/17
(6 months)

MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 166

Table 17, continued
Action Steps Human Resource Roles/
Responsibilities/Capacity
Timeframe
Run tests of software to ensure it is
pulling data required for the various
types of assessment.
Working Group 2 2/17 – 4/17
(4 months)
and then
ongoing
Develop communication plan including
messaging, timing, means, and feedback
loop on training to use software and
clear expectations on what should be
updated and when. Link to location of
where materials are as well as other
means of dissemination.
Working Group 2 9/16 – 2/17
(6 months)

Identify organization(s) and/or
individuals to provide training on
software, as well as within and across
each area (curricular and co-curricular).
Working Group 2 9/16 – 2/17
(6 months)

Develop and conduct post-training
benchmark assessment of stakeholders
as well as trainings.
Identified external organizational
representative(s).
4/17 and
5/17 then
ongoing
Run tests of software to ensure it is
pulling data required for the various
types of assessment. From this point
forward, tests will also be live use of
data for current assessment efforts;
requiring updates to training and
materials as needed.
Working Group 2 4/17 and
5/17 then
ongoing
Evaluate and assess training plan and
activities, make necessary adjustments.
Develop plan for coming year focused
on those stakeholders who have been
trained, and for those who have started
at the institution and not received any
training.
Working Group 2 4/17 and
5/17 then
ongoing
Conduct trainings/showcases of how the
software is working so stakeholders can
see value and the outcomes of the
change.
Working Group 2 5/17 then
ongoing
Ongoing evaluation and assessment of
training plan and functionality of
software, make necessary adjustments.
Develop plan for coming year focused
on those stakeholders who have been
trained, and for those who have started
at the institution and not received any
training.
Working Group 2 5/17 then
ongoing

MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 167

Theme: Stakeholder Capacity to Define, Implement and Encourage Standards of
Excellence in Assessment
Table 18
Solution #3 Establish a Comprehensive Stakeholder Assessment Development Program
Action Steps Human Resource Roles/
Responsibilities/Capacity
Timeframe
Determine stakeholder groups
engaged in assessment at various
levels (e.g., faculty, assessment
coordinators, committee members,
staff).
Working group consisting of
Deans, Chairs of Assessment
Committee and Co-Curricular
Sub-Committee, Director Center
for Teaching and Learning,
Director of Human Resources
(Working Group 1).
9/16 – 1/17
(5 months)

Establish a system of rewards and
incentives to encourage stakeholders
to continually improve assessment
efforts aligned to training plan.
Working Group 1 9/16 – 1/17
(5 months)
Determine available funding and
source(s) of funding across the
institution.
Working Group 1 9/16 – 1/17
(5 months)
Determine location to share training
materials and information on
scheduling, resources, etc. that is
accessible to participants.
Working group consisting of
Deans, Chairs of Assessment
Committee and Co-Curricular
Sub-Committee, Director
Academic Administration,
Director of Human Resources,
Marketing Representative, IT
Representative (Working Group
3).
9/16 – 1/17
(5 months)

Develop communication plan
including messaging, timing, means,
and feedback loop. Link to location
of where materials are as well as
other means of dissemination.
Working Group 3 9/16 – 1/17
(5 months)

Identify organization(s) and/or
individuals to provide training on
assessment holistically, as well as
within and across each area
(curricular and co-curricular). This
will include both on campus training
for larger groups, as well as
conferences and other off-campus
training for targeted stakeholders.
Working Group 1 9/16 – 12/16
(3 months)

MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 168

Table 18, continued
Action Steps Human Resource Roles/
Responsibilities/Capacity
Timeframe
Align scheduling for trainings to
academic calendar, reporting dates,
across areas (curricular and co-
curricular) needs and limitations.
Will include integration to existing
assessment programming which is not
focused on training.
Working Group 1, and identified
external organizational
representative(s).
9/16 – 2/17
(6 months)

Launch system of rewards and
incentives to encourage stakeholders
to continually improve assessment
efforts.
Working Group 1 1/17 and then
ongoing
Develop and conduct a pre-training
benchmark assessment of
stakeholders. Reconfirmation of prior
evaluation.
Identified external organizational
representative(s).
3/17 and
5/17 then
onwards
Develop and distribute supplemental
materials on assessment. Time to
training schedule, pre-training
benchmark data, and communication
plan.
Working Group 1 3/17 and
5/17
Conduct trainings (ongoing). Identified external
organizational representative(s).
3/17 then
ongoing
Develop and conduct post-training
benchmark assessment of
stakeholders as well as trainings.
Identified external
organizational representative(s).
3/17 and
5/17 then
ongoing
Develop and distribute additional
supplemental materials on
assessment. Time to training
schedule, post-training benchmark
data, and communication plan.
Working Group 1 3/17 and
5/17 then
ongoing
Evaluate and assess training plan and
activities, make necessary
adjustments. Develop plan for
coming year focused on those
stakeholders who have been trained,
and for those who have started at the
institution and not received any
training.
Working Group 1 3/17 and
5/17 then
ongoing
Evaluate and assess rewards and
incentives program, make necessary
adjustments. Develop plan for
coming year focused on those
stakeholders who have been trained,
and for those who have started at the
institution and not received any
training.
Working Group 1 5/17 and
11/17 then
ongoing

MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 169

Evaluation Plan
After implementation, the proposed solutions should be evaluated to determine if they
were effective in closing the knowledge, motivation, and organizational performance needs.
Kirkpatricks’s (2006, 2007) four levels of evaluation form the foundation of the evaluation plan.
Level one measures reactions, or how participants like a particular program (Kirkpatrick, 2006).
Level two measures learning, or the impact of the program on participants and what principles,
facts and techniques they understood (Clark & Estes, 2008; Kirkpatrick, 2006). Level three
measures behavior or if the participant utilizes and applyies what they learned in the training, and
whether the gains made are lasting (Clark & Estes, 2008; Kirkpatrick, 2006). The final level
measures results, which ultimately goes back to the original need, evaluating if the solutions
achieved the desired results (Kirkpatrick, 2006).
Level 1: Reactions
The first level of evaluation will measure how stakeholders respond to change. If North
College is successful in implementing the solutions to develop into a learning organization that
supports assessment, coupled with clearer goals and oversight of assessment, the expected
reaction is that the stakeholders will notice that support and professional learning opportunities
in this area are available, coupled with clearer expectations and responsibilities regarding
assessment and broad engagement with the assessment development program.
To evaluate these changes, specific elements need to be identified as useful to measure,
and feedback should be collected with a simple instrument (Kirkpatrick, 2006). Immediate
honest feedback is critical to this level of evaluation (Kirkpatrick, 2006), so the surveys should
be distributed directly following engagement with any part of the assessment development
program.
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 170

After the stakeholders experiences new support and training offered by the institution,
they should be asked to complete a simple survey. The survey should utilize questions that are
answered on a continuous scale, such as a 5-point Likert scale of strongly agree to strongly
disagree, so that the results can be quantified (Kirkpatrick, 2006). The questions should elicit
feedback for the effectiveness of the organizational support as well as assessment development
program.
The results of the survey should be reviewed, as outlined in the implementation plan,
with a baseline score established for the continuous scale questions. If scores do not meet the
established baseline, then considerations need to be made to change the next intervention
(Kirkpatrick, 2006).
Level 2: Learning
The second level of evaluation will measure how much the stakeholders who engage with
assessment of student learning, have learned. If the solutions are effective, there should be
observable change in skill, knowledge, or attitude (Kirkpatrick, 2006). For the stakeholders the
level two evaluation of measuring learning would demonstrate knowledge gained during the
assessment development program.
A pre-post survey would be recommended, as outlined in the implementation plan, as the
level two measurement tool (Kirkpatrick, 2006). Prior to engagement in any part of the
assessment development program, a pre-survey should be administered. The purpose of this pre-
survey is to establish the baseline knowledge level that can be compared to the knowledge
gained through the assessment development program. Questions should probe stakeholders’
prior knowledge about the particular topic that will be discussed or presented. The questions
should be designed to provide stakeholders opportunities to demonstrate knowledge, such as
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 171

asking short-answer questions relevant to the topics covered. The questions might also be short
scenarios that ask stakeholders to develop a component of an assessment model or identify
problem a link between an SLO, institutional learning outcomes, and the AAC&U framework.
The post-meeting survey should mirror the questions asked in the pre-meeting survey
(Kirkpatrick, 2006). This will allow for direct comparison of the impact of the assessment
development program on stakeholders’ knowledge about assessment. If scenario questions are
asked in the pre-survey, similar scenarios should be included in the post-survey so that learning
can be measured. The post-meeting survey should also include an open question asking how the
training could be improved. If stakeholders appear to not have learned during the activity, then
the responses of the open question about how the activity could be improved should be reviewed
and adjustments made per the suggestions for improvement.
Level 3: Behavior
The expected outcome at the behavior level is that the stakeholders will develop a shared
understanding of “excellence” and incorporate strategies to create a holistic multilayered
assessment model that ensures both SLOs, as well as institutional needs, are met. Assuming that
the proposed solutions are effective, the level three evaluations would indicate that the
stakeholders are actively engaging with the institution and with one another about how to
continually assure, as well as improve, assessment of SLOs. In addition, there should also be
seen proactive seeking of resources to improve quality in order to achieve “excellence” in
assessment.
Level three evaluation measurements will need to occur over time. Behavior does not
always immediately change, so evaluations of behavior should be done in intervals over a period
of time (Kirkpatrick, 2006). A suggested time frame would be an evaluation every three months
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 172

for a period of two years, which is embedded into the ongoing training program offerings. As
assessment will look at end of term data, completing two academic cycles will be necessary to
see if changes are occurring longitudinally. The same questions asked in the level two
evaluation can be used to assess whether or not the program had lasting impact (Clark & Estes,
2008). Additional questions should include, for example, “What new strategies that you have
learned in these activities have you incorporated in your assessment model?” These additional
questions should elicit a more detailed picture of how proactive or not the stakeholders have
been in taking steps to improve their assessment work. Additionally, with the knowledge capture
component of the solution plan, syllabi and course materials should be reviewed every term for
at least two years to reinforce the necessity of capturing and reviewing this data and to see if
changes are made to SLOs across the curriculum and within specific courses. Sustaining the
evaluation over a two-year period will allow for comparisons over time to see if the behavior
lasts across academic years.
Level 4: Impact
At the fourth, and final, level of evaluation, the expectation is that stakeholders of North
College who have engaged in the assessment development program will have increased their
skills and knowledge regarding developing more comprehensive assessment and that the
institution as a whole is able to capture and track assessment data in a holistic systemic
multilayered manner. If the proposed solution is effective, this will be seen in improved
assessment results in global education, and institutional data on assessment, and the need met.
The measurement tool for level four evaluation is the assessment data collected by the
institution. As assessment occurs at the end of each term, and North College is now on the Open
Pathways assessment model for HLC, a baseline of information will begin to exist for
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 173

comparison. While in some areas it is possible that baseline data exists, it would likely not be
holistic or systemic in nature given the data findings. The institution should, both at a course
level as well as per AAC&U not connected to any particular course of study, be able to see if
there is improvement in SLOs. If the data improved, then the solution can be deemed effective.
However, if improvements in SLOs are not achieved or data is not collected as expected, then the
need and assumed causes should be revisited and revised.
Strengths and Weaknesses of the Approach
The needs analysis framework provides a systemic method to diagnose and solve a
performance problem (Clark & Estes, 2008). The process requires a clearly articulated goal, and
provides an opportunity to uncover root causes of a problem by examining the knowledge,
motivation, and organizational factors that may be barriers to achieving the goals. The needs
analysis process is a holistic approach to problem-solving that encourages constant process
improvement by investigating root causes, identifying appropriate solutions, creating an
implementation plan, and evaluating the solutions to assess whether or not the goal is reached.
This systemic framework encourages an evidence-based approach, and does not allow for
reactive solutions to a performance problem. It encourages a more thoughtful, data-driven
process that will yield solutions that address root needs.
However, the need analysis process necessitates a clear organizational goal as a starting
point for problem identification and generation of assumed needs. At the time of writing, two
major initiates were under way and linked to the solutions that could confuse institutional goals
or help provide clarity. First, was the institutional review of the general education requirements.
Second was the consideration of transition of the institution from a college to a university.
While both of these initiatives could provide organizational stimulus, and capacity, for the
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 174

solutions they could likewise alter the timeline as well as the way in which the proposed
solutions are viewed by the stakeholders. The institutional shift to the Open Pathways HLC
accreditation process, however, is a constant goal that overrides both of these other factors. This
new accreditation process, and requirements, highlights the need for the organizational
performance goal, and yet, this innovation would need to be achieved while the institution
continues to operate and collect assessment data within existing systems and frameworks to
ensure accreditation.
Limitations
The full description of limitations was discussed in Chapter Three. The primary
delimitation of the project was that it was context specific to North College and addressed this
specific organization’s mission and innovation goal; neither of which can be generalized.
However, other institutions may benefit from the application of this project’s use of Clark and
Estes (2008) needs analysis process to bring about innovation application in a similar manner.
Future Research
The literature on assessment and global education highlights the continually shifting
nature of both fields, as well as their intersection. When coupled with the limitations of this
study, six areas of possible research are seen.
First, would be the expansion of the stakeholder group to all stakeholders involved in the
assessment process. In this study, this group was limited to a select set of stakeholders based
upon the cultural framework linked to the oversight of assessment at one institution. As such, the
data collected represents only one stakeholder involved in the assessment process at North
College and does not reflect the interaction with other stakeholder groups. Future research might
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 175

expand the scope of this study to explore additional stakeholder groups, leading to more
comprehensive ways to address the organization’s needs.
Second would be to replicate and, per the above, expand research at organizations that are
not utilizing the AAC&U framework for either SLOs or assessment. While the framework, and
definitions it provides, was instrumental in the data findings and the ability to correlate across
knowledge, motivation and organizational areas, data could well be divergent for organizations
that utilize a different framework for assessment. Such research could be of benefit to the field
of assessment as well as global education.
Third, would be to would be to replicate, and per the above, expand research at
organizations which are of a different category and type than North College. As a liberal arts
institution, though it has graduate degrees, the data again were specific to the organizational
model and context of North College and the results focused primarily on undergraduate SLOs.
For example, research was not included as a factor in the model, whereas at another institutional
type it could be central to the mission and findings. Likewise, institutional size could well alter
the findings, and possibly correlated to institutional type. Many research institutions are larger
organizations and the factors of institutional size as well as mission could alter the findings.
Fourth, would be the conducting of longitudinal research at a specific institution given
the shifting nature of the fields of assessment and global education. AAC&U is conducting
ongoing evaluation of the assessment framework, and has made changes since it was released.
However, as AAC&U is a tool which is implemented within a particular organizational culture
and model, how root needs shift over time and need to be addressed are likely central to
assessment of student learning as well as global education, and could provide insight for the
institution as well as the realms of research in both student learning and global education.
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 176

Fifth, would be conducting a comparative analysis across United States as well as foreign
institutions in the area of assessment of global education. Such research would depend on
identified institutions, at some level, assessing SLOs in global education. Whether similar rubrics
were utilized or not, findings could be triangulated. The data from this research could provide
insight into where there is divergence, as well as alignment, not only in the definitions of global
education as a learning outcome but also into how different pedagogies, institutional structures,
cultures, and styles of teaching intersect in assessment of SLOs. Given broad alignment of the
internationalization of higher education and the public discourse around required skill sets
students need to enter a global knowledge economy, this data could prove useful to multiple
areas of educational research and policy in an increasingly interconnected world.
Finally, longitudinal research should track students to determine whether attending an
institution where global education was a stated learning outcome correlated to their employment
over time. Given the public discourse around return on investment of the cost of education, and
research tracking students post-graduation for other reasons by career and alumni offices, such
research could explore the correlation between global education, or other SLOs, to jobs attainted
and salaries earned. The resulting data may correlate SLOs to return on investment in new ways
which could, in turn, provide additional specifics to the discourse around return on investment
related to tuition and other costs of education.
Conclusion
The purpose of this study was to conduct a needs analysis in the areas of knowledge and
skill, motivation, and organizational resources necessary to reach the organizational performance
innovation of developing a systemic multilayered model of assessment for global education at
North College. Such an innovation, while of importance to the institution, was viewed as
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 177

significant to both the areas of assessment and global education as well as to higher education
broadly given the increased focus on global education across the field.
The literature outlined broad trends for the continually shifting field of global education.
The field is accelerating in how it affects higher education, in particular in the breadth of
activities that embody internationalization. Higher education continues to alter and evolve
practices to integrate global components into academic and co-curricular programs, research, and
operations in all three areas in the realms of direct services, management, and strategy and
policy. There is great variance, based upon institutional type and other factors, in how
globalization affects and is implemented at an institution.
SLOs define students’ institutionally expected knowledge, skills, and attitudes developed
as a result of a learning experience. Institutions must determine whether they offer sufficient
opportunities to produce the intended learning, select assessment measures (e.g., rubrics), and
then assess if evidence of SLOs data matched the institutional expectations. Results should, if
the SLOs are not being met, result in review and improvement of existing programs as well as
possible creation of new programs and opportunities as needed.
Literature highlighted a lack of a common definition of what criteria‐ based broad-based
assessment means or what it implies for practice. The significance of formative assessment in
measuring SLOs is generally acknowledged, but it remains in its infancy, not always clearly
defined across higher education. In recent years, there has been increasing emphasis on
assessment results as well as what use is made of the results. As such, there is increasing interest
in alternative forms of assessments, particularly complex assessments. Current literature refers
to a need for models of learning assessment that measure global education that cross within the
classical domains of curricular and co-curricular programs and with other types of programming.
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 178

Student learning assessment goes beyond the definition of SLOs. For global education,
experience, assessment requires gathering evidence that demonstrates student achievement of
specific competencies as a result of educational experiences abroad as well as on the campus and
similar measures in the co-curricular realms regardless of location. As was highlighted in this
study, global education is knowledge, not skills. As such, this framework requires
comprehensive multi-dimensional institutional program analysis of the measurement process to
assure global education SLO goals are supported by actual evidence of successful achievement.
Higher education institutions have been challenged not only by a shift toward increased
assessment, but also by the need to systemically measure increasingly complex learning
outcomes. As Robert Barr and John Tagg (1995) argued, the mission, vision, culture, and
structure of higher education institutions must undergo a paradigm shift from providing
instruction to producing learning and knowledge in students. Their framework, like the AAC&U
model, was broad, encompassed all graduates, and called for learning assessment to be done
without regard to a particular curriculum or educational experience. As the frameworks also
focused on learning and knowledge, not teaching, accreditation bodies along with other
stakeholders increasingly require institutions to reliably assess learning outcomes across all
graduates and the institution as a whole. As infusion of global education occurs throughout an
entire institution, assessment must be multilayered, both in location and time. Systemic
accountability of SLOs relating to global education, or assessment, increasingly proved
problematic due to the multi-dimensional nature of internationalization. It is, therefore,
imperative that assessments occur at the various levels, in the interaction between levels, and
over time (NAFSA, 2010).
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 179

Reflecting the international need, to continually explore the intersection of SLOs and
global education, three reports were released in 2016. First, in January, the World Economic
Forum put forth a model for the fourth industrial revolution, and linked this to a set of new job
skills for 2020 (World Economic Forum, 2016). Within these skills, and linked to the new
global industrial revolution, are many aspects of global education including both skills as well as
knowledge. Second, in February, the Organisation for Economic Co-operation and Development
(OECD) called for international standards of SLOs assessment (Andreas-Schleicher, 2016).
Third, in May, the OECD proposed the Programme for International Student Assessment
including measures of global learning and intercultural competence in 2018. These
conversations are the latest in a global public discourse on higher education SLOs, on attempts to
quantify and define these across a continually expanding global field, and on the importance of
global education to both. The discourse, likely, will not diminish but, rather, continue to increase
in complexity.

MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 180

References
ACPA: College Student Educators International. (2008, March). Advancing a vision that leads to
action. March, 2008, draft document. Retrieved March 2, 2015 from:
http://www.myacpa.org/sites/default/files/vision_to_action.pdf
Aghion, P., Dewatripont, M., Hoxby, C., Mas-Colell, A., & Sapir, A. (2009). The governance
and performance of research universities: Evidence from Europe and the U.S. NBER
Working Paper Series 14851. Cambridge: National Bureau of Economic Research.
Altbach, P., & Knight, J. (2007). The internationalization of higher education: Motivations and
Realities. Journal of Studies in International Education, 11(3-4), 290 – 306.
Alexander, F. (2000). The changing face of accountability: Monitoring and assessing
institutional performance in higher education. The Journal of Higher Education, 71(4),
411-431.
Allwood, M., Hemlin, S., Martin B, & Muckenhuber, J. (2007). Creative knowledge
environments. The influences on creativity in research and innovation. Cheltenham, UK,
Northampton; MA, USA: Edward Elgar publishing.
Ambrose, S. A., Bridges, M. W., DiPietro, M., Lovett, M. C., & Norman, M. K. (2010). How
learning works. San Francisco: Jossey-Bass.
American Council on Education. (2011). Report of the blue ribbon panel on global engagement
strength through global leadership and engagement: U.S. higher education in the 21st
Century. Campus internationalization. Retrieved from http://www.acenet.edu/news-
room/Documents/2011-CIGE-BRPReport.pdf
American Council on Education. (2014). Campus internationalization. Retrieved from:
http://www.acenet.edu/higher-education/topics/Pages/Campus-Internationalization.aspx
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 181

Association of American Colleges and Universities. (2014). Essential learning outcomes.
Retrieved from https://www.aacu.org/leap/essential-learning-outcomes. (2009).
Association of American Colleges and Universities. (2015a). VALUE rubric development
project. Retrieved from https://www.aacu.org/value/rubrics
Association of American Colleges and Universities. (2015b). 2015 Global Learning in College:
Defining, Developing, and Assessing Institutional Roadmaps. Retrieved from
http://www.aacu.org/meetings/global/15
American College Personnel Association. (2015c). Student Affairs Assessment Institute.
Retrieved from: http://www.myacpa.org/events/2015-student-affairs-assessment-institute
Anderson, L.W. & Krathwohl, D.R. (Eds.). (2001). A taxonomy for learning, teaching, and
assessing: A revision of Bloom's Taxonomy of Educational Objectives. New York:
Longman.
Armstrong, L. (2007). Competing in the global higher education marketplace: Outsourcing,
twinning, and franchising. New Directions for Higher Education, 140, 131-138.
Astin, A. W., & Antonio, A. L. (2012). Assessment for excellence: The philosophy and practice
of assessment and evaluation in higher education. Lanham, MD: Rowman & Littlefield
Publishers.
Austin, A. E., & Sorcinelli, M. D. (2013). The future of faculty development: Where are we
going? New Directions for Teaching and Learning, 2013(133), 85-97.
Barr, R., & Tagg, J. (1995). From teaching to learning: A new paradigm for undergraduate
education. Change, November/December, 13–25.
Beerkens, E. (2003). Globalisation and higher education research. Journal of Studies in
International Education, 7(2), 128-148.
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 182

Bennett, J. (2008). Transformative training: Designing programs for culture learning.
Contemporary leadership and intercultural competence: Understanding and utilizing
cultural diversity to build successful organizations, 95-110. Thousand Oaks, CA: Sage.
Blimling, G. (2013). Challenges of assessment in student affairs. New Directions for Student
Services, 2013(142), 5-14.
Bolman, L.G. & Deal, T.E. (2007). Reframing Organizations. San Francisco: Jossey-Bass.
Brownell, J., & Swaner, L. (2012). Five High-Impact Practices: Research on Learning
Outcomes, Completion, and Quality. Peer Review, 14(3), 29.
Braskamp, L. A., Braskamp, D. C., & Merrill, K. C. (2009). Assessing progress in global
learning and development of students with education abroad experiences. Frontiers, 18,
101-118.
Braskamp, L. A., Trautvetter, L. C., & Ward, K. (2006). Putting students first: How colleges
develop students purposefully. Bolton, MA: Anker Publishing Company, Inc.
Burnett, S., & Huisman, J. (2010). Universities’ responses to globalisation: the influence of
organisational culture. Journal of Studies in International Education, 14(2), 117-142.
Carr, S. (1999). Distance-Learning group blends offerings of 2- and 4-Year colleges. The
Chronicle of Higher Education. Retrieved from: www.chonicle.com
Childress, L. (2009). Internationalization plans for higher education institutions. Journal of
Studies in International Education, 13(3), 289-309.
Clark, R. E. (1999). The CANE model of motivation to learn and work: A two-stage process of
goal commitment and effort. In J. Lowyck (Ed.), Trends in Corporate Training. Leuven,
Belgium: University of Leuven Press.
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 183

Clark, R. E. (2004). See the forest, tend the trees: Analyzing and solving accountability
problems. UrbanEd: The Magazine of the USC Rossier School of Education, 20-22.
Clark, R. (2005). Five research-tested team motivation strategies. Performance improvement,
44(1), 13-16.
Clark, R., & Estes, F. (2008). Turning research into results: A guide to selecting the right
performance solutions. Atlanta GA: CEP Press.
Clark, R. E., Howard, K. and Early, (2006). Motivational challenges experienced in highly
complex learning environments. In J. Elen and R. E. Clark (Eds.). Handling complexity
in learning environments: Research and Theory (pp. 27-43). Oxford, G.B.: Elsevier
Science Ltd.
Coughlan, P., Coghlan, D., Dromgoole, T., Duff, D., Caffrey, R., Lynch, K., Rose, I., Stack, P.,
McGill, A., & Sheridan, P. (2002). Effecting operational improvement through inter‐ organisational action learning. Integrated Manufacturing Systems, 13(3), 131 – 140.
Coughlin, E. K. (1994). The emergence of the 'global city'. The Chronicle of Higher Education.
Retrieved from: www.chonicle.com
Deardorff, D. (2006). Identification and assessment of intercultural competence as a student
outcome of internationalization. Journal of Studies in International Education, 10(3),
241-266.
Dembo, M. H., & Seli, H. (2008). Motivation and Learning Strategies for College Success: A
Self-Management Approach (3rd ed.). New York: Lawrence Erlbaum.
DeJong, A. J. (1992). Making sense of church-related higher education. In D. S. Guthrie & R. L.
Noftzger (Eds.), Agendas for church-related colleges and universities (pp. 19-27). San
Francisco, CA: Jossey-Bass.
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 184

Desruisseaux, P. (1991). Congress approves program to support overseas study. The Chronicle
of Higher Education. Retrieved from: www.chonicle.com
Desruisseaux, P. (1998). What parents should know about study-abroad programs. The
Chronicle of Higher Education. Retrieved from: www.chonicle.com
Desruisseaux, P. (1999a). 15% rise in American students abroad shows popularity of non-
European destinations. The Chronicle of Higher Education. Retrieved from:
www.chonicle.com
Desruisseaux, P. (1999b, December 10, 1999). Foreign Students Continue to Flock to the U.S.
The Chronicle of Higher Education. Retrieved from: www.chonicle.com
Desruisseaux, P. (2000,). As exchanges lose a political rationale, their role is debated. The
Chronicle of Higher Education. Retrieved from: www.chonicle.com
De Wit, H. (2010). Internationalization of higher education in Europe and its assessment, trends
and issues (NVAO). Retrieved from
http://nvao.net/page/downloads/Internationalisation_of_Higher_Education_in_Europe_D
EF_december_2010.pdf
Dill, D. D. (1999). Academic accountability and university adaptation: The architecture of an
academic learning organization. Higher education, 38(2), 127-154.
Eccles, J. C. (2010). Expectancy value motivational theory. Education.com Retrieved from:
http://www.education.com/reference/article/expectancy-value-motivational-theory/
Edwards, J. (2007). Challenges and opportunities for the internationalization of higher education
in the coming decade: Planned and opportunistic initiatives in American Institutions.
Journal of Studies in International Education, 11(3/4), 373-381.
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 185

El-Khawas, E. (2008, September 8). The Organisation for economic co-operation and
development, programme on institutional management in higher education. Outcomes of
higher education: Quality relevance and impact. Retrieved from:
http://www.oecd.org/site/eduimhe08/41216386.pdf
Elmore, R. F. (2002). Bridging the gap between standards and achievement: The imperative for
professional development in education. Washington, DC: Albert Shanker Institute.
Farnsworth, K. (2005). A new model for recruiting international students: The 2+2. International
Education, 35(1), 5-14.
Filback, R. & Green, A. (2013). A framework of educator mindsets and consequences. Los
Angeles, CA: University of Southern California. Retrieved from
http://rossier.usc.edu/files/2013/08/Educator-Mindsets-and-Consequences-Table-Filback-
Green-2013.pdf
Fink, L. D. (2013). Creating significant learning experiences: An integrated approach to
designing college courses. John Wiley & Sons.
Fischer, C., & King, R. (1995). Authentic assessment: A guide to implementation. Thousand
Oaks, Calif: Corwin Press.
Friedlander, J., & Serban, A. (2004). Meeting the challenges of assessing student learning
outcomes. New Directions for Community Colleges, 2004(126), 101-109.
Gacel-Ávila, J. (2005). The Internationalisation of higher education: a paradigm for global
citizenry. Journal of Studies in International Education, 9(2), 121-136.
Gallimore, R., & Goldenberg, C. (2001). Analyzing cultural models and settings to connect
minority achievement and school improvement research. Educational Psychologist,
31(1), 45-56.
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 186

Georgia Learning Outcomes of Students Studying Abroad Research Initiative, (2010.) Retrieved
from http://glossari.uga.edu
Goodman, A. E. (1999). America is devaluing international exchanges for students and scholars.
The Chronicle of Higher Education. Retrieved from: www.chonicle.com
Green, M. F. (2002). Internationalizing undergraduate education: Challenges and lessons of
success. In Promising Practices (pp. 9-21). Washington, D. C.: American Council on
Education.
Green, M. F. (2005). Internationalization in U. S. higher education. Washington, D. C.:
American Council on Education.
Green, M. F. (2005). Measuring internationalization at comprehensive universities. Washington,
D. C.: American Council on Education.
Green, M. (2013). Improving and assessing global education. Washington, DC: NAFSA:
Association of International Educators.
Green, M. (2012). Measuring and accessing Internationalization. Washington: NAFSA:
Association of International Educators.
Green, M. F., & Olsen, C. L. (2003). Internationalizing the campus: A user’s guide. Washington,
DC: American Council on Education.
Green, M. F., & Shoenberg, R. (2006). Where faculty live: Internationalizing the disciplines.
Washington, DC: American Council on Education.
Green, M. F., &, L. (2005). Measuring internationalization at liberal arts colleges. Washington,
DC: American Council on Education.
Green, M. F., Luu, D., & Burris, B. (2008). Mapping internationalization on U.S. campuses.
Washington, DC: American Council on Education.
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 187

Hart, D. (1994). Authentic assessment: A handbook for educators. Menlo Park, Calif: Addison-
Wesley.
Hill, Y., Lomas, L., & MacGregor, J. (2003). Students' perceptions of quality in higher
education. Quality Assurance in Education, 11(1), 15-20.
Hoekstra, A., & Crocker, J. R. (2015). Design, implementation, and evaluation of an ePortfolio
approach to support faculty development in vocational education. Studies in Educational
Evaluation, 46, 61-73.
Hser, M.P. (2005) Campus internationalization: A study of American universities’
internationalization efforts. International Education, 35(1), 35-48.
Huang, F. (2007). Internationalization of higher education in the developing and emerging
countries: A focus on transnational higher education in Asia. Journal of Studies in
International Education, 11(3-4), 421-432.
Hudzik, J. (2011). Comprehensive internationalization: From concept to action. NAFSA:
Association of International Educators. Retrieved from: http://www.nafsa.org
Institute of International Education. (2014). Open Doors 2014. Institute of International
Education, November 17, 2014.
Janesick, V. (2006). Authentic assessment primer. New York: Peter Lang.
Jones, E., & Killick, D. (2013). Graduate attributes and the internationalized curriculum:
Embedding a global outlook in disciplinary learning outcomes. Journal of Studies in
International Education, 17(2), 165-182.
Kehm, B., & Teichler, U. (2007). Research on internationalisation in higher education. Journal
of Studies in International Education, 11(3-4), 260-273.
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 188

Kirkpatrick, D. L. (2006). Seven keys to unlock the four levels of evaluation. Performance
Improvement, 45(7), 5-8.
Kirkpatrick, D.L., & Kirkpatrick, J.D. (2007). Implementing the four levels. San Francisco, CA:
Berret-Koehler Publishers, Inc.
Knight, J. (1993). Internationalization management strategies and issues. International Education
Magazine, 9, 6, 21-22.
Knight, J. (1997). Internationalization of higher education: A conceptual framework. In J.
Knight, J. (2003). Updating the definition of internationalisation. International Higher
Education, 33, 2.
Knight, J. (2001). Internationalization. Journal of Studies in International Education, 5(3), 228-
243.
Knight, J. (2004). Internationalization remodeled: definition, approaches, and rationales. Journal
of Studies in International Education, 8(1), 5-31.
Knowles, M. S. (1980). The modern practice of adult education. New York: Cambridge The
Adult Education Company.
Krathwohl, D. (2002). A revision of Bloom’s taxonomy: An overview. Theory Into Practice,
41(4), 212-218.
Krause, K. L. (2012). Addressing the wicked problem of quality in higher education: Theoretical
approaches and implications. Higher Education Research & Development, 31(3), 285-
297.
Kuh, G. D. (1993). Cultural perspectives in student affairs work. Washington, D.C.: American
College Personnel Association.
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 189

Kuh, G. (2012). High-impact educational practices: What they are, who has access to them, and
why they matter. Peer Review, 14(3), 29.
Kuh, G., & O'Donnell, K. (2013). Ensuring quality and taking high-impact practices to scale.
Peer Review, 15(2), 32.
Kuh, G. D., Schuh, J. H., Whitt, E. J., & & Associates. (1991). Involving colleges: Successful
approaches to fostering student learning and development outside the classroom. San
Francisco, CA: Jossey-Bass.
Lead365. (2015). National Conference. Retrieved from http://www.lead365.org/national-
conference/2015
Lindsay, N., Stroud, D., & Tubbs, A. (2013). Review of “successful assessment for student
affairs.” Research & Practice in Assessment, 8. 83-85, 8.
Little, B., & Locke, W. (2011). Conceptions of excellence in teaching and learning and
implications for future policy and practice. In Questioning excellence in higher education
(pp. 119-137). SensePublishers.
Leask, B. (2001). Bridging the gap: internationalizing university curricula. Journal of Studies in
International Education, 5(2), 100-115.
Leask, B. (2008). A holistic approach to internationalisation: Connecting institutional policy
and the curriculum with the everyday reality of student life (Doctoral dissertation,
Bournemouth University). Retrieved from: http://jsi.sagepub.com.libproxy.usc.edu
Lewis, T., & Niesenbaum, R. (2005). Extending the stay: Using community-based research and
service learning to enhance short-term study abroad. Journal of Studies in International
Education, 9(3), 251-264.
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 190

Linn, R., Baker, E., & Dunbar, S. (1991). Complex, performance-based assessment:
Expectations and validation criteria. Educational Researcher, 20(8), 15-21.
Lipsman, M. (2000). Globalization isn't good or inevitable. The Chronicle of Higher Education.
Retrieved from: www.chonicle.com
Lyons, R. E. (2007). Deepening our understanding of adjunct faculty. In R. Lyons (Ed.), Best
practices for supporting adjunct faculty, Bolton, MA: Anker Publishing (pp. 1-13).
Mayer, R. E. (2011). Applying the science of learning. Boston, MA: Pearson Education.
Matthews, J. (2002). International education and internationalisation are not the same as
globalisation: emerging issues for secondary schools. Journal of Studies in International
Education, 6(4), 369-390.
Mazzarol, T., & Soutar, G.N. (2002). “Push-pull” factors influencing international student
destination choice. The International Journal of Education Management, 16(2/3), 82-90.
McCabe, L. (2001). Globalization and internationalization: the impact on education abroad
programs. Journal of Studies in International Education, 5(2), 138-145.
Melanie, A. (2013). Strategic planning: an examination of the role of disciplines in sustaining
internationalization of the university. Journal of Studies in International Education,
17(2), 183-202.
Merriam, S. B. (2009). Qualitative research: A guide to design and implementation. San
Francisco CA: Jossey-Bass.
Merriam, S. B., & Bierema, L. L. (2013). Adult learning: Linking theory and practice. John
Wiley & Sons.
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 191

Mestenhauser, J. (1998). Portraits of an international curriculum: an uncommon
multidimensional perspective. Reforming the higher education curriculum:
Internationalizing the campus, 3-39.
Miller, E. K., & Cohen, J. D. (2001). An integrative theory of prefrontal cortex function. Annual
Review of Neuroscience, 24, 167-202.
NAFSA: Association of International Educators. (2015). International Education Professional
Competencies. Retrieved from http://www.nafsa.org/Professional_Resources/
Career_Center/IE_Competencies/International_Education_Professional_Competencies/
NAFSA: Association of International Educators. (2010). Members of the Teaching, Learning,
and Scholarship Knowledge Community 2009 Task Force on Assessment and
Evaluation. Retrieved from http://www.nafsa.org/uploadedFiles/NAFSA_Home/
Resource_Library_Assets/Publications_Library/Assess%20and%20Eval%20in%20IE.pdf
NAFSA: Association of International Educators. (2010). Assessment and Evaluation for
International Educators. Retrieved from http://www.nafsa.org/uploadedFiles/NAFSA_
Home/Resource_Library_Assets/Networks/RS/AssessandEvalinIE.pdf
NAFSA: Association of International Educators. (2014). Curricular Designs for Global Learning.
(2014). Intercultural Competence and Student Development: Working Toward a Fuller
Picture of Global Learning. Retrieved from http://www.nafsa.org/Attend_Events/
Online/Webinars/Faculty_Conversations/Faculty_Conversations/
NAFSA: Association of International Educators. (2015). Framing the Global: Entry Points for
Learning, Research, and Campus Change. Retrieved from https://www.nafsa.org/
Professional_Resources/Learning_and_Training/Global_Learning_Conversations/Framin
g_the_Global__Entry_Points_for_Learning, _Research,_and_Campus_Change/
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 192

NASPA: Student Affairs Administrators in Higher Education. (2015a). 2015 NASPA annual
conference. Retrieved from: http://conference2015.naspa.org/register
NASPA: Student Affairs Administrators in Higher Education. (2015b). NASPA International
Symposium. Retrieved from https://www.naspa.org/events/2015IS
Nusche, D. (2007). Assessment of Higher Education Outcomes (Working Paper No. 15).
Retrieved from http://www.oecd.org/document/51/0,3343,en_2649_35961291_
40119475_1_1_1_1,00.html
Nygaard, C., Holtham, C., & Courtney, N. (2009). Improving students' learning outcomes.
Portland, OR: Copenhagen Business School Press.
Olsen, C. L., Green, M. F., & Hill, B. A. (2005). Building a strategic framework for
comprehensive internationalization. Washington, DC: American Council on Education.
Olson, C. L., Evans, R., & Shoenberg, R. F. (2007). At home in the world: Bridging the gap
between internationalization and multicultural education. Washington, DC: American
Council on Education.
Olson, C. L., Green, M. F., & Hill, B. A. (2005). Building a strategic framework for
comprehensive internationalization. Washington, DC: American Council on Education.
Olson, C. L., Green, M. F., & Hill, B. A. (2006). A handbook for advancing comprehensive
internationalization: What institutions can do and what students should learn.
Washington, DC: American Council on Education.
Organisation for Economic Co-operation and Development (2016). OECD proposes new
approach to assess young people’s understanding of global issues and attitudes toward
cultural diversity and tolerance. Retrieved from: https://www.oecd.org/pisa/oecd-
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 193

proposes-new-approach-to-assess-young-peoples-understanding-of-global-issues-and-
attitudes-toward-cultural-diversity-and-tolerance.htm
Palomba, C., & Banta, T. (1999). Assessment essentials: planning, implementing, and improving
assessment in higher education. San Francisco: Jossey-Bass.
Pascarella, E. T., & Terenzini, P. T. (2005). How college affects students: A third decade of
research, vol. 2. San Francisco, CA: Jossey-Bass. Pajares, F. (2010). Self-efficacy theory.
Education.com Retrieved, from: http://www.education.com/reference/article/self-
efficacy-theory/
Pintrich, P. R. (2003). A motivational science perspective on the role of student motivation in
learning and teaching contexts. Journal of Educational Psychology, 95(4), 667-686.
Qiang, Z. (2003). Internationalization of higher education: Towards a conceptual framework.
Policy Futures in Education, 1(2), 248-265.
Raby, R.L. (2009). Globalization and community college model development. In R.L. Raby and
E.J. Valeau (Eds.). Community college models: Globalization and high education reform
(pp. 21-38). Amsterdam: Springer Press. Rhodes, T., & Finley, A. (2014).
Rudzki, R. (2000). Implementing internationalisation: the practical application of the fractal
process model. Journal of Studies in International Education, 4(2), 77-90.
Rueda, R. (2011). The 3 dimensions of improving student performance. New York: Teachers
College Press.
Rueda, R. (2015). Unpublished course materials. Los Angeles, CA: University of Southern
California.
Sadler, D. R. (2005). Interpretations of criteria-based assessment and grading in higher
education. Assessment & Evaluation in Higher Education, 30(2), 175-194.
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 194

Sample, S. (2013). Developing Intercultural Learners Through the International Curriculum.
Journal of Studies in International Education, 17(5), 554-572.
Saroyan, A., & Trigwell, K. (2015). Higher education teachers’ professional learning: Process
and outcome. Studies in Educational Evaluation, 46(1), 92-101.
Sassen, S. (2001). The global city: New York, London, Tokyo. Princeton, NJ: Princeton
University Press.
Schein, E. H. (2004). Organizational culture and leadership. San Francisco, CA: Jossey-Bass.
Schein (2004). The concept of organizational culture: Why bother? Classics of organization
theory.
Schleicher, A. (2016, February). The Organisation for Economic Co-operation and Development
(OECD). Value-Added: How do you measure whether universities are delivering for their
students? HEPI 2015 Annual Lecture. Retrieved from: http://www.hepi.ac.uk/wp-
content/uploads/2016/01/Andreas-Schleicher-lecture.pdf
Schoorinan, D. (1999). The pedagogical implications of diverse conceptualizations of
internationalization: A U.S. based case study. Journal of Studies in International
Education, 3(2), 19-46.
Schraw, G., & McCrudden, M. (2013) Information processing theory. Retrieved from:
http://www.education.com/reference/article/information-processing-theory/
Schuh, J., & Upcraft M., (2001). Assessment practice in student affairs: An applications manual.
San Francisco: Jossey-Bass Publishers.
Schunk, D. H., Pintrich, P. R., & Meece, J. (2010). Motivation. Retrieved from
http://www.education.com/reference/article/motivation/
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 195

Senge, P., Smith, P., Kruschwitz, N., Laur, J., & Schley, S. (2008). The necessary revolution:
How individuals and organizations are working together to create a sustainable world.
Cambridge, MA: Doubleday Currency.
Shupe, D., & Shupe, D. (2007). Significantly better: the benefits for an academic institution
focused on student learning outcomes. On the Horizon, 15(2), 48-57.
Soria, K., & Troisi, J. (2013). Internationalization at home alternatives to study abroad:
Implications for students’ development of global, international, and intercultural
competencies. Journal of Studies in International Education, 18(3), 261-280.
Stone, N. (2006). Internationalising the student learning experience: possible indicators. Journal
of Studies in International Education, 10(4), 409-413.
Soupir-Fremstad, D. (2013) The expectancy theory: Teachers' perspectives of motivation and
compensation (Unpublished doctoral dissertation). The University of North Dakota,
Grand Forks, ND.
Spady, W. (1998). Paradigm lost: Reclaiming America’s educational future. Arlington, VA:
American Association of School Administrators.
Stromquist, N. P. (2002). Education in a globalized world: the connectivity of economic power,
technology and knowledge. Lanham, MD: Rowman & Littlefield.
Suskie, L. (2004). Assessing student learning: A common sense guide. Boston: MA: Anker
Publishing Company.
Tarr, T. A. (2010). Working with adjunct faculty members. In K. Gillespie & D. Reimondo
Robertson (Eds.). A Guide to Faculty Development (pp. 347-362). San Francisco: Jossey-
Bass.
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 196

Tarrow, S., and Hall, P. A. (1998). Globalization and area studies: When is too broad too
narrow? The Chronicle of Higher Education. Retrieved from: http://chronicle.com/
article/GlobalizationArea/99332/
The Forum on Education Abroad. (2009). Standards of good practice. Standards of good
practice for short-term education abroad programs. Retrieved from http://www.mobility.
unimelb.edu.au/resources/readings/ForumEAStandardsShortTermProg.pdf
The Forum on Education Abroad (2011). Standards of good practice. Retrieved from:
http://www.forumea.org/resources/standards-of-good-practice
The Forum on Education Abroad (2015). Quality Assurance Programs. Retrieved from
https://www.forumea.org/get-involved/quality-assurance-programs
Tierney, W. & Hentschke, G. (2007). Understanding the rise of for-profit colleges and
universities new players, different game. Baltimore: The Johns Hopkins University Press.
Tisdell, E. J. (2003). Exploring spirituality and culture in adult and higher education. San
Francisco: Jossey-Bass.
Tuckman, B. (2010). Operant conditioning. Retrieved from http://www.education.com/
reference/article/operant-conditioning/
Umbach, P.D. & Wawrzynski, M. R. (2005). Faculty do matter: The role of college faculty in
student learning and engagement. Research in Higher Education, 46(2), 153-184.
VALUE rubrics for improvement of learning and authentic assessment. Peer Review, 16(3), 32.
van der Wende, M. (2007). Internationalization of higher education in the OECD countries:
challenges and opportunities for the coming decade. Journal of Studies in International
Education, 11(3-4), 274-289.
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 197

Vukasović, M. (2012). Effects of higher education reforms: Change dynamics. Rotterdam: Sense
Publishers.
Wächter, B. (2003). An introduction: internationalisation at home in context. Journal of Studies
in International Education, 7(1), 5-11.
Whitsed, C. & Green, W. (2014). What’s in a name? A theoretical exploration of the
proliferation of labels for international education across the higher education Sector.
Journal of Studies in International Education, 18(2), 105-119.
World Economic Forum. (2016). Retrieved from http://www.weforum.org/agenda/archive/
fourth-industrial-revolution
Woolf, M. (2002). Harmony and dissonance in international education: the limits of
globalisation. Journal of Studies in International Education, 6(1), 5-15.
Yang, R. (2002). University internationalization: Its meanings, rationales and implications.
Intercultural Education, 13(1), 81-95.
Yeh, Y. M. C. (2011). The implementation of knowledge management system in Taiwan’s
higher education. Journal of College Teaching & Learning, 2(9). 35-42.
Yorke, M. (2003). Formative assessment in higher education: Moves towards theory and the
enhancement of pedagogic practice. Higher Education, 45(4), 477-501.
Zack, M.H (1999). Developing a knowledge strategy. California Management Review, 41(3),
125-145.

MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 198

Appendix A
Survey Instrument Protocol – Sample Group
Dear ___________:

My name is Eric Canny, and I am a doctoral candidate in the Rossier School of Education at
University of Southern California. I am conducting a research study as part of my dissertation,
examining assessment of global education at North College. The research has, as well, been
approved by North College.

You are cordially invited to participate in the study.

If you agree, you are invited to:

- Complete an anonymous online survey of questions that is anticipated to take no more
than 15 minutes to complete.

- Participate in an in-person interview. The interview is voluntary, and anticipated to last
approximately 45 minutes. You do not have to answer any questions you don’t want to; if
you don’t want the interview to be taped, handwritten notes will be taken.

Participation in this study is completely voluntary. Your identity as a participant will
remain confidential at all times during and after the study.

If you would like to participate in the survey, please begin the survey via the link.

If you would like to participate in the in-person interview please email or call me at the contact
information below.

If you have any questions, please feel free to contact me at canny@North.edu or 612-330-1383.

Thank you in advance for your participation.

Eric Canny
Doctoral Candidate - Rossier School of Education
University of Southern California

MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 199

1. How many years have you worked at North College?
Drop down list (logic for 2012 financial question #14)

2. As of September 1, 2015, which of the following committees do you serve on? Please check
all that apply.
Assessment Committee,
Academic Affairs Committee
Graduate Academic Affairs Committee
Student Affairs Committee Co-Curricular Assessment Sub-Committee

3. Please rate how you feel about the following statements.
I have the knowledge to conduct assessment within my area in a comprehensive manner.
Strongly disagree
Disagree
Agree
Strongly agree

I have the knowledge to develop new comprehensive models of assessment in my area if
I wanted or needed to.
Strongly disagree
Disagree
Agree
Strongly agree

I have the knowledge to develop new comprehensive models of assessment across areas
(e.g., multiple academic areas, and/or academic and co-curricular) if I wanted or needed
to.
Strongly disagree
Disagree
Agree
Strongly agree

4. Please rate how you feel about the following statements.
I have the knowledge to conduct assessment of global education within my area in a
comprehensive manner.
Strongly disagree
Disagree
Agree
Strongly agree

I have the knowledge to develop new comprehensive models of assessment of global
education in my area if I wanted or needed to.
Strongly disagree
Disagree
Agree
Strongly agree
MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 200

I have the knowledge to develop new comprehensive models of assessment of global
education across areas if I wanted or needed to.
Strongly disagree
Disagree
Agree
Strongly agree

6. Please rank the following in importance when you think of assessment at North College
broadly.
Student learning outcomes
Unimportant
Important
Extremely important

North College Mission
Unimportant
Important
Extremely important

Accreditation by Higher Learning Commission
Unimportant
Important
Extremely important

Other accrediting bodies (please list)
Unimportant
Important
Extremely important

Faculty Review
Unimportant
Important
Extremely important

Staff Review
Unimportant
Important
Extremely important

Academic Program Review
Unimportant
Important
Extremely important

MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 201

Co-Curricular Program Review
Unimportant
Important
Extremely important

7. Of the AAC&U Rubrics, please select those that you see as intersecting with global education
at North College in the programs you do for students.
Inquiry and analysis
Critical thinking
Creative thinking
Written communication
Oral communication
Reading
Quantitative literacy
Information literacy
Teamwork
Problem solving
Civic engagement—local and global
Intercultural knowledge and competence
Ethical reasoning
Foundations and skills for lifelong learning
Global learning
Integrative learning

8. Please rate how you feel about the following statements.
Conducting assessment within my area is important to me.
Strongly disagree
Disagree
Agree
Strongly agree

I see value in developing new ways of comprehensive assessment in my area.
Strongly disagree
Disagree
Agree
Strongly agree

I feel confident in my ability to develop more comprehensive assessment in my area.
Strongly disagree
Disagree
Agree
Strongly agree

MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 202

I see value in developing more comprehensive assessment across areas of North
College.
Strongly disagree
Disagree
Agree
Strongly agree

9. Please check all reasons below for why assessment is important to your specific work at
North College. Answer these not thinking why it might be important to the college as a whole,
but to your specific work.
Knowing my work has an effect on student learning outcomes.
North College Mission
Accreditation by Higher Learning Commission
Accreditation by other accreditation bodies (please list)
Faculty Review
Staff Review
Academic Program review
Co-curricular Program review

10. Please check all boxes for why assessment is important to North College as an institution.
Knowing my work has an effect on student learning outcomes.
North College Mission
Accreditation by Higher Learning Commission
Accreditation by other accreditation bodies (please list)
Faculty Review
Staff Review
Academic Program review
Co-curricular Program review

11. In terms of importance, how would you rate assessment in measuring global education
student learning outcomes.

Within academic areas
Unimportant
Important
Extremely important

Within co-curricular areas
Unimportant
Important
Extremely important

MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 203

Aligned across academic and co-curricular areas
Unimportant
Important
Extremely important

12. Please rate your desire to learn new kinds of assessment models that include global
education.
Low
Moderate
High

13. Please rate your desire to design new assessment models for North College that include
global education.
Low
Moderate
High

14. Please rate how the financial crisis of 2012 affected assessment for you in any of the
following ways: (logic linked to question #1, only deployed for 2012 and prior)

Decreased importance I placed upon it.
Strongly disagree
Disagree
Agree
Strongly agree

Decreased non-staffing resources dedicated to it.
Strongly disagree
Disagree
Agree
Strongly agree

Decreased staffing resources dedicated to it.
Strongly disagree
Disagree
Agree
Strongly agree

I was unclear if it remained a goal of the college.
Strongly disagree
Disagree
Agree
Strongly agree

MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 204

I had less time to undertake assessment due to other tasks I had to assume specifically
because of the financial crisis.
Strongly disagree
Disagree
Agree
Strongly agree

The financial crisis of 2012 continues to affect assessment for me today.
Strongly disagree
Disagree
Agree
Strongly agree

15. Please rate how you feel about the following statements.

I believe the academic model at North College makes assessment difficult.
Strongly disagree
Disagree
Agree
Strongly agree

I believe we could improve assessment while maintaining the academic model at North
College.
Strongly disagree
Disagree
Agree
Strongly agree

16. Which of the following have oversight of assessment at North College?
Please check all that apply.
Assessment Committee,
Academic Affairs Committee
Graduate Academic Affairs Committee
Student Affairs Committee Co-Curricular Assessment Sub-Committee

MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 205

17. Who has responsibility for assessment at North College?
Faculty
To no extent
To some extent
To great extent
Administration
To no extent
To some extent
To great extent
Staff
To no extent
To some extent
To great extent

18. Please rate how you feel about this statement.

North College is committed to assessment of global education.
Strongly disagree
Disagree
Agree
Strongly agree

19. Please rate how strongly you feel about the following statements.

North College, today, has the staffing resources to implement a multilayered
assessment model that includes global education.
Strongly disagree
Disagree
Agree
Strongly agree

North College, today, has clear goals to implement a multilayered assessment model
that includes global education.
Strongly disagree
Disagree
Agree
Strongly agree

North College, today, has the infrastructural resources to implement a multilayered
assessment model that includes global education.
Strongly disagree
Disagree
Agree
Strongly agree

MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 206

North College, today, has staff with the necessary knowledge to implement a
multilayered assessment model that includes global education.
Strongly disagree
Disagree
Agree
Strongly agree

North College, today, has faculty with the necessary knowledge to implement a
multilayered assessment model that includes global education.
Strongly disagree
Disagree
Agree
Strongly agree

20. Please rate how you feel about the following statements.

I feel supported by North College to conduct assessment within my area.
Strongly disagree
Disagree
Agree
Strongly agree

I feel supported by North College to conduct assessment across areas of the institution.
Strongly disagree
Disagree
Agree
Strongly agree

I feel there are clear goals at North College to conduct assessment within my area.
Strongly disagree
Disagree
Agree
Strongly agree

I feel there are clear goals at North College to conduct assessment across areas.
Strongly disagree
Disagree
Agree
Strongly agree

I feel there are clear assessment goals on the committee(s) I sit on.
Strongly disagree
Disagree
Agree
Strongly agree

MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 207

I feel there are clear assessment goals on the committee(s) I sit on which include Global
Education.
Strongly disagree
Disagree
Agree
Strongly agree

MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 208

Appendix B
Interview Protocol – Sample Group
Respondent (Name): _______________________________________________

Location of Interview: ____________________________________________________

Time in / Time Out: _________________________________________

Introduction

Thank you for meeting with me today. I am conducting this exercise as part of my dissertation
research with my doctoral program at USC, exploring assessment of global education at North
College, and possible new models of assessment of this student learning outcome. I anticipate
taking no more than 45 minutes of your time, and I have 19 questions for your consideration.

Your participation is completely voluntary. We can skip any question you want at any time, and
you may stop the interview at any time. Any identifiable information obtained in connection with
this study will remain confidential. Your responses will be coded with a false name (pseudonym)
and maintained separately from your answers.

If you are comfortable with it, I would like to record our conversation and the recording will be
destroyed after it is transcribed. Are you comfortable if I record the conversation?

Do you have any questions?

Ready to begin?

1. Please define, as you understand them, each of the following:
Assessment
Evaluation
Testing

2. Please define, as you understand it, global education at North College.

3. Please define, as you understand them, student learning outcomes at North College.

4. Please describe what the term multilayered assessment means to you.

5. To what extent are you familiar with North’s recent change from the AQUIP framework to
the Open Pathway framework for the Higher Education Commission?

MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 209

6. Probing Question:
If familiar:
How has this change affected the demands placed on you surrounding assessment, if at
all?

7. How successful do you believe you have been in assessment of SLOs in your area of
responsibility, in particular of global education? Is there any particular knowledge that you
feel would improve your assessment work around global education in your area moving
forward?

8. How successful do you believe North College has been as an institution in assessment of
SLOs in your area of responsibility?

9. What would a comprehensive model of assessment that included global education across
the institution look like to you?

10. The XXX Committee(s) has/have assessment in its/their charge. Was this part of your
motivation to join the Committee(s), and how would you rank it amongst all the charges the
committee(s) has/have? (Note alter for Assessment/Student Affairs)

11. Please describe who you feel has responsibility for assessment at North College and your
role in it.

12. What kinds of communication and interaction do you have with others around assessment
goals, other than those that take place in the formal North Committees charged with
overseeing assessment, if any?

13. Probing questions:
Are there other forms of communication or interaction, either formal or informal, that
would enhance assessment at North College?

If so, what might those look like?

14. Please discuss how you monitor the assessment you do, including adjusting as needed.

15. Do you see, or not, the AAC&U Global Learning and Intercultural Rubrics of value to your
assessment work?

16. Probing question: Can you please explain why you feel that way?

MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 210

17. Do you currently do assessment utilizing the AAC&U Global Learning and Intercultural
Rubrics?

18. Probing question:

Can you please explain why you do/don’t utilize the rubrics?

19. Please describe if you think assessment of global education joined between co-curricular
and academic student learning outcomes could, or not, be important to North College.

20. To what extent do you feel that North College currently commits the resources needed for
comprehensive assessment, and please explain why you feel this way?

21. What changes would you recommend to how North College considers, or conducts,
assessment relating to global education?

MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 211

Appendix C
Document Protocol

Document Analysis Builder Worksheet
Assumed Cause Knowledge Motivation Organization

MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 212

Appendix D
American Association of Colleges and Universities Global Education and Intercultural Rubrics

MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 213

MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 214

MULTILAYERED ASSESSMENT OF GLOBAL AWARENESS 215

From American Association of Colleges and Universities, 2014.

Abstract (if available)

Abstract This study utilized the Clark and Estes (2008) gap analysis framework to conduct a needs analysis that examined the knowledge, motivation, and organizational needs to propel North College toward the innovation goal of the implementation of a systemic multilayered assessment model for global education linking curricular and co-curricular student learning outcomes. Assumed knowledge, motivation, and organizational needs were generated from related literature, learning and motivation theories, and personal knowledge. The analysis of data from this qualitative case study validated 18 needs that led to three recommended solutions. The proposed solutions in Chapter Five provide concrete strategies and implementation plans for addressing the knowledge, motivation and organization needs to achieving this innovative systemic multilayered assessment model. The three solutions fall under two broad categories: (A) establish measures and capacity of excellence to reinforce institutional accountability, and (B) define, establish and encourage standards of excellence. The three solutions are (1) establish clear goals, communication and oversight of assessment at North College

Linked assets

University of Southern California Dissertations and Theses

Conceptually similar

PDF

Creating "excellent" learning experiences: a gap analysis of a university extension program

PDF

An innovation study: designing a social entrepreneurship coaching model for entrepreneurs

PDF

Improving educational attainment at a bridge program in Saudi Arabia: a gap analysis

PDF

Service-learning and character development: an analysis of Up with People resulting in a model of global citizens for servant leadership

PDF

Linking shared governance and strategic planning processes: an innovation study

PDF

Examining liberal arts colleges achievement of student learning outcomes for a global perspective: an innovation gap analysis study

PDF

College and career readiness through independent study: an innovation study

PDF

Supporting administrators in successful online co-curriculum development: a promising practices study of contributing factors

PDF

Creativity and innovation in undergraduate education: an innovation study

PDF

Application of professional learning outcomes into the classroom: an evaluation study

PDF

The importance of being a global citizen: creating and implementing a global curriculum for the Cutting Edge Youth Summit

PDF

Collaborative instructional practice for student achievement: an evaluation study

PDF

Embedded academic support for high school student success: an innovation study

PDF

Building computations thinking through teaching of computer programming: an evaluation of learning centers in China

PDF

Integrating education for social justice and social innovation: a gap analysis of a high school program innovation to increase justice-oriented action

PDF

Assessing and articulating the impact of the Daniel K. Inouye Asia-Pacific Center for Security Studies: an innovation study

PDF

Chinese high school students' lack of non-academic skills preparation to succeed in their studies abroad

PDF

Creating a comprehensive professional development program for MBA students: a needs analysis

PDF

Effective course design for improving student learning: a case study in application

PDF

Establishing the presence of Georgetown University's School of Continuing Studies in the Russian Federation: a gap analysis

Asset Metadata

Creator Canny, Eric (author)

Core Title Systemic multilayered assessment of global awareness in undergraduate students: an innovation study

School Rossier School of Education

Degree Doctor of Education

Degree Program Global Executive

Publication Date 08/02/2016

Defense Date 05/13/2016

Publisher University of Southern California (original), University of Southern California. Libraries (digital)

Tag assessment,assessment rubrics,experiential learning,gap analysis,global awareness,global eduction,innovation,need analysis,OAI-PMH Harvest,student learning outcomes

Format application/pdf (imt)

Language English

Contributor Electronically uploaded by the author (provenance)

Advisor Krop, Cathy (committee chair), Filback, Robert (committee member), Seli, Helena (committee member)

Creator Email ecanny@usc.edu,ericcanny@gmail.com

Permanent Link (DOI) https://doi.org/10.25549/usctheses-c40-292370

Unique identifier UC11279463

Identifier etd-CannyEric-4706.pdf (filename),usctheses-c40-292370 (legacy record id)

Legacy Identifier etd-CannyEric-4706.pdf

Dmrecord 292370

Document Type Dissertation

Format application/pdf (imt)

Rights Canny, Eric

Type texts

Source University of Southern California (contributing entity), University of Southern California Dissertations and Theses (collection)

Access Conditions The author retains rights to his/her dissertation, thesis or other graduate work according to U.S. copyright law. Electronic access is being provided by the USC Libraries in agreement with the a...

Repository Name University of Southern California Digital Library

Repository Location USC Digital Library, University of Southern California, University Park Campus MC 2810, 3434 South Grand Avenue, 2nd Floor, Los Angeles, California 90089-2810, USA