University of Southern California Dissertations and Theses

Validation of a modified pediatric early warning system (PEWS) score

(USC Thesis Other)

Validation of a modified pediatric early warning system (PEWS) score

PDF

Download

Open document

Flip pages

Download a page range

Download transcript

Copy asset link

Request this asset

Transcript (if available)

Content
VALIDATION OF A MODIFIED PEDIATRIC EARLY WARNING SYSTEM (PEWS) SCORE

by

Sarah Sunshine Rubin

A Thesis Presented to the
FACULTY OF THE USC KECK SCHOOL OF MEDICINE
UNIVERSITY OF SOUTHERN CALIFORNIA
In Partial Fulfillment of the
Requirements for the Degree
MASTER OF SCIENCE
CLINICAL AND BIOMEDICAL INVESTIGATION

May 2014

Copyright 2014 Sarah Sunshine Rubin

ii

Acknowledgements

I would like to thank my mentors, Dr. Roberta Mckean-Cowdin, Dr. Christopher Newth, Dr. Randall
Wetzel and Dr. Stanley Azen for their support and guidance in conducting my thesis project and
preparing this manuscript.

Table of Contents

Acknowledgements ii

List of Tables iv

List of Figures v

Abstract vi

Introduction: 1

Chapter One: Background 2
Subchapter Early Warning Scores
Subchapter Pediatric Early Warning Scores
Subchapter PEWS at Children’s Hospital Los Angeles
Subchapter Research Objectives

Chapter Two: Methods 7
Subchapter Derivation and Implementation of a Modified Brighton PEWS Score
Subchapter Development of a PEWS Education Curriculum
Subchapter Experimental Design
Subchapter Data Collection- Accuracy and Inter-rater Reliability
Subchapter Data Collection- Usability
Subchapter Data Collection- Association with ICU Admission
Subchapter Data Analysis- Accuracy and Inter-rater Reliability
Subchapter Data Analysis- Usability
Subchapter Data Analysis- Association with ICU Admission

Chapter Three: Results 15
Subchapter Accuracy and Inter-rater Reliability
Subchapter Usability
Subchapter Association with ICU Admission
Chapter Four: Discussion 23
Subchapter Accuracy and Inter-rater Reliability
Subchapter Usability
Subchapter Association with ICU Admission

Conclusion 31

References 32

Appendix 35

iv

List of Tables

Table 1: Percent Agreement With PEWS Expert Committee For Each Clinical Vignette

Table 2: Number Of Nurse Responses For Each Clinical Vignette

Table 3: Inter-rater Reliability For Maximum Total PEWS Scores And PEWS Score
Components

Table 4: Patient Demographics

Table 5: Maximum Total PEWS Scores Categorized By Outcome Group

Table 6: Association Of Total PEWS Score And ∆PEWS Score With ICU Admission

Table 7: Performance Of Threshold PEWS Scores For ICU Admission

Table 8: Performance of Threshold ∆PEWS Scores For ICU Admission

Table 9: Patient Demographics Including All Chronic Diagnoses

v

List of Figures

Figure 1: Modified Pediatric Early Warning System (PEWS) Score

Figure 2: Normal Vital Signs for Age

Figure 3: Action To Be Taken Based On PEWS Score

Figure 4: Example of Clinical Vignette For Accuracy And Inter-rater Reliability Testing

Figure 5: PEWS Survey Results For Usability Testing

vi

Abstract

Objectives: To evaluate inter-rater reliability, usability and predictive performance of a modified
Brighton PEWS score.
Study Design: This is a prospective, cohort study conducted on eight inpatient medical-surgical
pediatric wards in a 337-bed freestanding tertiary care children’s hospital. Clinical vignettes were
used to determine accuracy and inter-rater reliability of nurse-assigned PEWS scores. PEWS
score usability was determined by survey of all registered nurses caring for children on any of the
eight pediatric wards. Association of PEWS with clinically relevant decompensation, defined as
need for Pediatric Intensive Care Unit (PICU) or Cardiothoracic Intensive Care Unit (CTICU)
admission, was determined based on prospective evaluation of data from the Electronic Health
Record (EHR).
Results: Five clinical vignettes were distributed to 487 registered nurses. Of these, 352 (72%)
participated in accuracy and inter-rater reliability testing. Overall percent agreement of nurse-
assigned scores with PEWS expert committee assigned scores was 92%. Inter-rater reliability for
all five clinical vignettes was substantial (Fleiss’s Kappa 0.78). Overall 81% of nurses were
satisfied with the PEWS score at CHLA, however usability varied by pediatric ward. Total PEWS
and ∆PEWS were higher in children admitted to the ICU vs. those not admitted to the ICU
(median 4, interquartile range (IQR) 3-6 vs. median 1, IQR 0-2) and (median 3, IQR 1-5 vs.
median 1, IQR 0-2), respectively. Higher total Maximum PEWS score and ∆PEWS were
significantly related to greater ICU admission after adjusting for comorbid conditions (OR
PEWS

2.17, 95% CI 1.86-2.53, OR
ΔPEWS
1.99, 95% CI 1.71-2.32). Adding number of scheduled
medications at hospital admission and complex congenital heart disease increased predictive

vii

performance of ∆PEWS (unadjusted AUC 0.807 vs. adjusted AUC 0.863, P=0.004). Threshold
Maximum PEWS score ≥ 5 had high positive likelihood ratio (+LR 16.5) but resulted in a
relatively low post-test probability of ICU admission of 20.6%.
Conclusion: Following implementation of a comprehensive education curriculum, nurse-assigned
PEWS scores are accurate and reliable. Overall, usability was adequate, but could be improved in
specific nursing wards. Higher modified Brighton PEWS score was associated with greater risk of
ICU admission in a cohort of children admitted to a tertiary care children’s hospital. Predictive
performance may be improved before PEWS score thresholds can be used as trigger criteria for
ICU admission. Incorporation of number of scheduled medications at hospital admission and
presence of chronic cardiovascular condition into PEWS scores may improve predictive
performance. Prospective multicenter evaluation is needed to validate these results.

1

Introduction

Hospitalized children urgently transferred to the Pediatric Intensive Care Unit (PICU) are often at
the point of extreme clinical decompensation. Higher acuity at PICU admission places these
children at greater risk of morbidity and mortality compared to other children admitted to the
PICU. Previous studies have shown a major cause of urgent PICU transfer is failure to recognize
signs and symptoms of impending clinical decompensation (McQuillan et al., 1998). The
Pediatric Early Warning System (PEWS) score is a screening tool developed to improve
recognition of children at risk of clinical decompensation on the pediatric ward. Early recognition
of high risk children will provide an opportunity for clinical intervention with potential to change
the course of illness, prevent catastrophic decompensation and improve outcomes.

A number of different PEWS scores have been developed and implemented at children’s
hospitals around the world. Many of these scores are based on the Brighton PEWS score
developed by Alan Monaghan in 2005 (Monaghan, 2005). Children’s Hospital Los Angeles
(CHLA) implemented a modified version of the Brighton PEWS score in 2010. Despite
acceptance by many children’s hospital, including CHLA, the internal validity, usability and
predictive performance of the Brighton PEWS score is not well established. In particular PEWS
has not been validated in the growing number of children with chronic disease and technology
dependence (gastrostomy feeding tubes, tracheostomy breathing tubes, home ventilation or
indwelling central venous catheters) who are more typical of children admitted to pediatric wards
over the last 10 years.

2

In this single-center study, we investigated the internal validity, usability and predictive
performance of the PEWS score at CHLA. The results of this study may be generalizable to other
tertiary care hospitals using Brighton-based PEWS scoring systems to detect clinical deterioration
in similar patient populations.

3

Chapter One: Background

Early Warning Scores. Early Warning Scores (EWS) were designed to detect and prevent
catastrophic deterioration in adult patients hospitalized on the inpatient medical ward (Duckitt et
al., 2007; Goldhill, McNarry, Mandersloot, & McGinley, 2005; C. Subbe, Kruger, Rutherford, &
Gemmel, 2001). The first EWS was developed in response to a series of studies conducted from
1990-1999 investigating the medical care adult inpatients received prior to acute cardiopulmonary
arrest. Studies demonstrated that signs and symptoms of impending cardiopulmonary arrest often
went undetected due to lack of nursing experience or overburdened nursing staff due to high
patient acuity (Franklin & Mathew, 1994; Garrard & Young, 1998; Goldhill, White, & Sumner,
1999; McQuillan et al., 1998). In addition, when clinical decompensation was recognized, the
bedside nurse often failed to notify the charge nurse or physician, leading to a delay in
implementing potentially life-saving interventions. As a group, these studies highlighted the need
for a screening tool that could quickly and easily assess risk of cardiopulmonary arrest and
provide guidelines for clinical management based on this assessment. A number of EWS have
since been developed with little agreement on criteria that should be included in a standardized
score (Prytherch, Smith, Schmidt, & Featherstone, 2010). Nevertheless, EWS scores have been
increasingly adopted on adult medical wards and have become a widely accepted tool for
detecting clinical deterioration in hospitalized patients.

In most hospitals, EWS are assigned by a bedside nurse at regular pre-determined time intervals.
Aggregate scoring systems require nurses to assign points to a number of physiologic components
based on deviation from normal parameters. Component points are then added together to
determine the total EWS score. Most often, EWS are accompanied by clinical management
4

guidelines recommending action to be taken based on a calculated aggregate score (Gao et al.,
2007). Typically, guidelines include consultation of a Medical Emergency Team for EWS scores
surpassing a pre-defined threshold. Medical Emergency Teams usually consist of an ICU nurse,
ICU physician and respiratory therapist, among others (Jones, DeVita, & Bellomo, 2011). These
teams are called to the bedside to assess patients and help the primary medical team determine
appropriate intervention, which may include transfer to the ICU. Adult studies have demonstrated
higher EWS scores are associated with poorer outcomes including ICU admission, cardiac arrest
and mortality (Reini, Fredrikson, & Oscarsson, 2012; Smith, Prytherch, Meredith, Schmidt, &
Featherstone, 2013); however, it is difficult to translate these results to children as components
included in adult EWS scores do not account for the normal age-based variation in physiologic
parameters in children. It was therefore necessary to develop a Pediatric Early Warning System
(PEWS) score where age-based clinical parameters were defined for identifying clinical
decompensation in children.

Pediatric Early Warning System (PEWS) Scores. The first Pediatric Early Warning System
(PEWS) score was developed by Alan Monaghan at Royal Alexandra Children’s Hospital
(Brighton, UK). Points were assigned to five components based on a child’s behavior, respiratory
and cardiovascular exam, frequency of inhaled medication and presence of persistent post-
operative vomiting (Monaghan, 2005). Total PEWS scores ranged from 0-13 with higher scores
indicating greater risk of clinical decompensation. While a number of other PEWS scores have
since been developed, many of these use components similar to the Brighton score. One
limitation of the Brighton PEWS is a failure to account for the growing prevalence of children
with chronic disease and technology dependence (gastrostomy feeding tubes, tracheostomy
breathing tubes, home ventilation or indwelling central venous catheters) being admitted to
5

pediatric wards in hospitals across the country (Cohen et al., 2011). Previous studies suggest this
population may have an increased risk of clinical decompensation independent of physical exam
or medical history components captured by PEWS (Dosa, Boeing, & Kanter, 2001; Meštrovi et
al., 2006). In 2006, Heather Duncan and colleagues developed and validated a PEWS score where
chronic disease and technology dependence components were included (Duncan, Hutchison, &
Parshuram, 2006). Subsequent studies have shown the Duncan score to outperform other PEWS
scores when directly comparing the outcome of ICU admission. However, the complexity of the
16-component Duncan PEWS score makes it difficult to apply quickly and easily at the bedside,
particularly on a busy inpatient pediatric ward. For this reason, many hospitals have chosen to use
the simpler Brighton PEWS score, with modifications in an attempt to broaden the scores
application to varied patient populations.

PEWS at Children’s Hospital Los Angeles. Based on the high prevalence of children with
chronic disease and technology dependence cared for at CHLA, the PEWS committee chose to
implement a Brighton PEWS score, modified for application to this patient population. The
primary goal of PEWS score implementation was to identify children at high risk for clinical
decompensation on the pediatric ward. Because clinical decompensation is not a well-defined
term, previous PEWS studies have used outcomes such as cardiopulmonary arrest or ICU transfer
as a surrogate for clinical decompensation (Duncan, Hutchison, & Parshuram, 2006b; Parshuram,
Hutchison, & Middaugh, 2009). As the incidence of cardiopulmonary arrest at our hospital was
less than 1.5 % (of all hospitalized patients), the PEWS committee defined clinically relevant
decompensation as need for transfer to the PICU or Cardiothoracic Intensive Care Unit
(CTICU) at our hospital. In parallel, a clinical management guideline was developed, which
recommended actions to take based on total PEWS score.
6

Research Objectives. The objectives of this study were to determine the inter-rater reliability,
usability and predictive performance of a modified Brighton PEWS score in children admitted to
the pediatric ward, including those with chronic disease and technology dependence.

7

Chapter Two: Methods

Derivation and Implementation of a Modified Brighton PEWS Score. The original five-
domain Brighton PEWS was identified as a candidate scoring system for our hospital by a
committee of PEWS experts. The 10-person PEWS expert committee included nurse
administrators, bedside nurses, pediatricians, intensive care physicians and respiratory care
providers. Modifications to the original Brighton score were decided by modified Delphi method
based on literature review and evaluation of critical events in our hospital. Modifications included
removal of two PEWS criteria (frequency of inhaled breathing treatments and presence of
persistent post-operative vomiting) used in the original Brighton score, as these were not easily
extracted from electronic health record (EHR) at CHLA. We added criteria to our score based on
a large number of children with technology dependence, chronic respiratory failure and complex
congenital heart disease cared for at our institution. These included need for positive pressure
ventilation (oxygen delivered via a pressurized system including home ventilator or face mask),
single ventricle cardiac physiology, history of Rapid Response Team consult (RRT; medical
emergency team composed of ICU staff who assist the primary medical team managing patients
on the pediatric ward) or ICU admission within the previous two weeks. The finalized PEWS
score, shown in Figure 1, includes four domains with possible total PEWS scores ranging from
0-10. Normal vital signs for age are posted on the pediatric wards and populated in the EHR for
easy viewing during PEWS score assignment at a patient’s bedside. These vital signs are shown
in Figure 2. PEWS-based management guidelines shown in Figure 3, recommend clinical
actions to be taken based on total PEWS score. Our modified Brighton PEWS scoring system and
PEWS-based management guidelines were implemented in January 2010 after PEWS committee
members completed our hospital-wide education program.
8

Figure 1. Modified Pediatric Early Warning System (PEWS) Score

Single Ventricle Physiology= single cardiac ventricle pumps blood to body or to body and lungs.
Assisted ventilation= Biphasic Positive Airway Pressure or Continuous Positive Airway Pressure
machine via full face mask or mechanical ventilation via home ventilator. FiO2= Fraction of
inspired oxygen. RRT= Rapid Response Team. Code Blue= cardiac or respiratory arrest. PICU=
Pediatric Intensive Care Unit. CTICU= Cardiothoracic Intensive Care Unit.
* as defined in patient’s orders in the electronic health record.

Figure 2. Normal Vital Signs for Age

Heart Rate Respiratory Rate
0-3 months 95-160 30-60
4-11 months 85-150 25-50
1-4 years 80-130 20-40
5-12 years 70-110 20-30
≥ 13 years 60-100 12-20
9

Figure 3. Action To Be Taken Based On PEWS Score
PEWS 0-2 Reassess in 4 hours as per routine
Notify charge nurse
PEWS 3 Notify 1st call physician
Reassess per routine, or sooner as clinically indicated
Notify charge nurse
Notify 1st call physician and Senior physician
PEWS 4 Senior physician to consider notifying Attending
physician
Determine plan of care
Reassess in 1-2 hours
Consider Rapid Response Team consult
Notify charge nurse
Notify 1st call physician and Senior physician
Charge nurse will notify the House Supervisor
PEWS ≥ 5 Senior physician to notify Attending physician
Determine plan of care
Reassess in 1 hour or sooner
Strongly consider Rapid Response Team consult

Development of a PEWS Education Curriculum. Accurate and reliable PEWS score
assignment requires adequate training and education of medical staff. As PEWS scores on the
pediatric ward are typically assigned by the bedside nurse, all registered nurses on each of the
eight pediatric wards were required to complete our comprehensive PEWS education program.
The education program consisted of a series of lectures, group learning exercises and bedside
PEWS score assignment supervised by members of the PEWS committee over a period of six
months. Lectures and exercises developed by the PEWS committee were modified based on
feedback by nurses, physicians and administrators prior to housewide rollout. The education
program continued for one year prior to initiation of our study. A nurse from each of the
pediatric wards was asked to be a “PEWS champion”, helping PEWS committee members
facilitate integration of PEWS into nursing culture, and communicating feedback and questions
from nursing staff back to the PEWS committee.
10

Experimental Design. This was a prospective, cohort study conducted on eight inpatient
medical-surgical pediatric wards in a 337-bed freestanding urban children’s hospital. Clinical
vignettes were used to determine accuracy and inter-rater reliability of nurse-assigned PEWS
scores. PEWS score usability was determined by survey of all registered nurses caring for
children on any of the eight pediatric wards. Association of PEWS with ICU admission was
determined based on prospective collection of PEWS scores from the EHR.
Data Collection - Accuracy and Inter-rater Reliability. Five clinical vignettes were developed
by the PEWS committee for accuracy and inter-rater reliability testing. PEWS expert committee
members agreed upon PEWS score assignation for each of the clinical vignettes and used this as
the answer key for evaluation of PEWS score accuracy. All registered nurses who care for
patients on the pediatric ward at CHLA were asked to participate in accuracy and inter-rater
reliability testing. PEWS committee members attended staff meetings at the beginning of 12-hour
nursing shift on each of the eight pediatric wards to explain the purpose of the study and recruit
participants. A paper packet with the five clinical vignettes was distributed to every registered
nurse working at least one 12-hour shift during the study period. The front page of the packet was
blank. Nurses could choose to participate or not participate by checking the appropriate box on
the second page of the packet. Participants were asked to assign PEWS scores to the fictional
patients presented in each of the five clinical vignettes. These vignettes represented typical
patients cared for on the pediatric ward at CHLA. An example of a clinical vignette is given in
Figure 4. Participants were asked not to confer with others while filling out their response sheets.
Responses were collected anonymously, however, ward numbers were collected to evaluate the
distribution of participants on each of the pediatric wards. PEWS scores assigned by participants
were then recorded into an Excel spreadsheet (Microsoft Excel, 2013) for analysis.
11

Figure 4. Example Of Clinical Vignette for Accuracy And Inter-rater
Reliability Testing

Data Collection - Usability. A 10 question survey was developed by the PEWS committee to
assess PEWS acceptance by nursing staff, and determine frequency and efficiency of bedside use.
Questions were designed using a Likert scale response system. An e-mail was sent out to all
registered nurses who care for patients on the pediatric wards at CHLA to inform them about the
survey and ask for voluntary participation. Paper surveys were distributed to all registered nurses
on each of the eight pediatric wards. Responses were collected anonymously, however, ward
numbers were collected to evaluate the distribution of participants on each of the pediatric wards.

Data Collection - Association with ICU Admission. All children admitted to the pediatric
wards from October 2011 through December 2011 were screened for eligibility. PEWS scores are
typically recorded every four hours on each of the eight pediatric wards. Children ≤ 21 years of
age, with at least two PEWS scores assigned in a 24-hour period were included in the study.
Children with “Accept Natural Death” orders were not routinely assigned PEWS scores and were
therefore excluded from participation. Enrollment began with the first assigned PEWS score and
Clinical Vignette #1
John J. 12yo male
 12yo M with cerebral palsy and
moderate mental retardation
admitted to 5E from ED with
diagnosis of dehydration.
 Cries with IV placement but calmed
by mother.
 HR 125, skin pink, cap refill 2 sec,
diaphoretic (no heart disease)
 RR 20, no retractions, SpO2 97%
on room air
 Discharged home from another
hospital’s PICU 1 week ago
Assign PEWS Score
 Behavior score: 1
 Cardiovascular score: 1
 Respiratory score: 0
 Medical History Score: 0
 Total PEWS score: 2
12

continued until ICU admission, hospital discharge, or transfer to the operating room. The study
protocol was completed following approval by the Institutional Review Board at CHLA.
Demographic, ICU admission and PEWS score data was extracted from the hospital-based EHR
(Cerner Corporation, Kansas City, MO). PEWS scores are routinely documented in the EHR by
the bedside nurse and audited by the hospital’s PEWS quality assurance committee to ensure
accurate and complete documentation. All PEWS scores documented throughout the length of
study participation were extracted into an Excel spreadsheet. PEWS score and demographic data
was verified by manual review of hospital and ICU quality assurance databases and the EHR by
the principal investigator. The highest PEWS score during study enrollment (maximum PEWS
score) was identified for each patient. Change in PEWS score (∆PEWS) was calculated by
subtracting the lowest PEWS score during hospital admission from maximum PEWS score. Both
total PEWS score and ∆PEWS were evaluated for association with ICU admission.
Data Analysis - Accuracy and Inter-rater Reliability. Accuracy of nurse-assigned PEWS
scores was determined by percent agreement with PEWS scores assigned by the PEWS expert
committee. Percent agreement was calculated as concordant cells divided by the sum of
concordant plus discordant cells. Concordance of total PEWS scores was defined as a nurse-
assigned PEWS score that was within 1 point of PEWS expert committee assigned PEWS score.
For example, if the PEWS expert committee assigned a total PEWS score of 3 for a given clinical
vignette, concordant nurse-assigned scores would include total PEWS scores of 2, 3 and 4.
Percent agreement was determined for each clinical vignette and for all clinical vignettes
combined. Inter-rater reliability was defined as consistency of PEWS score assignment among all
nurse participants. Since the PEWS-based clinical management algorithm at CHLA recommends
Rapid Response Team consult for children with PEWS scores ≥ 4, the PEWS expert committee
13

determined this was a clinically important cutoff for inter-rater reliability testing. Therefore
nurse-assigned PEWS scores were dichotomized into < 4 or ≥ 4 for inter-rater reliability analysis.
Inter-rater reliability was also tested separately for each PEWS score component. Inter-rater
reliability was assessed using Fleiss’s Kappa for comparison of greater than two raters.
Data Analysis - Usability. Percent response was analyzed for each of the 10 questions separately
and then grouped into the following four categories: efficiency, usefulness, frequency of use and
overall satisfaction with PEWS. Differences in responses categorized by pediatric ward was
determined using RxC Chi squared analysis with significant p-values < 0.05.
Data Analysis - Association with ICU Admission. The primary goals of our statistical analysis
were to evaluate the association of PEWS with PICU or CTICU admission and to assess the
predictive ability of threshold PEWS scores. Descriptive statistics were performed and data
reported as frequencies, proportions or medians with interquartile ranges (IQR). Two-group
comparison was performed using Wilcoxon rank sum test for non-parametric data and Chi square
test for proportions. Potential predictor variables were identified by univariate analysis. Variables
with P-values ≤ 0.1 in the univariate analysis were eligible for inclusion in our multiple logistic
regression model. Modified hierarchical logistic regression was used to generate the final model.
Odds ratios and 95% confidence intervals were calculated for PEWS score and for categorical
∆PEWS scores. Receiver operating characteristic (ROC) curves were generated and area under
the ROC curve (AUC) calculated to evaluate discrimination of adjusted and unadjusted PEWS
scores. Goodness of fit was evaluated by the Hosmer-Lemeshow test.
Sensitivity, specificity, likelihood ratio and post-test probability were calculated to determine
predictive performance and clinical utility of threshold PEWS scores. Sensitivity was defined as
the percentage of children correctly identified as requiring ICU admission based on nurse-
14

assigned PEWS score. Specificity was defined as the percentage of children correctly identified
as not requiring ICU admission based on nurse-assigned PEWS score. Sensitivity and specificity
were combined to generate likelihood ratios, which indicated the impact of threshold PEWS
scores on certainty of ICU admission. Likelihood ratios and pre-test probabilities were used to
calculate post-test probabilities of ICU admission for each threshold score (Altman & Bland,
1994; Hayden & Brown, 1999). Threshold PEWS scores with high post-test probability (≥ 80%)
were considered clinically useful indicators of ICU admission. ROC curves were generated to
evaluate predictive performance of each threshold PEWS score. Statistical analysis was
performed using Stata 12 (StataCorp. 2011. Stata Statistical Software: Release 12. College
Station, TX: StataCorp LP).

15

Chapter Three: Results

Accuracy and Inter-rater Reliability. Clinical vignettes were distributed to 487 registered
nurses. Of these, 352 (72%) nurses completed the clinical vignettes and agreed to have their
answers anonymously included in our study. Overall percent agreement (nurse-assigned PEWS
score within 1 point of PEWS expert committee-assigned PEWS score) of total PEWS scores for
all five clinical vignettes was 92%. Percent agreement for individual clinical vignettes is given in
Table 1. Percent agreement for clinically relevant threshold PEWS score of 4 is shown in Table
2. Total PEWS score of 4 was determined by the PEWS expert committee to be a clinically
relevant threshold because this is the score at which Rapid Response Team consult is
recommended. Inter-rater reliability for all clinical vignettes combined was substantial with
Fleiss’s Kappa=0.78 (Table 3). Fleiss’s Kappa for individual PEWS score components varied
widely (Table 3).

Table 1. Percent Agreement With PEWS Expert Committee
For Each Clinical Vignette

Expert
Answer
% Agreement
with PEWS
Experts
Clinical Vignette #1 2 94
Clinical Vignette #2 4 86
Clinical Vignette #3 7 91
Clinical Vignette #4 3 93
Clinical Vignette #5 5 98
Expert Answer=Total PEWS score agreed upon by the PEWS expert
committee to be the correct answer.

16

Table 2. Percent Agreement with PEWS Expert Committee for
Threshold PEWS Score of 4.

Total
PEWS < 4
Total
PEWS ≥ 4
Clinical Vignette #1 95% 5%
Clinical Vignette #2 16% 84%
Clinical Vignette #3 1% 99%
Clinical Vignette #4 94% 6%
Clinical Vignette #5 2% 98%
PEWS of 4 chosen as clinically important threshold by the PEWS expert
committee as this is the score at which a Rapid Response Team consult is
recommended.

Table 3. Inter-rater Reliability for Maximum Total PEWS Scores And PEWS Score
Components
PEWS Score Fleiss's Kappa
Total 0.78
Behavior Component 0.82
Respiratory Component 0.62
Cardiovascular Component 0.33
Medical History Component 0.12

Usability. Surveys were distributed to 487 registered nurses. Of these, 400 (82%) surveys were
returned. Responses for each of the 10 survey questions are given in Figure 5. Efficiency of use
was determined by responses to questions 1, 2 and 4. Usefulness of PEWS scores was determined
by questions 3, 5, 6, 7 and 10. Frequency of use was determined by question 9. Overall
satisfaction was determined by question 8. Of the 400 nurses who responded, 84% reported that
PEWS scores were able to be assigned efficiently, 47% nurses believed PEWS scores were
useful, 47% frequently reported PEWS scores when signing over to other nurses. Only 19% of
nurses were not satisfied with the PEWS tool at CHLA.
17

Figure 5. PEWS Survey Results For Usability Testing

Association with ICU Admission. There were 3,242 patients admitted to the eight hospital
wards during the 3-month study period (Table 1). Of these, 51 (1.6%) were transferred to the
ICU, including 43 (1.3%) to the PICU and 8 (0.3%) to the CTICU. Of the 51 ICU admissions, 36
(71%) patients were admitted after RRT consultation and 15 (29%) were admitted without RRT
consult, after discussion between the ICU attending and the attending physician on the pediatric
ward. Respiratory distress 21 (41%) and cardiovascular compromise 13 (25%) were the main
reasons for ICU admission.
There were no statistically significant differences in age, weight, gender or ethnicity between
children admitted and those not admitted to the ICU. ICU admissions had a higher number of
scheduled medications, chronic conditions and technology dependence than children not admitted
to the ICU (Table 4). Technology dependent conditions that were associated with ICU admission
18

in our univariate analysis included presence of a gastrostomy feeding tube, tracheostomy
breathing tube and central venous vascular catheter. Chronic diagnoses including neurologic,
cardiovascular, respiratory, genetic, and immunocompromised condition were also statistically
significantly associated with ICU admission in the univariate model (Table 4). A table with all
eleven of the chronic diagnoses evaluated in our study is presented in the appendix. One
thousand three hundred and eighteen children (41%) had at least one of eleven chronic diagnoses
and 74 children (2%) had ≥ 3 chronic diagnoses.

19

Table 4. Patient Demographics

ICU ADMIT
NO
ICU ADMIT
P value
n=51 n=3191
Age (months) 55 (13, 167) 71 (16, 155) 0.8
Admit Weight (kg) 15.2 (9.4, 34.2) 20.1 (10.1, 44.0) 0.29
Gender (male) 25 (49%) 1709 (54%) 0.52
Ethnicity

0.57
Hispanic 34 (67%) 1986 (62%)
Non-Hispanic 17 (33%) 1150 (36%)
Unknown

55 (2%)
No. Scheduled Medications*

< 0.001
≤ 3 11 (22%) 2114 (66%)
4-9 24 (47%) 840 (26%)
≥ 10 16 (31%) 237 (7%)
Technology Dependent

Gastrostomy Tube 21 (41%) 652 (20%) < 0.001
Tracheostomy 8 (16%) 94 (3%) < 0.001
Central Line in situ 22 (43%) 406 (13%) < 0.001
Chronic Diagnoses

Neurologic 15 (29%) 289 (9%) < 0.001
Cardiovascular 11 (22%) 148 (5%) < 0.001
Respiratory 14 (28%) 391 (12%) 0.001
Genetic 5 (10%) 70 (2%) < 0.001
Immunocompromised 6 (12%) 128 (4%) 0.006
Malignancy 8 (16%) 247 (8%) 0.04
≥ 1 Chronic Diagnoses 38 (75%) 1280 (40%) < 0.001
≥ 3 Chronic Diagnoses 6 (12%) 68 (2%) < 0.001
Neurologic=cerebral palsy, developmental delay, epilepsy, hydrocephalus.
Cardiovascular=cardiomyopathy, primary dysrhythmia or congenital heart disease (excluding single
ventricle cardiac physiology, simple atrial septal defect or ventricular septal defect). Respiratory=chronic
lung disease or bronchopulmonary dysplasia, obstructive sleep apnea, central apnea, asthma or anatomic
airway anomalies. Genetic=chromosomal anomaly including Down syndrome. Immunodeficiency=primary
immunologic disease or transplant recipient (solid organ or hematopoietic). Malignancy=leukemia,
lymphoma, solid tumors.
* Number of scheduled medications at hospital admission

Children admitted to the ICU had higher PEWS scores (median 4, interquartile range (IQR) 3- 6)
than those not admitted to the ICU (median 1, IQR 0- 2, P< 0.001; Table 5). The ∆PEWS scores
20

were also higher for children admitted to the ICU (median 3, IQR 1- 5) than for those not
admitted to the ICU (median 1, IQR 0- 2, P < 0.001). The proportion of children admitted to the
ICU increased from 0.4% to 21% as PEWS score increased from ≤ 2 to ≥ 5 (appendix). Forty-
one percent of children admitted to the ICU had PEWS scores ≥ 5 compared to 3% of children
not admitted to the ICU.

Table 5. Maximum Total PEWS Scores Categorized By Outcome Group

ΔPEWS= Maximum PEWS score – minimum PEWS score during current hospitalization. PEWS scores
presented as median with interquartile range.
*Wilcoxon rank sum test.

Covariates significantly related to ICU admission in both univariate and multivariate analysis
included number of scheduled medications and cardiovascular condition. After adjusting for these
variables we found that children with higher PEWS scores were at significantly increased risk of
ICU admission, with risk of ICU admission increasing by 117% for each 1-point increase in
PEWS score (Table 6).

ICU Admit
(N=51)
Total
(N=3191)
P value*
PEWS at Hospital Admission 1 (0,3) 0 (0,1) < 0.001
Maximum PEWS 4 (3,6) 1 (0,2) < 0.001
∆PEWS 3 (1,5) 1 (0,2) < 0.001
21

Table 6. Association of Total PEWS Score and ∆PEWS Score With ICU Admission

OR=odds ratio. AUC=Area Under Receiver Operating Characteristic Curve. ΔPEWS=Maximum PEWS
score – minimum PEWS score during current hospitalization.
*Adjusted for number of scheduled medications and cardiovascular condition (cardiomyopathy,
dysrhythmia or complex congenital heart disease excluding simple atrial septal defect, ventricular septal
defect, or single ventricle cardiac physiology).
a
Unadjusted AUC vs. adjusted AUC (0.876 vs. 0.896, P=0.09)
b
Unadjusted AUC vs. adjusted AUC (0.807 vs. 0.863, P=0.004)

We attempted to identify thresholds for Maximum PEWS score and ∆PEWS score where a need
for ICU admission could be predicted (Tables 7 and 8). We found a Maximum PEWS score of 3
had the best combination of sensitivity (74%) and specificity (83%). This resulted in a positive
likelihood ratio (+LR) of 4.73 and post-test probability of ICU admission of only 6.9%.
Maximum PEWS scores of 4 and 5 had lower sensitivity (61% and 41% respectively) than
Maximum PEWS score of 3, but higher +LRs resulting in higher post-test probabilities of ICU
admission of 12.8% and 20.6%. Given the pre-test probability (prevalence) of ICU admission for
the entire cohort was 1.6%, a threshold PEWS score of 4 increased probability of ICU admission
by 11.2% (from 1.6% to 12.8%) and a threshold PEWS score of 5 increased probability of ICU
admission by 19% (from 1.6% to 20.6%). While a threshold score of 3 had good predictive
performance (AUC=0.809), predictive ability of other threshold scores was fair to poor
(AUC=0.795 - 0.680).
When ∆PEWS scores were analyzed, we found a threshold score of 2 had the best combination of
sensitivity (75%) and specificity (73%). This resulted in a +LR of 2.75 and relatively low post-
OR P value AUC OR P value AUC
Maximum PEWS 2.35 (2.03-2.72) <0.0001 0.876 2.17 (1.86-2.53) <0.0001 0.896
a
∆ PEWS 2.18 (1.88-2.52) <0.0001 0.807 1.99 (1.71-2.32) <0.0001 0.863
b
0
1 1.93 (0.65-5.77) 0.24 1.52 (0.50-4.57) 0.46
2 3.90 (1.30-11.68) 0.02 2.79 (0.92-8.46) 0.07
3 6.09 (2.03-18.27) 0.001 4.10 (1.34-12.52) 0.01
4 14.97 (4.74-47.28) <0.001 8.03 (2.44-26.49) 0.001
Unadjusted Adjusted*
22

test probability of 4%. ∆PEWS scores of 2 and 3 had fair predictive ability for ICU admission
(AUC=0.737) while predictive performance of other threshold scores was fair to poor
(AUC=0.712 - 0.668).
Table 7. Performance of Threshold Total PEWS Scores For ICU Admission
Threshold PEWS
Score
Sensitivity Specificity LR+
Positive
Post-test
Probability
AUC
(unadjusted)
< 1, ≥ 1 94.1 41.9 1.6 2.5% 0.680
< 2, ≥ 2 90.2 68.7 2.9 4.4% 0.795
< 3, ≥ 3 78.4 83.4 4.7 6.9% 0.809
< 4, ≥ 4 60.8 93.5 9.3 12.8% 0.771
< 5, ≥ 5 41.2 97.5 16.5 20.6% 0.693
Threshold PEWS scores compare incidence of ICU admission in children with PEWS < threshold to those
with PEWS ≥ threshold value. +LR=Positive Likelihood Ratio. AUC=Area Under Receiver Operating
Characteristic Curve. Positive Post-test Probability=probability of ICU admission based on PEWS score

Table 8. Performance of Threshold ∆PEWS Scores For ICU Admission
ΔPEWS Score Sensitivity Specificity LR+
Positive
Post-test
Probability
AUC
(unadjusted)
≥ 1 88.2 45.4 1.62 2.4% 0.668
≥ 2 74.5 72.9 2.75 4.0% 0.737
≥ 3 60.8 86.5 4.51 6.2% 0.737
≥ 4 47.1 95.3 9.93 11.9% 0.712
≥ 5 35.3 98.3 20.96 19.9% 0.668
Threshold PEWS scores compare ICU admission in children with PEWS < threshold to those with PEWS ≥
threshold value. +LR=Positive Likelihood Ratio. AUC=Area Under Receiver Operating Characteristic
Curve.

23

Chapter Four: Discussion

We conducted a prospective cohort study to evaluate the inter-rater reliability, usability and
predictive performance of a modified Brighton PEWS score in a population of children with high
prevalence of technology dependence and chronic illness. We found substantial inter-rater
reliability when using categorical PEWS scores < 4 or ≥ 4, the clinically relevant threshold at
which RRT consult is recommended. In addition, usability was high, with 97% of bedside nurses
reporting PEWS could be efficiently assigned and 74% reporting that PEWS was useful in caring
for children on the pediatric ward. Only 19% of nurses indicated they were not satisfied with the
PEWS tool at CHLA, with respect to usability. PEWS scores had good discrimination for ICU
admission based on ROC curves; however, a clinically useful threshold for ICU admission could
not be identified due to low post-test probability (maximum post-test probability of 20.6% for a
threshold PEWS score of 5).
Though nurse-assigned PEWS scores are quickly becoming the standard of care in many
children’s hospitals, the accuracy and inter-rater reliability of PEWS scoring systems, including
the modified Brighton PEWS score used at CHLA, has not been well studied. The few studies
evaluating inter-rater reliability have limited testing to ≤ 4 raters, which does not account for the
variability in PEWS score assignment occurring when multiple raters with different levels of
experience and training are involved (C. P. Subbe, Gao, & Harrison, 2007; Tucker, Brewer,
Baker, Demeritt, & Vossmeyer, 2009). Our study represents the largest evaluation of PEWS
score accuracy and inter-rater reliability to date. We showed that implementation of a
comprehensive PEWS training and education program for all registered nurses resulted in
clinically relevant PEWS scores being accurately and reliably assigned. Nurse-assigned PEWS
24

scores were in agreement with PEWS expert assigned scores 92% of the time. In addition,
clinically relevant PEWS scores < 4 and ≥ 4 were reliably and consistently assigned by bedside
nurses. These results support expansion of our education program to train other medical staff
assigning PEWS scores in a hospital setting, including transport and emergency department
nurses and physicians. This education program may also serve as a model for PEWS score
training at other hospitals. With appropriate training, PEWS scores may be consistently assigned
by nurses with different levels of experience. While inter-rater reliability of dichotomized PEWS
scores was substantial, reliability decreased when additional categories of PEWS scores were
considered. For example, when scores were divided into 4 groups (0-2, 3, 4, ≥ 5), Fleiss’s Kappa
was only 0.40. These results suggest that while PEWS scores are reliably assigned for higher
acuity patients, PEWS score assignment is not as reliable for patients who are lower acuity (those
who have PEWS scores < 4). We also evaluated inter-rater reliability for each of the PEWS score
components. While behavior and respiratory components had substantial inter-rater reliability, the
reliability of cardiovascular and medical history components was poor. Based on verbal feedback
from nurses who had completed the clinical vignettes, it appears that disagreement in
cardiovascular scores centered around scoring a patient’s skin color as mottled or not mottled. As
infants often appear mottled when crying, many nurses felt that the mottled skin of a crying infant
described in one of the clinical vignettes was normal. Nurses therefore, assigned a cardiovascular
score of 0 to the infant. However, because it is often difficult to determine if mottled skin is due
to crying or due to worsening of a patient’s disease process, the PEWS committee decided a
priori that mottled skin should always receive a cardiovascular score of 2. In addition, many
nurses did not realize that within the medical history category, “PICU or CTICU admission
within the last 2 weeks” referred only to ICU admission at our hospital. The logic behind this is
that ICU admission may not confer the same level of high acuity at other hospitals as it does at
25

our own. Low inter-rater reliability in the cardiovascular and medical history categories suggested
additional training was required and led to modification of our education program after study
completion to address this issue.
Nursing perception of the efficiency and utility of PEWS scores greatly influences acceptance by
medical staff. Therefore, interference with nursing workflow is the primary concern when
implementing screening tools such as PEWS. Our survey results demonstrate PEWS score
assignment is easy and efficient, with the majority of nurses spending ≤ 2 minutes to assign and
document a bedside PEWS score in the EHR. Since PEWS scores are routinely assigned every
four hours on the pediatric ward, a nurse caring for four children requires only eight minutes
every four hours for PEWS score assignment, resulting in minimal impact on nursing workflow.
Surprisingly, only half of bedside nurses felt PEWS scores were useful in identifying clinical
deterioration, helping care for deteriorating patients and improving communication between
members of the medical team. A similar number consistently reported PEWS scores when signing
out their patient to another nurse. When we analyzed survey responses by pediatric ward we
found nurses who did not find PEWS to be useful or did not consistently report PEWS scores
were mostly limited to those caring for patients on the oncology ward. One reason for this may be
the lack of a dedicated oncology ward “PEWS champion” to facilitate acceptance and integration
of PEWS into the nursing culture. This is in contrast to the other pediatric wards where nurses or
physicians have been involved in PEWS development and implementation and have led the
cultural shift on their wards. Despite this finding, overall satisfaction with the PEWS score was
reported by a majority of nurses.
The modified Brighton PEWS score evaluated in our study included unique criteria to capture the
increased risk of clinical deterioration reported in children with chronic disease and technology
26

dependence (Dosa et al., 2001; Feudtner, Christakis, & Connell, 2000; Marcin, Slonim, Pollack,
& Ruttimann, 2001). Addition of “static” criteria (criteria that do not change over time) to
dynamic vital sign and physical exam criteria, has been previously shown to improve PEWS
score discrimination (Duncan et al, 2006). The addition of static criteria, including single
ventricle physiology, baseline need for assisted ventilation and medical history, differentiate our
score from the modified Brighton PEWS scores evaluated in previous studies (Akre et al., 2010;
Skaletzky, Raszynski, & Totapally, 2011; Tucker et al., 2009). In addition, this is the first study
to evaluate the effect of chronic disease and technology dependence on predictive performance of
the Brighton PEWS score. We found PEWS scores were higher in children admitted to the ICU
than in children not admitted to the ICU and each 1-point increase in PEWS score more than
doubled the risk of ICU admission. These results remained significant, even after adjusting for
number of scheduled medications and cardiovascular condition. Children with higher PEWS
scores may benefit from increased frequency of monitoring and clinical evaluation by senior
medical staff while on the pediatric ward. Nevertheless, the discrete score at which these
interventions should be initiated remained unclear.
We found a statistically significant increase in risk of ICU admission with increasing ΔPEWS
score, even after adjusting for number of scheduled medications and cardiovascular condition. It
is likely the number of scheduled medications recorded at hospital admission is a surrogate for
chronic disease, as previous studies support that children with chronic disease have more
scheduled medications at baseline than children without pre-existing medical conditions (Stein,
Bauman, Westbrook, Coupey, & Ireys, 1993). Studies also suggest children with complex
congenital heart disease may be more medically fragile than other children (McLellan & Connor,
2013). These children may require prolonged hospitalization and are at greater risk of hospital
acquired infection and combined cardiac and respiratory decompensation than patients without
27

complex congenital heart disease. Based on our results, children receiving ≥ 3 scheduled
medications and those with complex congenital heart disease have an increased risk of clinical
deterioration independent of PEWS score value. Incorporation of these variables may improve
discrimination ability of Brighton PEWS scores in similar patient populations.

Other hospitals have also recognized the need to tailor PEWS scores to specific populations,
particularly in patients with complex congenital heart disease. Boston Children’s Hospital
designed a cardiac early warning score that included variables such as cardiac ectopy (irregular
heart beat), arrhythmia (irregular rhythm), seizure activity and staff or family concern. In
addition, the Cardiac Children’s Hospital Early Warning Score (Cardiac-CHEWS) assigned
points based on change from baseline oxygen requirement and gave more points to small
increases in supplemental oxygen. This group found that the Cardiac-CHEWS had higher AUC
(0.917) compared to PEWS (0.785) for unplanned cardiothoracic ICU transfer or cardiac arrest in
a population of children with complex congenital heart disease (McLellan et al., 2013). While
staff and family concern is recognized as a risk for ICU admission, this variable is not well
documented and therefore difficult to evaluate retrospectively {{313 Brady,P.W. 2013}}.
However, situational awareness tools that require regular documentation of staff and family
concern are being increasingly implemented at CHLA and other children’s hospitals.
Incorporation of these tools into the EHR will enable future investigation of predictive
performance if staff and family concern are added to the CHLA PEWS score and applied to a
heterogeneous patient population. More conservative scoring within the respiratory category may
also improve discrimination of the CHLA PEWS score and should be investigated in future
studies.

28

Though previous studies suggest ∆PEWS scores increased prior to a deterioration event, this is
the first study to quantify the relationship between ∆PEWS and ICU admission (Akre et al., 2010;
Duncan et al., 2006b; Parshuram et al., 2009). We found children with ∆PEWS ≥ 3 were more
than four times as likely to be admitted to the ICU compared to children with ∆PEWS of 0. These
children may benefit from increased frequency of monitoring and evaluation by medical staff on
the pediatric ward. Future studies should evaluate the association of ∆PEWS with additional
outcomes such as RRT consultation to establish the clinical utility of incorporating ∆PEWS
scores into PEWS-based management guidelines.

Various threshold PEWS scores were evaluated to determine if any score had sufficient predictive
ability to be used as a trigger for ICU admission. We demonstrated a threshold PEWS score of 3
to have the best combination of sensitivity and specificity. When compared to similar thresholds
identified in other Brighton-based PEWS studies our score was less sensitive but more specific
than other scores in their respective populations (Akre et al., 2010; Skaletzky et al., 2011; Tucker
et al., 2009). Brighton-based PEWS scores evaluated by Tucker et al. and Akre et al. included
two criteria (frequency of nebulizer treatments and post-operative vomiting) not included in our
study score. Although we did not evaluate predictive performance of individual scoring criteria, it
is possible the exclusion of these criteria contributed to the low sensitivity of our study score.
Underlying differences in patient population, including comorbid conditions, may have also
contributed to differences in sensitivity and specificity.

We evaluated likelihood ratios and post-test probabilities to further test the predictive ability of
threshold PEWS scores. Generally, a threshold score with a positive likelihood ratio ≥ 10
indicates high post-test probability of ICU admission if pre-test probability (prevalence) of ICU
29

admission is ≥ 30. However, the pre-test probability of ICU admission in our study cohort was
only 1.6%. Therefore, post-test probability of ICU admission remained low, even for threshold
PEWS scores with likelihood ratios ≥ 10. A relatively high threshold PEWS score of 5 resulted in
post-test probability of ICU admission of only 20.6%. Clinically, this means children with PEWS
scores ≥ 5 had only a 20.6% likelihood of requiring ICU admission. Although knowing a
patient’s PEWS score was ≥ 5 increased the likelihood of ICU admission from 1.6% to 20.6%,
overall predictive ability was too low to be considered useful for determining need for ICU
admission. Our results suggest predictive performance needs to be improved before using
threshold PEWS score as trigger criteria for ICU admission. Realistically, the low pre-test
probability of ICU admission makes it unlikely that an adequately high post-test probability of
ICU admission can be achieved when applying PEWS to the general pediatric population. It is
possible, that PEWS may be more effective when used in high-risk sub-populations, such as those
admitted to an intensive care step down unit or those with multiple cardiac risk factors. While this
study was not powered for subgroup analysis, the predictive performance of PEWS in high-risk
sub-populations should be evaluated in future studies.

Our study had some important limitations. Data collection was limited by the need for manual
entry of PEWS scores and other data into the EHR by the bedside nurse. This problem could be
alleviated by automated extraction of vital sign data from central monitoring systems into the
EHR (Churpek et al. 2014). Unfortunately, this technology has not yet been implemented at our
hospital. Failure to document elevated PEWS scores during decompensation events that resulted
in ICU admission would have increased the number of false positives and decreased the
sensitivity of our study score. The relatively low prevalence of our primary outcome and
restriction of data collection to a three-month time frame limited our ability for subgroup
30

analysis. As a result, direct comparison of PEWS score performance within comorbidity
subgroups will need to be addressed in future studies. Because of the timeframe we were also
unable to account for seasonal variability in ICU admission rate. It must also be noted that
although a number of different PEWS scoring systems exist our results are most generalizable to
hospitals using Brighton-based scoring systems. Generalizability may also be limited to
quaternary or tertiary care centers caring for children with similar demographics to our patient
population.

31

Conclusion

We conclude that following implementation of a comprehensive education curriculum, nurse-
assigned PEWS scores are accurate and reliable. Overall, usability was adequate, but could be
improved in specific nursing wards. Higher modified Brighton PEWS score was associated with
greater risk of ICU admission in a cohort of children admitted to a tertiary care children’s
hospital. Predictive performance may be improved before PEWS score thresholds can be used as
trigger criteria for ICU admission. Incorporation of number of scheduled medications at hospital
admission and presence of chronic cardiovascular condition into PEWS scores may improve
predictive performance. Prospective multicenter evaluation is needed to validate these results.

32

References

Akre, M., Finkelstein, M., Erickson, M., Liu, M., Vanderbilt, L., & Billman, G. (2010).
Sensitivity of the pediatric early warning score to identify patient deterioration. Pediatrics,
125(4), e763.
Altman, D. G., & Bland, J. M. (1994). Statistics notes: Diagnostic tests 2: Predictive values. Bmj,
309(6947), 102.
Brady, P., Muething, S., Kotagal, U., Ashby, M., Gallagher, D.H., Goodfriend, M., et al. (2013).
Improving Situation Awareness to Reduce Unrecognized Clinical Deterioration and Serious
Safety Events. Pediatrics, 131; e298.
Cohen, E., Kuo, D. Z., Agrawal, R., Berry, J. G., Bhagat, S. K., Simon, T. D., & Srivastava, R.
(2011). Children with medical complexity: An emerging population for clinical and research
initiatives. Pediatrics, 127(3), 529-538. doi:10.1542/peds.2010-0910; 10.1542/peds.2010-
0910.
Churpek, M., Yuen, T., Park, SY., Gibbons, R., Edelson, D. (2014). Using Electronic Health
Record Data to Develop and Validate a Prediction Model for Adverse Outcomes in the
Wards. Crit Care Med, 42; 841-848.
Dosa, N. P., Boeing, N. M., & Kanter, R. K. (2001). Excess risk of severe acute illness in children
with chronic health conditions. Pediatrics, 107(3), 499-504.
Duckitt, R. W., Buxton-Thomas, R., Walker, J., Cheek, E., Bewick, V., Venn, R., & Forni, L. G.
(2007). Worthing physiological scoring system: Derivation and validation of a physiological
early-warning system for medical admissions. an observational, population-based single-
centre study. British Journal of Anaesthesia, 98(6), 769-774. doi:10.1093/bja/aem097
Duncan, H., Hutchison, J., & Parshuram, C. S. (2006). The pediatric early warning system score:
A severity of illness score to predict urgent medical need in hospitalized children. Journal of
Critical Care, 21(3), 271-278.
Feudtner, C., Christakis, D. A., & Connell, F. A. (2000). Pediatric deaths attributable to complex
chronic conditions: A population-based study of washington state, 1980–1997. Pediatrics,
106(Supplement 1), 205-209.
Franklin, C., & Mathew, J. (1994). Developing strategies to prevent inhospital cardiac arrest:
Analyzing responses of physicians and nurses in the hours before the event. Critical Care
Medicine, 22(2), 244-247.
Gao, H., McDonnell, A., Harrison, D. A., Moore, T., Adam, S., Daly, K.,et al. (2007). Systematic
review and evaluation of physiological track and trigger warning systems for identifying at-
risk patients on the ward. Intensive Care Medicine, 33(4), 667-679.
33

Garrard, C., & Young, D. (1998). Suboptimal care of patients before admission to intensive care.
is caused by a failure to appreciate or apply the ABCs of life support. BMJ (Clinical Research
Ed.), 316(7148), 1841-1842.
Goldhill, D., McNarry, A., Mandersloot, G., & McGinley, A. (2005). A physiologically ‐ based
early warning score for ward patients: The association between score and outcome*.
Anaesthesia, 60(6), 547-553.
Goldhill, D., White, S., & Sumner, A. (1999). Physiological values and procedures in the 24 h
before ICU admission from the ward. Anaesthesia, 54(6), 529-534.
Hayden, S. R., & Brown, M. D. (1999). Likelihood ratio: A powerful tool for incorporating the
results of a diagnostic test into clinical decisionmaking. Annals of Emergency Medicine,
33(5), 575-580.
Jones, D. A., DeVita, M. A., & Bellomo, R. (2011). Rapid-response teams. New England Journal
of Medicine, 365(2), 139-146.
Marcin, J. P., Slonim, A. D., Pollack, M. M., & Ruttimann, U. E. (2001). Long-stay patients in
the pediatric intensive care unit. Critical Care Medicine, 29(3), 652-657.
McLellan, M. C., & Connor, J. A. (2013). The cardiac children's hospital early warning score (C-
CHEWS). Journal of Pediatric Nursing, 28(2), 171-178.
McQuillan, P., Pilkington, S., Allan, A., Taylor, B., Short, A., Morgan, G., et al. (1998).
Confidential inquiry into quality of care before admission to intensive care. Bmj, 316(7148),
1853-1858.
Meštrovi, J., Kardum, G., Poli, B., Meštrovi, M., Marki, J., Šusti, A., & Krželj, V. (2006). The
influence of chronic health conditions on susceptibility to severe acute illness of children
treated in PICU. European Journal of Pediatrics, 165(8), 526-529.
Monaghan, A. (2005). Detecting and managing deterioration in children. Paediatr Nurs, 17(1),
32-35.
Parshuram, C., Hutchison, J., & Middaugh, K. (2009). Development and initial validation of the
bedside paediatric early warning system score. Critical Care, 13(4), R135.
Prytherch, D. R., Smith, G. B., Schmidt, P. E., & Featherstone, P. I. (2010). ViEWS—towards a
national early warning score for detecting adult inpatient deterioration. Resuscitation, 81(8),
932-937.
Reini, K., Fredrikson, M., & Oscarsson, A. (2012). The prognostic value of the modified early
warning score in critically ill patients: A prospective, observational study. European Journal
of Anaesthesiology, 29(3), 152-157. doi:10.1097/EJA.0b013e32835032d8;
10.1097/EJA.0b013e32835032d8
34

Skaletzky, S. M., Raszynski, A., & Totapally, B. R. (2011). Validation of a modified pediatric
early warning system score: A retrospective Case–Control study. Clinical Pediatrics,
Smith, G. B., Prytherch, D. R., Meredith, P., Schmidt, P. E., & Featherstone, P. I. (2013). The
ability of the national early warning score (NEWS) to discriminate patients at risk of early
cardiac arrest, unanticipated intensive care unit admission, and death. Resuscitation, 84(4),
465-470.
Stein, R. E., Bauman, L. J., Westbrook, L. E., Coupey, S. M., & Ireys, H. T. (1993). Framework
for identifying children who have chronic conditions: The case for a new definition. The
Journal of Pediatrics, 122(3), 342-347.
Subbe, C. P., Gao, H., & Harrison, D. A. (2007). Reproducibility of physiological track-and-
trigger warning systems for identifying at-risk patients on the ward. Intensive Care Medicine,
33(4), 619-624.
Subbe, C., Kruger, M., Rutherford, P., & Gemmel, L. (2001). Validation of a modified early
warning score in medical admissions. QJm, 94(10), 521.
Tucker, K. M., Brewer, T. L., Baker, R. B., Demeritt, B., & Vossmeyer, M. T. (2009).
Prospective evaluation of a pediatric inpatient early warning scoring system. Journal for
Specialists in Pediatric Nursing, 14(2), 79-85.

35

Appendix
Table 9. Patient Demographics Including All Chronic Diagnoses

ICU ADMIT
NO
ICU ADMIT
P value
n=51 n=3191
Age (months) 55 (13, 167) 71 (16, 155) 0.8
Admit Weight (kg) 15.2 (9.4, 34.2) 20.1 (10.1, 44.0) 0.29
Gender (male) 25 (49%) 1709 (54%) 0.52
Ethnicity

0.57
Hispanic 34 (67%) 1986 (62%)
Non-Hispanic 17 (33%) 1150 (36%)
Unknown

55 (2%)
No. Scheduled Medications*

< 0.001
≤ 3 11 (22%) 2114 (66%)
4-9 24 (47%) 840 (26%)
≥ 10 16 (31%) 237 (7%)
Technology Dependent

Gastrostomy Tube 21 (41%) 652 (20%) < 0.001
Tracheostomy 8 (16%) 94 (3%) < 0.001
Central Line in situ 22 (43%) 406 (13%) < 0.001
Chronic Diagnoses

Neurologic 15 (29%) 289 (9%) < 0.001
Cardiovascular 11 (22%) 148 (5%) < 0.001
Respiratory 14 (28%) 391 (12%) 0.001
Genetic 5 (10%) 70 (2%) < 0.001
Immunocompromised 6 (12%) 128 (4%) 0.006
Malignancy 8 (16%) 247 (8%) 0.04
Gastrointestinal 2 (4%) 108 (3%) 0.83
Hematologic 1 (2%) 117 (4%) 0.52
Metabolic 1 (2%) 74 (2%) 0.87
Renal 0 (0%) 93 (3%) 0.22
Rheumatologic 0 (0%) 22 (1%) 0.55
≥ 1 Chronic Diagnoses 38 (75%) 1280 (40%) < 0.001
≥ 3 Chronic Diagnoses 6 (12%) 68 (2%) < 0.001
Neurologic=cerebral palsy, developmental delay, epilepsy, hydrocephalus.
Cardiovascular=cardiomyopathy, primary dysrhythmia or complex congenital heart disease (excluding
single ventricle cardiac physiology, simple atrial septal defect or ventricular septal defect).
Respiratory=chronic lung disease or bronchopulmonary dysplasia, obstructive sleep apnea, central apnea,
asthma or anatomic airway anomalies. Genetic=chromosomal anomaly including Down syndrome.
Immunodeficiency=primary immunologic disease or transplant recipient (solid organ or hematopoietic).
Malignancy=leukemia, lymphoma, solid tumors. Gastrointestinal=inflammatory bowel disease, congenital
anomalies, chronic liver disease. Hematologic=sickle cell anemia, aplastic anemia, coagulation defects.
Metabolic=storage disorders, inborn errors of metabolism, mitochondrial disease, diabetes mellitus.
Renal=end stage renal disease, congenital genitourinary anomalies. Rheumatologic=Systemic lupus
erythematosis, vasculitis, juvenile rheumatoid arthritis.
* Number of scheduled medications at hospital admission

Abstract (if available)

Abstract Objectives: To evaluate inter‐rater reliability, usability and predictive performance of a modified Brighton PEWS score. ❧ Study Design: This is a prospective, cohort study conducted on eight inpatient medical‐surgical pediatric wards in a 337‐bed freestanding tertiary care children’s hospital. Clinical vignettes were used to determine accuracy and inter‐rater reliability of nurse‐assigned PEWS scores. PEWS score usability was determined by survey of all registered nurses caring for children on any of the eight pediatric wards. Association of PEWS with clinically relevant decompensation, defined as need for Pediatric Intensive Care Unit (PICU) or Cardiothoracic Intensive Care Unit (CTICU) admission, was determined based on prospective evaluation of data from the Electronic Health Record (EHR). ❧ Results: Five clinical vignettes were distributed to 487 registered nurses. Of these, 352 (72%) participated in accuracy and inter‐rater reliability testing. Overall percent agreement of nurse‐assigned scores with PEWS expert committee assigned scores was 92%. Inter‐rater reliability for all five clinical vignettes was substantial (Fleiss’s Kappa 0.78). Overall 81% of nurses were satisfied with the PEWS score at CHLA, however usability varied by pediatric ward. Total PEWS and ∆PEWS were higher in children admitted to the ICU vs. those not admitted to the ICU (median 4, interquartile range (IQR) 3-6 vs. median 1, IQR 0-2) and (median 3, IQR 1-5 vs. median 1, IQR 0-2), respectively. Higher total Maximum PEWS score and ∆PEWS were significantly related to greater ICU admission after adjusting for comorbid conditions (ORPEWS 2.17, 95% CI 1.86-2.53, ORΔPEWS 1.99, 95% CI 1.71-2.32). Adding number of scheduled medications at hospital admission and complex congenital heart disease increased predictive performance of ∆PEWS (unadjusted AUC 0.807 vs. adjusted AUC 0.863, P=0.004). Threshold Maximum PEWS score ≥ 5 had high positive likelihood ratio (+LR 16.5) but resulted in a relatively low post-test probability of ICU admission of 20.6%. ❧ Conclusion: Following implementation of a comprehensive education curriculum, nurse‐assigned PEWS scores are accurate and reliable. Overall, usability was adequate, but could be improved in specific nursing wards. Higher modified Brighton PEWS score was associated with greater risk of ICU admission in a cohort of children admitted to a tertiary care children’s hospital. Predictive performance may be improved before PEWS score thresholds can be used as trigger criteria for ICU admission. Incorporation of number of scheduled medications at hospital admission and presence of chronic cardiovascular condition into PEWS scores may improve predictive performance. Prospective multicenter evaluation is needed to validate these results.

Linked assets

University of Southern California Dissertations and Theses

Conceptually similar

PDF

Association of Pediatric Early Warning Score with early intensive care unit readmission

PDF

Validation of the pancreatitis activity scoring system in a large prospective cohort

PDF

A retrospective assessment of the safety of sildenafil use in pediatric pulmonary hypertension

PDF

Outcomes for small intestinal atresia at high and low-volume hospitals in California

PDF

Age related macular degeneration in Latinos: risk factors and impact on quality of life

PDF

Cardiac function in children and young adults treated with MEK inhibitors: a retrospective cohort study of routinely collected health data

Asset Metadata

Creator Rubin, Sarah Sunshine (author)

Core Title Validation of a modified pediatric early warning system (PEWS) score

School Keck School of Medicine

Degree Master of Science

Degree Program Clinical and Biomedical Investigations

Publication Date 04/26/2014

Defense Date 04/01/2014

Publisher University of Southern California (original), University of Southern California. Libraries (digital)

Tag accuracy,early warning score,inter-rater reliability,OAI-PMH Harvest,Prevention

Format application/pdf (imt)

Language English

Contributor Electronically uploaded by the author (provenance)

Advisor Azen, Stanley P. (committee chair), Mckean-Cowdin, Roberta (committee member), Newth, Christopher J. L. (committee member), Wetzel, Randall C. (committee member)

Creator Email sarrubin@chla.usc.edu

Permanent Link (DOI) https://doi.org/10.25549/usctheses-c3-384187

Unique identifier UC11296611

Identifier etd-RubinSarah-2418.pdf (filename),usctheses-c3-384187 (legacy record id)

Legacy Identifier etd-RubinSarah-2418.pdf

Dmrecord 384187

Document Type Thesis

Format application/pdf (imt)

Rights Rubin, Sarah Sunshine

Type texts

Source University of Southern California (contributing entity), University of Southern California Dissertations and Theses (collection)

Access Conditions The author retains rights to his/her dissertation, thesis or other graduate work according to U.S. copyright law. Electronic access is being provided by the USC Libraries in agreement with the a...

Repository Name University of Southern California Digital Library

Repository Location USC Digital Library, University of Southern California, University Park Campus MC 2810, 3434 South Grand Avenue, 2nd Floor, Los Angeles, California 90089-2810, USA