Close
About
FAQ
Home
Collections
Login
USC Login
Register
0
Selected
Invert selection
Deselect all
Deselect all
Click here to refresh results
Click here to refresh results
USC
/
Digital Library
/
Computer Science Technical Report Archive
/
USC Computer Science Technical Reports, no. 888 (2007)
(USC DC Other)
USC Computer Science Technical Reports, no. 888 (2007)
PDF
Download
Share
Open document
Flip pages
Contact Us
Contact Us
Copy asset link
Request this asset
Transcript (if available)
Content
On the Prevalence of Sensor Faults in Real-World
Deploymets
Abhishek Sharma, Leana Golubchik, and Ramesh Govindan
Department of Computer Science
University of Southern California
(Email:fabsharma, leana, rameshg@usc.edu)
Abstract— Various sensor network measurement studies
have reported instances of transient faults in sensor read-
ings. In this work, we seek to answer a simple question:
How often are such faults observed in real deployments? To
do this, we first explore and characterize three qualitatively
different classes of fault detection methods. Rule-based
methods leverage domain knowledge to develop heuristic
rules for detecting and identifying faults. Estimation meth-
ods predict “normal” sensor behavior by leveraging sensor
correlation, flagging anomalous sensor readings as faults.
Finally, learning-based methods are trained to statistically
identify classes of faults. We find that these three classes of
methods sit at different points on the accuracy/robustness
spectrum. Rule-based methods can be highly accurate,
but their accuracy depends critically on the choice of
parameters. Learning methods can be cumbersome, but
can accurately detect and classify faults. Estimation meth-
ods are accurate, but cannot classify faults. We apply
these techniques to four real-world sensor data sets and
find that the prevalence of faults as well as their type
varies with data sets. All three methods are qualitatively
consistent in identifying sensor faults, lending credence
to our observations. Our work is a first-step towards
automated on-line fault detection and classification.
I. INTRODUCTION
With the maturation of sensor network software, we
are increasingly seeing longer-term deployments of wire-
less sensor networks in real mode settings. As a result,
research attention is now turning towards drawing mean-
ingful scientific inferences from the collected data [1].
Before sensor networks can become effective replace-
ments for existing scientific instruments, it is important
to ensure the quality of the collected data. Already,
several deployments have observed faulty sensor read-
ings caused by incorrect hardware design or improper
calibration, or by low battery levels [2], [1], [3].
Given these observations, and the realization that it
will be impossible to always deploy a perfectly calibrated
network of sensors, an important research direction for
the future will be automated detection, classification, and
root-cause analysis of sensor faults, as well as techniques
that can automatically scrub collected sensor data to
ensure high quality. A first step in this direction is an
understanding of the prevalence of faulty sensor readings
in existing real-world deployments. In this paper, we take
such a step.
We start by focusing on a small set of sensor faults
that have been observed in real deployments: single-
sample spikes sensor readings (we call these SHORT
faults, following [2]), longer duration noisy readings
(NOISE faults), and anomalous constant offset readings
(CONSTANT faults). Given these fault models, our
paper makes the following two contributions.
Detection Methods. We first explore three qualitatively
different techniques for automatically detecting such
faults from a trace of sensor readings. Rule-based meth-
ods leverage domain knowledge to develop heuristic
rules for detecting and identifying faults. Estimation
methods predict “normal” sensor behavior by leveraging
sensor correlation, flagging deviations from the normal
as sensor faults. Finally, learning-based methods are
trained to statistically detect and identify classes of
faults.
By artificially injecting faults of varying intensity into
sensor datasets, we are able to study the detection perfor-
mance of these methods. We find that these methods sit
at different points on the accuracy/robustness spectrum.
While rule-based methods can detect and classify faults,
they can be sensitive to the choice of parameters. By
contrast, the estimation method we study is a bit more
robust to parameter choices but relies on spatial corre-
lations and cannot classify faults. Finally, our learning
method (based on Hidden Markov Models) is cumber-
some, partly because it requires training, but it can fairly
accurately detect and classify faults. Furthermore, at low
fault intensities, these techniques perform qualitatively
differently: the learning method is able to detect more
NOISE faults but with higher false positives, while the
rule-based method detects more SHORT faults, with
the estimation method’s performance being intermediate.
This suggests the use of hybrid detection techniques,
which combine these three methods in ways that can
be used to reduce false positives or false negatives,
whichever is more important for the application.
Evaluation on Real-World Datasets. Armed with this
evaluation, we apply our detection methods (or, in some
cases, a subset thereof) to four real-world data sets. The
longest of our data sets spans almost 100 days, and
the shortest spans one day. We examine the frequency
of occurrence of faults in these real data sets, using
a very simple metric: the fraction of faulty samples
in a sensor trace. We find that faults are relatively
infrequent: often, SHORT faults occur once in about
two days in one of the data sets that we study, and
NOISE faults are even less frequent. We find no spatial
or temporal correlation among faults. However, different
data sets exhibits different levels of faults: for example,
in one month-long dataset we found only six instances of
SHORT faults, while in another 3-month long dataset, we
found several hundred. Finally, we find that our detection
methods incur false positives and false negatives on these
data sets, and hybrid methods are needed to eliminate
one or the other.
Our study informs the research on ensuring data
quality. Even though we find that faults are relatively
rare, they are not negligibly so, and careful attention
needs to be paid to engineering the deployment and to
analyzing the data. Furthermore, our detection methods
could be used as part of an online fault diagnosis system,
i.e., where corrective steps could be taken during the
data collection process based on the diagnostic system’s
results.
II. SENSOR FAULTS
In this section, we visually depict some faults in sensor
readings observed in real datasets. These examples are
drawn from the same real-world datasets that we use to
evaluate prevalence of sensor faults; we describe details
about these datasets later in the paper. These examples
give the reader visual intuition for the kinds of faults that
occur in practice, and motivate the fault models we use
in this paper.
Before we begin, a word about terminology. We use
the term sensor fault loosely. Strictly speaking, what
we call a sensor fault is really a visually or statistically
anomalous reading. In one case, we have been able to
establish that the behavior we identified was indeed a
fault in the design of an analog-to-digital converter. That
said, the kinds of faults we describe below have been
observed by others as well [2], [1], and that leads us to
believe that the readings we identify as faults actually
correspond to hardware-level malfunctions in sensors.
Finally, in this paper we do not attempt to precisely
establish the cause of a fault.
0 0.5 1 1.5 2 2.5 3
x 10
5
0
500
1000
1500
2000
2500
Time (seconds)
Sensor Readings
0 1000 2000 3000 4000 5000 6000 7000 8000 9000
1.5
2
2.5
3
3.5
4
4.5
5
5.5
6
x 10
4
Sensor Readings
Sample Number
(a) Unreliable Readings (b) SHORT
Fig. 1. Errors in sensor readings
0 20 40 60 80 100
3.248
3.25
3.252
3.254
3.256
3.258
3.26
3.262
3.264
3.266
x 10
4
Sample Number
Sensor Readings
0 20 40 60 80 100
3.248
3.25
3.252
3.254
3.256
3.258
3.26
3.262
3.264
3.266
x 10
4
Sample Number
Sensor Readings
(a) Sufficient voltage (b) Low voltage
Fig. 2. Increase in variance
Figure 1.a shows readings from a sensor reporting
chlorophyll concentration measurements from a sensor
network deployment on lake water. Due to faults in
the analog-to-digital converter board the sensor starts
reporting values 4 5 times greater than the actual
chlorophyll concentration. Similarly, in Figure 1.b, one
of the samples reported by a humidity sensor is roughly
3 times the value of the rest of the samples, resulting in
a noticeable spike in the plot. Finally, Figure 2 shows
that the variance of the readings from an accelerometer
attached to a MicaZ mote measuring ambient vibration
increases when the voltage supplied to the accelerometer
becomes low.
The faults in sensor readings shown in these figures
characterize the kind of faults we observed in the four
data sets from wireless sensor network deployments that
we analyze in this paper. We know of two other sensor
network deployments [1], [2] that have observed similar
faults.
2
In this paper, we explore the following three fault
models motivated by these examples (further details are
given in Section IV):
1) SHORT: A sharp change in the measured value
between two successive data points (Figure 1.b).
2) NOISE: The variance of the sensor readings in-
creases. Unlike SHORT faults that affect a single
sample at a time, NOISE faults affect a number of
successive samples (see Figure 2).
3) CONSTANT: The sensor reports a constant value
for a large number of successive samples. The
reported constant value is either very high or very
low compared to the “normal” sensor readings
(Figure 1.a) and uncorrelated to the underlying
physical phenomena.
SHORT and NOISE faults were first identified and
characterized in [2] but only for a single data set.
III. DETECTION METHODS
In this paper, we explore and characterize three qual-
itatively different detection methods – Linear Least-
Squares estimation (LLSE), Hidden Markov Models
(HMM), and a Rule-based method which leverages do-
main knowledge (the nature of faults in sensor readings)
to develop heuristic rules for detecting and identifying
faults. The Rule-based methods analyzed in this paper
were first proposed in [2].
Our motivation for considering three qualitatively dif-
ferent detection methods is as follows. As one might
expect, and as we shall see later in the paper, no single
method is perfect for detecting the kinds of faults we
consider in this paper. Intuitively, then, it makes sense to
explore the space of detection techniques to understand
the trade-offs in detection accuracy versus the robustness
to parameter choices and other design considerations.
This is what we have attempted to do in a limited
way, and our choice of qualitatively different approaches
exposes differences in the trade offs.
A. Rule-based (Heuristic) Methods
Our first class of detection methods uses two intuitive
heuristics for detecting and identifying the fault types
described in Section (II).
NOISE Rule: Compute the standard deviation of sample
readings within a window N. If it is a above a certain
threshold, the samples are corrupted by the NOISE fault.
To detect CONSTANT faults, we use a slightly modified
NOISE rule where we classify the samples as corrupted
by CONSTANT faults if the standard deviation is zero.
The window size N can be in terms of time or number of
samples. Clearly, the performance of this rule depends
on the window size N and the threshold.
SHORT Rule: Compute the rate of change of the physical
phenomenon being sensed (temperature, humidity etc.)
between two successive samples. If the rate of change is
above a threshold, it is an instance of a SHORT fault.
For well-understood physical phenomena like temper-
ature, humidity etc., the thresholds for the NOISE and
SHORT rules can be set based on domain knowledge.
For example, [2] uses feedback from domain scientists
to set a threshold on the rate of change of chemical
concentration in soil.
For automated threshold selection, [2] proposes the
following technique:
Histogram method: Plot the histogram of the stan-
dard deviations or the rate of change observed for
the entire time series (of sensor readings) being
analyzed. If the histogram is multi-modal, select one
of the modes as threshold.
For the NOISE rule, the Histogram method for auto-
mated threshold selection will be most effective when,
in the absence of faults, the histogram of standard
deviations is uni-modal and sensor faults affect the mea-
sured values in such a way that the histogram becomes
bi-modal. However, this approach is sensitive to the
choice of N; the number of modes in the histogram of
standard deviations depends on N. Figure 3 shows the
effect of N on the number of modes in the histogram
computed for sensor measurements taken from a real-
world deployment. The measurements do not contain a
sensor fault, but choosing N = 1000 gives a multi-modal
histogram, and would result in false positives.
0 5 10 15 20 25 30
0
10
20
30
40
50
60
70
80
90
Std. Deviation
Frequency
5 10 15 20 25 30 35
0
1
2
3
4
5
6
7
8
9
Std. Deviation
Frequency
(a) N = 100 (b) N = 1000
Fig. 3. Histogram Shape
Selecting the right parameters for the rule-based meth-
ods requires a good understanding of the reasonable
sensor readings. In particular, a domain expert would
have to suggest that N = 1000 in our previous example
was an unrealistic choice of parameter.
3
B. An Estimation-Based Method
Is there a method that perhaps requires less domain
knowledge in setting parameters? For physical phenom-
ena like ambient temperature, light etc. that exhibit
a diurnal pattern, statistical correlation between sensor
measurements can be exploited to generate estimates for
the sensed phenomenon based on the measurements of
the same phenomenon at other sensors. Regardless of the
cause of the statistical correlation, we can exploit the ob-
served correlation in a reasonably dense sensor network
deployment to detect anomalous sensor readings.
More concretely, suppose the temperature values re-
ported by sensors s
1
and s
2
are correlated. Let
ˆ
t
1
(t
2
) be
the estimate of temperature at s
1
based on the tempera-
ture t
2
reported by s
2
. Let t
1
be the actual temperature
value reported by s
1
. If jt
1
ˆ
t
1
j > δ, for some threshold
δ, we classify the reported reading t
1
as erroneous. If the
estimation technique is robust, in the absence of faults,
the estimate error (jt
1
ˆ
t
1
j) would be small whereas a
fault of the type SHORT or CONSTANT would cause the
reported value to differ significantly from the estimate.
In this paper we consider the Linear Least-Squares
Estimation (LLSE) method as the estimation technique
of choice. In scalar form, the LLSE equation is
ˆ
t
1
(t
2
) = m
t
1
+
λ
t
1
t
2
λ
t
2
(t
2
m
t
2
) (1)
where m
t
1
and m
t
2
are the average temperature at s
1
and s
2
, respectively. λ
t
1
t
2
is the covariance between the
measurements reported by s
1
and s
2
and λ
t
2
is the
variance of the measurements reported by s
2
.
In the real-world, the value t
2
might itself be faulty.
In such situations, we can estimate
ˆ
t
1
based on measure-
ments at more than one sensor using the LLSE equations
for the vector case (a straight forward and well-known
generalization of the scalar form equation).
In general, the apriori probability distribution for the
measurements reported by sensors may not be available
to compute m
t
1
, m
t
2
, λ
t
1
t
2
and λ
t
2
. In applying the LLSE
method to the real-world data sets, we divide the data
set into training and test. We compute m
t
1
, m
t
2
, λ
t
1
t
2
and λ
t
2
based on the training data set and use them to
detect faulty samples in the test data set. This involves
an assumption that, in the absence of faults or external
perturbations, the physical phenomenon being sensed
does not change dramatically between the time when
the training and test samples were collected. We found
this assumption to hold for many of the data sets we
analyzed.
Finally, we set the threshold δ used for detecting
faulty samples based on the LLSE estimation error for
the training data set. We use the following two heuristics
for determining δ:
Maximum Error: If the training data has no faulty
samples, we can set δ to be the maximum estima-
tion error for the training data set, i.e. δ = maxfjt
1
ˆ
t
1
j : t
1
2 T Sg where T S is set of all samples in the
training data set.
Confidence Limit: In practice, the training data set
will have faults. If we can reasonably estimate, e.g.,
from historical information, the fraction of faulty
samples in the training data set, (say) p%, we can
set δ to be the upper confidence limit of the (1
p)% confidence interval for the LLSE estimation
errors on the training data set.
0 2000 4000 6000 8000 10000
20
40
60
80
100
120
140
160
180
Sample Number
Sensor Reading
Actual Readings
LLSE estimate
0 2000 4000 6000 8000 10000
0
10
20
30
40
50
60
70
80
90
Sample Number
Estimation Error
Estimation error
(a) LLSE Estimate (b) Estimation Error
Fig. 4. LLSE on NAMOS data set
Figure 4 is a visual demonstration of the feasibility of
LLSE, derived from one of our data sets. It compares
the LLSE estimate of sensor readings at a single node
A based on the measurements reported by a neighboring
node B, with the actual readings at A. The horizontal
line in Figure 4(b) represents the threshold δ using the
Maximum Error criterion. The actual sensor data had
no SHORT faults and the LLSE method classified only
one out of 11;678 samples as faulty. We return to a more
detailed evaluation of LLSE-based fault detection in later
sections.
Finally, although we have described an estimation-
based method that leverages spatial correlation, this
method can equally well be applied by only leveraging
temporal correlation at a single node. By extracting cor-
relations induced by diurnal variations at a node, it might
be possible to estimate readings, and thereby detect
faults, at that same node. We have left an exploration
of this direction for future work.
4
C. A Learning-based Method
For phenomena that may not be spatio-temporally
correlated, a learning-based method might be more
appropriate. For example, if the pattern of “normal”
sensor readings and the effect of sensor faults on the
reported readings for a sensor measuring a physical phe-
nomenon is well understood, then we can use a Hidden
Markov Model (HMM) [4] to construct a model for the
measurements reported by that sensor. The states in an
HMM mirror the characteristics of both the physical
phenomenon being sensed as well as the sensor fault
types. For example, based on our characterization of
faults in Section (II), for a sensor measuring ambient
temperature, we can use a 5 state HMM with the states
corresponding to day, night, SHORT faults, NOISE faults
and CONSTANT faults. Such an HMM can capture
not only the diurnal pattern of temperature but also the
distinct patterns in the reported values in the presence of
faults.
For brevity, we omit a formal definition of HMMs; the
interested reader is referred to [4]. Intuitively, however,
HMMs work as follows. Given a set of states and a
training data set annotated with the state to which each
sample corresponds, the HMM method derives a Markov
model that associates transition probabilities with those
states. (For example, each sample in a temperature data
set can be annotated with whether it was a sample taken
during the day or during the night or whether it was
a faulty sample of a particular fault type. This would
give us a 5 state HMM, assuming we used our three
fault types.) Then, given a timeseries of readings from
a sensor, we can use the derived model to calculate
the most likely state associated with a sample. If that
state corresponds to a faulty state (NOISE, SHORT,
or CONSTANT in the temperature sensor example),
we classify the sample as corrupted by fault of the
corresponding type. Thus, an HMM-based method is
able to simultaneously detect and classify faults, but
requires an initial annotated data set for training.
D. Hybrid Methods
Finally, observe that we can use combinations of
the Rule-based, LLSE and HMM methods to elimi-
nate/reduce the false positives and negatives. In this
paper, we study two such schemes:
Hybrid(U): Over two (or more) methods, this
method identifies a sample as faulty if at least one
of the methods identifies the sample as faulty. Thus,
Hybrid(U) is intended for reducing false negatives
(it may not eliminate them entirely, since all meth-
ods might incorrectly flag a sample to be faulty).
However, it can suffer from false positives.
Hybrid(I): Over two (or more) methods, this method
identifies a sample as faulty only if both (all) the
methods identify the sample as faulty. Essentially,
we take an intersection over the set of samples
identified as faulty by different methods. Hybrid(I)
is intended for reducing false positives (again, it
may not eliminate them entirely) but suffers from
false negatives.
IV. EVALUATION: INJECTED FAULTS
Before we can evaluate the prevalence of faults in real-
world datasets using the methods discussed in the pre-
vious section, we need to characterize the accuracy and
robustness of these methods. To do this, we artificially
injected faults of the types discussed in Section II into
a real-world dataset. Before injecting faults, we verified
that the real-world sensor dataset did not contain any
faults.
This methodology has two advantages. Firstly, inject-
ing faults into a dataset gives us an accurate “ground
truth” that helps us better understand the performance
of a detection method. Secondly, we are able to control
the intensity of a fault and can thereby explore the
limits of performance of each detection method as well
as comparatively assess different schemes at low fault
intensities. Many of the faults we have observed in
existing real datasets are of relatively high intensity; even
so, we believe it is important to understand behavior
across a range of fault intensities, since it is unclear if
faults in future datasets will continue to be as pronounced
as those in today’s datasets.
Below, we discuss the detection performance of vari-
ous methods for each type of fault. We describe how we
generate faults in the corresponding subsections. We use
three metrics to understand the performance of various
methods: the number of faults detected, false negatives,
and false positives. More specifically, we use the fraction
of samples with faults as our metric, to have a more
uniform depiction of results across the data sets. For the
figures pertaining to this section and Section V, the labels
used for different detection methods are:R: Rule-based,
L: LLSE, H: HMM, U:Hybrid(U) and I: Hybrid(I).
A. SHORT Faults
To inject SHORT faults, we picked a sample i and
replaced the reported value v
i
with ˆ v
i
= v
i
+ f v
i
. The
multiplicative factor f determines the intensity of the
5
R L H U I R L H U I R L H U I R L H U I
0
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
1.8
2
x 10
−3
Fraction of Samples with Faults
Intensity = 1.5, 2, 5, 10
Detected
False Negative
False Positive
Fig. 5. Injected SHORT Faults
SHORT fault. We injected SHORT faults with intensity
f =f1:5;2;5;10g. Injecting SHORT faults in this man-
ner (instead of just adding a constant value) does not
require the knowledge of the range of “normal” sensor
readings.
Figure 5 compares the accuracy of the SHORT rule,
LLSE, HMM, Hybrid(U), and Hybrid(I) for SHORT
faults. The horizontal line in the figure represents the
actual fraction of samples with injected faults. The four
sets of bar plots correspond to increasing intensity of
SHORT faults (left to right).
The SHORT rule and LLSE do not have any false
positives; hence, the Hybrid(I) method exhibits no false
positives. However, for faults with low intensity ( f =
1:5;2), the SHORT rule as well as LLSE have significant
false negatives. The choice of threshold used to detect
a faulty sample governs the trade-off between false
positives and false negatives; reducing the threshold
would reduce the number of false negatives but increase
the number of false positives. For the SHORT rule, the
threshold was selected automatically using the histogram
method and for LLSE the threshold was set using the
Maximum Error criterion.
The HMM method has fewer false negatives compared
to SHORT rule and LLSE but it has false positives for
lowest intensity ( f = 1:5). While training the HMM for
detecting SHORT faults, we observed that if the training
data had sufficient number of SHORT faults (on the order
of 15 faults in 11000 samples), the intensity of the faults
did not affect the performance of HMM.
In these experiments, Hybrid(U) performs like the
method with more detections and Hybrid(I) performs like
the method with less detections (while eliminating the
false positives). However, in general this does not have
to be the case, e.g., in the absence of false positives,
Hybrid(U) could detect more faults than the best of the
R L H U I R L H U I R L H U I
0
0.02
0.04
0.06
0.08
0.1
0.12
0.14
0.16
Fraction of Samples with Faults
Intensity = Low, Medium, High
Detected
False Negative
False Positive
Fig. 6. Injected NOISE Fault: 3000 samples with errors
R L H U I R L H U I R L H U I
0
0.02
0.04
0.06
0.08
0.1
0.12
Fraction of Samples with Faults
Intensity = Low, Medium, High
Detected
False Negative
False Positive
Fig. 7. Injected NOISE Fault: 2000 samples with errors
methods and Hybrid(I) could detect fewer faults than the
worst of the methods (as illustrated on the real data sets
in Section V).
B. NOISE Faults
To inject NOISE faults, we pick a set of successive
samples W and add a random value drawn from a
normal distribution, N(0;σ
2
), to each sample in W. We
R L U I R L H U I R L H U I
0
0.01
0.02
0.03
0.04
0.05
0.06
0.07
0.08
Fraction of Samples with Faults
Intensity = Low, Medium, High
Detected
False Negative
False Positive
Fig. 8. Injected NOISE Fault: 1000 samples with errors
6
R L U I R L U I R L U I
0
0.005
0.01
0.015
0.02
0.025
0.03
0.035
Fraction of Samples with Faults
Intensity = Low, Medium, High
Detected
False Negative
False Positive
Fig. 9. Injected NOISE Fault: 100 samples with errors
vary the intensity of NOISE faults by choosing different
values for σ. The Low, Medium and High intensity of
NOISE faults correspond to 0:5x, 1:5x and 3x increase
in standard deviation of the samples in W. Apart from
varying the intensity of NOISE faults, we also vary its
duration by considering different numbers of samples in
W. The total number of samples in the timeseries into
which we injected NOISE faults was 22;600.
To train the HMM, we injected NOISE faults in the
training data. These faults were of the same duration
and intensity as the faults used for comparing different
methods. There were no NOISE faults in the training
data for LLSE. For the NOISE rule N = 100 was used.
Figures (6, jWj = 3000), (7, jWj = 2000), (8, jWj =
1000) and (9,jWj= 100) show the performance of differ-
ent methods for NOISE faults with varying intensity and
duration. The horizontal line in each figure corresponds
to the number of samples with faults.
Impact of Fault Duration: The impact of fault duration
is most dramatic for the HMM method. For jWj = 100,
regardless of the fault intensity, there were not enough
samples to train the HMM model. Hence, Figure (9) does
not show results for HMM. ForjWj= 1000 and low fault
intensity, we again failed to train the HMM model. This
is not very surprising because for short duration (e.g.,
jWj = 100) or low intensity faults, the data with injected
faults is very similar to data without injected faults. For
faults with medium and high intensity or faults with
sufficiently long duration, e.g.,jWj 1000, performance
of the HMM method is comparable to NOISE rule and
LLSE.
The NOISE rule and LLSE method are more robust to
fault duration than HMM in the sense that we were able
to derive model parameters for those cases. However, for
jWj = 100 and low fault intensity, both the methods fail
to detect any of the samples with faults. The LLSE also
has a significant number of false positives for jWj =
100 and fault intensity 0:5x. The false positives were
eliminated by the Hybrid(I) method.
Impact of Fault Intensity: For medium and high intensity
faults, there are no false negatives for any method. For
low intensity faults, all the methods have significant false
negatives. For fault duration and intensities for which the
HMM training algorithm converged, the HMM method
gives lower false negatives compared to the NOISE rule
and LLSE. However, most of the time the HMM method
gave more false positives. Hybrid methods are able to
reduce the number of false positives and negatives, as
intended. High false negatives for low fault intensity
arise because the data with injected faults is very similar
to data without faults.
V. FAULTS IN REAL-WORLD DATA SETS
We analyze four data sets from real-world deploy-
ments for prevalence of faults in sensor traces. The
sensor traces contain measurements from a variety of
phenomena – temperature, humidity, light, pressure, and
chlorophyll concentration. However, all of these phe-
nomena exhibit a diurnal pattern in the absence of
outside perturbation or sensor faults.
A. Great Duck Island (GDI) data set
We looked at data collected using 30 weather motes
on the Great Duck Island over a period of 3 months [5].
Attached to each mote were temperature, light, and
pressure sensors, and these were sampled once every 5
mins. Of the 30 motes, the data set contained sampled
readings from the entire duration of the deployment for
only 15 motes. In this section, we present our findings
on the prevalence of faults in the readings for these 15
motes.
The predominant fault in the readings was of the type
SHORT. We applied the SHORT rule, the LLSE method
and Hybrid(I) to detect SHORT faults in light, humidity
and pressure sensor readings. Figure 10 shows the overall
prevalence (computed by aggregating results from all
the 15 nodes) of SHORT faults for different sensors in
the GDI data set. The Hybrid (I) technique eliminates
any false positives reported by the SHORT rule or the
LLSE method. The intensity of SHORT faults was high
enough to detect them by visual inspection of the entire
sensor readings timeseries. This ground-truth is included
for reference in the figure under the label V.
It is evident from the figure that SHORT faults are
relatively infrequent. They are most prevalent in the
light sensor (approximately 1 fault every 2000 samples).
7
R L I V R L I V R L I V
0
1
2
3
4
5
6
x 10
−4
Fraction of Samples with Faults
Light (left), Humidity (Center), Pressure (Right)
Detected
False Negative
False Positive
Fig. 10. SHORT Faults in GDI data set
101 103 109 111 116 118 119 121 122 123 124 125 126 129 900
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
x 10
−3
Fraction of Samples with Faults
SHORT Faults
Fig. 11. SHORT Faults: Light Sensor
Figure 11 shows the distribution of SHORT faults in light
sensor readings across various nodes. SHORT faults do
not exhibit any discernible pattern in the prevalence of
these faults across different sensor nodes; the same holds
for other sensors, but we have omitted the corresponding
graphs for brevity. Figure 12 shows the time at which
SHORT faults occur in the readings from the light sensor
at nodes 109 and 116. Here too, we observe no visible
correlation in the times at which the faults occur.
In this data set, NOISE faults were infrequent. Only
two nodes had NOISE faults with a duration of about
100 samples. The NOISE rule detected it, but the LLSE
method failed primarily because its parameters had been
optimized for SHORT faults.
B. INTEL Lab, Berkeley data set
54 Mica2Dot motes with temperature, humidity and
light sensors were deployed in the Intel Berkeley Re-
search Lab between February 28th and April 5th,
2004 [6]. In this paper, we present the results on the
prevalence of faults in the temperature readings (sampled
on average once every 30 seconds).
0 1 2 3 4 5 6 7
x 10
8
0
1
Time (seconds)
SHORT faults at 109
SHORT faults at 116
Fig. 12. Independent SHORT faults
0 0.5 1 1.5 2 2.5 3
x 10
6
0
20
40
60
80
100
120
140
Time (seconds)
Temperature
Fig. 13. Intel data set: NOISE faults
This dataset exhibited a combination of NOISE and
CONSTANT faults. Each sensor also reported the volt-
age values along with the samples. Inspection of the
voltage values reported showed that the faulty samples
were well correlated with the last few days of the
deployment when the lithium ion cells supplying power
to the motes were unable to supply the voltage required
by the sensors for correct operation.
R H U I
0
0.05
0.1
0.15
0.2
0.25
Fraction of Samples with Faults
Detected
False Negative
False Positive
Fig. 14. Intel data set: Prevalence of Noise faults
8
The samples with NOISE faults were contiguous in
time (Figure 13) and both the NOISE rule and a simple
two-state HMM model identified most of these samples.
Figure 14 shows the fraction of the total temperature
samples (collected by all the motes) with faults. Both the
NOISE rule and HMM have some false negatives while
the HMM also has some false positives. For this data set,
we could eliminate all the false positives using Hybrid(I)
with NOISE rule and HMM. However, combining the
NOISE rule and HMM for Hybrid(I) incurred more false
negatives.
Interestingly, for this dataset, we could not apply the
LLSE method to detect NOISE faults. NOISE faults
across various nodes were temporally correlated, since
all the nodes ran out of battery power at approximately
the same time. This breaks an important assumption
underlying the LLSE technique, that faults at different
sensors are uncorrelated.
Finally, in this data set, there were surprisingly few
instances of SHORT faults. A total of 6 faults were ob-
served for the entire duration of the experiment (Table I).
All of these faults were detected by the HMM method,
LLSE, and the SHORT rule.
ID # Faults Total # Samples
2 1 46915
4 1 43793
14 1 31804
16 2 34600
17 1 33786
TABLE I
INTEL LAB: SHORT FAULTS, TEMPERATURE
C. NAMOS data set
Nine buoys with temperature and chlorophyll con-
centration sensors (fluorimeters) were deployed in Lake
Fulmor, James Reserve for over 24 hours in August
2006 [7]. Each sensor was sampled every 10 seconds.
We analyzed the measurements from chlorophyll sensors
for the prevalence of faults.
The predominant fault was a combination of NOISE
and CONSTANT caused by hardware faults in the ADC
(Analog-to-Digital Converter) board. Figure 1.a shows
the measurements reported by buoy 103. We applied the
NOISE Rule to detect samples with errors. Figure 15
shows the fraction of samples corrupted by faults. The
sensors at 4 buoys was affected by the ADC board fault
and in the worst case, at buoy 103, 35% of the reported
values were erroneous. We could not apply LLSE and
101 102 103 106 107 109 110 112 114
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
Fraction of Samples with Faults
Node ID
Detected
False Negative
False Positive
Fig. 15. NAMOS data set (August): NOISE/CONSTANT faults
Station ID # Faults Total # samples
3 86 45396
21 1 84818
41 1 60680
TABLE II
SHORT FAULTS IN SENSORSCOPE DATA SET
HMM method because there was not enough data to train
the models (data was collected for 24 hours only).
D. SensorScope data set
The SensorScope project is an ongoing outdoor sensor
network deployment consisting of weather-stations with
sensors for sensing several environmental quantities such
as temperature, humidity, solar radiation, soil moisture,
and so on [8]. We analyzed the temperature measure-
ments reported once every 15 seconds during November,
2006 by 31 weather stations deployed on a university
campus.
We found the occurrence of faults to be the lowest for
this data set. The data from only 3 out of the 31 stations
that we looked at contained instances of SHORT faults.
We identified the faulty samples using SHORT rule and
the LLSE method. Neither of the methods generated any
false positives. We did not find any instance of NOISE
and CONSTANT faults. Table (II) presents our findings
from the SensorScope data set.
VI. RELATED WORK
Several papers on real-world sensor network deploy-
ments [1], [5], [2], [3] present results on meaningful
inferences drawn from the collected data. However, to
the best of our knowledge, only [2], [3], [1] do a detailed
analysis of the collected data. The aim of [2] is to
do root cause analysis using Rule-based methods for
online detection and remediation of sensor faults for a
specific type of sensor network monitoring the presence
9
of arsenic in groundwater. The SHORT and NOISE
detection rules analyzed in this paper were proposed
in [2]. Werner et al. [3] compares the fidelity of data
collected using a sensor network monitoring volcanic
activity to the data collected using traditional equipment
used for monitoring volcanoes. Finally, Tolle et al. [1]
examine spatio-temporal patterns in microclimate on a
single redwood tree. While these publications thoroughly
analyze their respective datasets, examining fault preva-
lence was not an explicit goal of those pieces of work.
Our work presents a thorough analysis of four different
real-world data sets. Looking at different data sets also
enables us to characterize the accuracy and robustness
of three qualitatively different methods.
A variety of models for sensor network data have
previously been proposed. Guestrin et al. [9] use a
distributed regression technique to optimally fit a global
function to the local measurements at each sensor. Sev-
eral proposed schemes for efficient querying in sensor
networks [10], [11] use probabilistic and time series
models for sensor data to reduce the communication
overhead between the sink and the sensors. While de-
tection and analysis of sensor faults has not been a
focus of these works, the proposed models provide a rich
set of techniques that can potentially be leveraged for
automated detection and classification of sensor faults.
VII. SUMMARY, CONCLUSIONS, AND FUTURE WORK
In this paper, we focused on a simple question: How
often are sensor faults observed in real deployments? To
answer this question, we first explored and characterized
three qualitatively different classes of fault detection
methods and then applied them to real world data sets.
The analysis of injected faults in Section IV demon-
strated that the three methods sit at different points on
the accuracy/robustness spectrum. For medium and high
fault intensities (e.g., f 5 in case of SHORT faults),
the rule-based methods worked very well. However, at
low fault intensities, the rule-based method worked better
for SHORT faults than NOISE faults primarily because
choosing good parameters for NOISE rule is much
more challenging than for SHORT rule. With sufficient
training data (longer fault duration) the learning-based
HMM method was able to detect more NOISE faults
than rule-based and LLSE methods but with higher
false positives. The LLSE method was more robust to
parameter choices than rule-based methods but it does
not always outperform the rule-based methods.
As illustrated in Section V, faults in real data sets
were relatively infrequent but of high intensity (and long
duration for NOISE and CONSTANT faults): e.g., in the
GDI data set SHORT faults occurred once in two days ,
and NOISE faults were even less frequent. We found no
spatial or temporal correlation among faults. However,
different data sets exhibited different levels of faults: e.g.,
in one month-long dataset we found only six instances
of SHORT faults, while in another 3-month long dataset,
we found several hundred. Finally, we found that our
detection methods incurred false positives and false
negatives on these data sets, and hybrid methods were
needed to eliminate one or the other.
Overall, we believe that our work opens up new
research directions in automated high-confidence fault
detection, classification, data rectification, and so on.
More sophisticated statistical and learning techniques
than we have presented can be brought to bear on this
crucial area.
REFERENCES
[1] G. Tolle, J. Polastre, R. Szewczyk, D. Culler, N. Turner, K. Tu,
S. Burgess, T. Dawson, P. Buonadonna, D. Gay, and W. Hong,
“A Macroscope in the Redwoods,” in SenSys ’05: Proceedings
of the 2nd international conference on Embedded networked
sensor systems. New York, NY , USA: ACM Press, 2005, pp.
51–63.
[2] N. Ramanathan, L. Balzano, M. Burt, D. Estrin, E. Kohler,
T. Harmon, C. Harvey, J. Jay, S. Rothenberg, and M. Srivastava,
“Rapid Deployment with Confidence: Calibration and Fault
Detection in Environmental Sensor Networks,” CENS, Tech.
Rep. 62, April 2006.
[3] G. Werner-Allen, K. Lorincz, J. Johnson, J. Lees, and M. Welsh,
“Fidelity and Yield in a V olcano Monitoring Sensor Network,”
in Proceedings of the 7th USENIX Symposium on Operating
Systems Design and Implementation (OSDI 2006), 2006.
[4] L. Rabiner, “A tutorial on Hidden Markov Models and selected
applications in speech recognition,” Proceedings of IEEE, vol.
77(2), pp. 257–286, 1989.
[5] A. Mainwaring, J. Polastre, R. Szewczyk, and D. C. J. Ander-
son, “Wireless Sensor Networks for Habitat Monitoring ,” in
the ACM International Workshop on Wireless Sensor Networks
and Applications. WSNA ’02, 2002.
[6] “The Intel Lab, Berkeley data set,” http://berkeley.intel-
research.net/labdata/.
[7] “NAMOS: Networked Aquatic Microbial Observing System,”
http://robotics.usc.edu/ namos.
[8] “The SensorScope project,” http://sensorscope.epfl.ch.
[9] C. Guestrin, P. Bodik, R. Thibaux, M. Paskin, and S. Madden,
“Distributed Regression: an Efficient Framework for Modeling
Sensor Network Data,” in Proceedings of the 3rd Intenational
Conference on Information Processing in Sensor Networks
(IPSN’04), 2004.
[10] A. Deshpande, C. Guestrin, S. Madden, J. M. Hellerstein, and
W. Hong, “Model-Drive Data Acquisition in Sensor Networks,”
in Proceedings of the 30th VLDB Conference, 2004.
[11] D. Tulone and S. Madden, “An Energy-efficient Querying
Framework In Sensor Networks for Detecting Node Similar-
ities,” in ACM/IEEE Symposium on Modeling, Analysis and
Simulation of Wireless and Mobile Systems (MSWIM), 2006.
10
Linked assets
Computer Science Technical Report Archive
Conceptually similar
PDF
USC Computer Science Technical Reports, no. 906 (2009)
PDF
USC Computer Science Technical Reports, no. 914 (2010)
PDF
USC Computer Science Technical Reports, no. 905 (2009)
PDF
USC Computer Science Technical Reports, no. 923 (2012)
PDF
USC Computer Science Technical Reports, no. 930 (2012)
PDF
USC Computer Science Technical Reports, no. 692 (1999)
PDF
USC Computer Science Technical Reports, no. 773 (2002)
PDF
USC Computer Science Technical Reports, no. 774 (2002)
PDF
USC Computer Science Technical Reports, no. 706 (1999)
PDF
USC Computer Science Technical Reports, no. 723 (2000)
PDF
USC Computer Science Technical Reports, no. 920 (2011)
PDF
USC Computer Science Technical Reports, no. 771 (2002)
PDF
USC Computer Science Technical Reports, no. 732 (2000)
PDF
USC Computer Science Technical Reports, no. 745 (2001)
PDF
USC Computer Science Technical Reports, no. 839 (2004)
PDF
USC Computer Science Technical Reports, no. 852 (2005)
PDF
USC Computer Science Technical Reports, no. 834 (2004)
PDF
USC Computer Science Technical Reports, no. 841 (2005)
PDF
USC Computer Science Technical Reports, no. 751 (2001)
PDF
USC Computer Science Technical Reports, no. 750 (2001)
Description
Abhishek Sharma, Leana Golubchik, Ramesh Govindan. "On the prevalence of sensor faults in real world deployments." Computer Science Technical Reports (Los Angeles, California, USA: University of Southern California. Department of Computer Science) no. 888 (2007).
Asset Metadata
Creator
Golubchik, Leana
(author),
Govindan, Ramesh
(author),
Sharma, Abhishek
(author)
Core Title
USC Computer Science Technical Reports, no. 888 (2007)
Alternative Title
On the prevalence of sensor faults in real world deployments (
title
)
Publisher
Department of Computer Science,USC Viterbi School of Engineering, University of Southern California, 3650 McClintock Avenue, Los Angeles, California, 90089, USA
(publisher)
Tag
OAI-PMH Harvest
Format
10 pages
(extent),
technical reports
(aat)
Language
English
Unique identifier
UC16269744
Identifier
07-888 On the Prevalence of Sensor Faults in Real World Deployments (filename)
Legacy Identifier
usc-cstr-07-888
Format
10 pages (extent),technical reports (aat)
Rights
Department of Computer Science (University of Southern California) and the author(s).
Internet Media Type
application/pdf
Copyright
In copyright - Non-commercial use permitted (https://rightsstatements.org/vocab/InC-NC/1.0/
Source
20180426-rozan-cstechreports-shoaf
(batch),
Computer Science Technical Report Archive
(collection),
University of Southern California. Department of Computer Science. Technical Reports
(series)
Access Conditions
The author(s) retain rights to their work according to U.S. copyright law. Electronic access is being provided by the USC Libraries, but does not grant the reader permission to use the work if the desired use is covered by copyright. It is the author, as rights holder, who must provide use permission if such use is covered by copyright.
Repository Name
USC Viterbi School of Engineering Department of Computer Science
Repository Location
Department of Computer Science. USC Viterbi School of Engineering. Los Angeles\, CA\, 90089
Repository Email
csdept@usc.edu
Inherited Values
Title
Computer Science Technical Report Archive
Description
Archive of computer science technical reports published by the USC Department of Computer Science from 1991 - 2017.
Coverage Temporal
1991/2017
Repository Email
csdept@usc.edu
Repository Name
USC Viterbi School of Engineering Department of Computer Science
Repository Location
Department of Computer Science. USC Viterbi School of Engineering. Los Angeles\, CA\, 90089
Publisher
Department of Computer Science,USC Viterbi School of Engineering, University of Southern California, 3650 McClintock Avenue, Los Angeles, California, 90089, USA
(publisher)
Copyright
In copyright - Non-commercial use permitted (https://rightsstatements.org/vocab/InC-NC/1.0/