University of Southern California Dissertations and Theses

Smart monitoring and autonomous situation classification of humans and machines

(USC Thesis Other)

Smart monitoring and autonomous situation classification of humans and machines

PDF

Download a page range

Download transcript

Copy asset link

Request this asset

Transcript (if available)

Content
Smart Monitoring and Autonomous Situation Classification
of Humans and Machines

by

Jens Windau

A Dissertation Presented to the
FACULTY OF THE USC GRADUATE SCHOOL
UNIVERSITY OF SOUTHERN CALIFORNIA

In Partial Fulfillment of the
Requirements for the Degree
DOCTOR OF PHILOSOPHY
(COMPUTER SCIENCE)

August 2019
Copyright 2019 Jens Windau
2

Contents

Abstract ....................................................................................................................................... 5
1. Introduction ........................................................................................................................ 6
1.1 Inertial Measurement Units ........................................................................................ 6
1.2 Measuring human motions with IMUs ....................................................................... 7
1.3 Measuring machine motions with IMUs ..................................................................... 8
2. Situation awareness via sensor-equipped eyeglasses ........................................................ 9
2.1 Abstract ....................................................................................................................... 9
2.2 Introduction .............................................................................................................. 10
2.3 The SAS Approach ..................................................................................................... 13
2.3.1 Placement and design of the sensor cluster ..................................................... 13
2.3.2 IMU data transformation into stable coordinate system ............................ 14
2.3.3 Identification of critical features ................................................................... 16
2.3.4 Creating an extendable database .................................................................. 16
2.3.5 Classification technique for activities ............................................................ 18
2.4 The Measurement in Detail ...................................................................................... 18
2.4.1 Overview ........................................................................................................... 18
2.4.2 Steps in Detail ................................................................................................... 19
2.4.2.1 Transformation (only for IMU data) ............................................................. 19
2.4.2.2 Feature Extraction for IMU data ................................................................... 21
2.4.2.3 Feature Extraction for video data ................................................................. 23
2.4.2.4 Classification ................................................................................................. 23
2.5 Test Results ............................................................................................................... 24
2.5.1 Performance tests of Naive Bayes and SVM ..................................................... 24
2.5.2 Full system testing (including HMM) ................................................................ 26
2.6 Discussion and Conclusion ........................................................................................ 28
3. Walking compass with head-mounted IMU sensor .......................................................... 29
3.1 Abstract ..................................................................................................................... 29
3.2 Introduction .............................................................................................................. 30
3.3 Analysis of IMU Data during walking activity ............................................................ 33
3.4 The Measurement Detail .......................................................................................... 35
3.4.1 Overview ........................................................................................................... 35
3

3.4.2 Steps in Detail ................................................................................................... 36
3.4.2.1 Step 1: Compensation for Roll and Pitch Motions ........................................ 36
3.4.2.2 Step 2: Walking Pattern Analysis .................................................................. 38
3.4.2.3 Step 3: Fusion Algorithm ............................................................................... 41
3.5 Test Results ............................................................................................................... 43
3.5.1 Performance Tests of HOS ................................................................................ 43
3.5.2 Basic Case Testing ............................................................................................. 44
3.5.3 Mixed Case Testing ........................................................................................... 47
3.6 Discussion and Conclusion ........................................................................................ 48
4. Inertial Machine Monitoring System for automated failure detection ............................ 49
4.1 Abstract ..................................................................................................................... 49
4.2 Introduction .............................................................................................................. 50
4.3 The IMMS Approach ................................................................................................. 53
4.3.1 Robust Hardware Design of the sensor network .............................................. 53
4.3.2 Communication Protocol .................................................................................. 54
4.3.3 Identification of critical features ....................................................................... 55
4.3.4 Creating an extendable database ..................................................................... 56
4.3.5 Classification technique for machine failure .................................................... 57
4.3.6 Optimization Strategy for sensors .................................................................... 57
4.4 The Measurement in Detail ...................................................................................... 57
4.4.1 Overview ........................................................................................................... 57
4.4.2 Steps in Detail ................................................................................................... 58
4.4.2.1 Collection of IMU data .................................................................................. 58
4.4.2.2 Feature Extraction ......................................................................................... 59
4.4.2.3 Classification ................................................................................................. 59
4.5 Test Results ............................................................................................................... 60
4.5.1 Test Machine ..................................................................................................... 60
4.5.2 Test Program ..................................................................................................... 63
4.5.3 Performance Tests ............................................................................................ 65
4.5.3.1 Classification Test (Statistical Features) ........................................................ 65
4.5.3.2 Classification Test (FFT Features) .................................................................. 66
4.5.3.3 Optimization Strategy ................................................................................... 68
4.6 Conclusion ................................................................................................................. 70
4

5. Inertial-based Motion Capturing and Smart Training System .......................................... 71
5.1 Abstract ..................................................................................................................... 71
5.2 Introduction .............................................................................................................. 72
5.3 The STS Approach ..................................................................................................... 76
5.3.1 Hardware Design of the IMU Body Sensor Network ........................................ 76
5.3.2 Communication Protocol ................................................................................ 78
5.3.3 Optimized Placement of sensors .................................................................... 78
5.3.4 Segmentation of critical swing phases .......................................................... 79
5.3.5 Identification of critical features ................................................................... 80
5.3.6 Effective training of virtual coach with small datasets ................................ 81
5.3.7 Audio-based feedback system ........................................................................ 82
5.4 The Measurement in Detail ...................................................................................... 83
5.4.1 Overview ........................................................................................................... 83
5.4.2 Steps in Detail ................................................................................................... 84
5.4.2.1 Collection of IMU data .................................................................................. 84
5.4.2.2 Segmentation of Swing Motion .................................................................... 85
5.4.2.3 Feature Extraction ......................................................................................... 86
5.4.2.4 Optimal Motion Feature Estimator ............................................................... 87
5.4.2.5 Critical Body Parts Classifier .......................................................................... 88
5.4.2.6 Feedback Creator and Trigger ....................................................................... 90
5.5 Test Results ............................................................................................................... 92
5.5.1 Test Setup ......................................................................................................... 92
5.5.2 Dataset and T-Tests ........................................................................................... 93
5.5.2.1 Test Details Overview ................................................................................... 93
5.5.2.2 The overall 2-sample t-test ........................................................................... 93
5.5.2.3 The individual 2-sample t-test ...................................................................... 95
5.5.3 Learning analysis ............................................................................................... 96
5.6 Conclusion and Future Work .................................................................................... 99
6. Discussion........................................................................................................................ 100
7. Acknowledgements ......................................................................................................... 103
8. References ...................................................................................................................... 104

5

Abstract

Emerging wearable and cloud-connected sensor technologies offer new sensor placement
options on the human body and machines. This opens new opportunities to explore cyber
robotics algorithms (sensors and human motor plant) and smart manufacturing algorithms
(sensors and manufacturing equipment). These algorithms process motion sensor data and
provide situation awareness for a wide range of applications. Smart management and training
systems assist humans in day-to-day living routines, healthcare and sports. Machines benefit
from smart monitoring in manufacturing, retail machinery, transportation, and construction
safety. During my PhD Research, I have developed several approaches for motion analysis and
classification. (1) A situation awareness system (SAS) for head-mounted smartphones to
respond to user activities (e.g., disable incoming phone calls in elevators, activate video
recording while car driving), (2) a filter for head-mounted sensors (HOS) to allow full-body
motion capturing by removing interfering head-motions, (3) an Inertial Machine Monitoring
System (IMMS) to detect equipment failure or degraded states of a 3D-Printer, and (4) a “Smart
Teaching System” (STS) for targeted motion feedback to refine physical tasks. To capture real-
world sensor data, we designed hardware prototypes or used state-of-the-art wearable
technology. We developed novel sensor fusion algorithms, implemented feature extraction
methods based on gist, statistics, physics, frequency diagrams and validated classifiers: SAS
achieved high accuracy (81.5%) when distinguishing between 20 real-world activities. HOS
reduced the positional error of a traveled distance below 2.5 % with head-mounted sensors
for pedestrian dead reckoning applications. IMMS yielded 11-way classification accuracy over
99% when distinguishing between normal operation vs. 10 types of real-world abnormal
equipment behavior. STS demonstrated that combining motion sensors and provide targeted
feedback yield significantly improved golf swing training (3.7x increased performance score).
6

1. Introduction
1.1 Inertial Measurement Units
Inertial measurement units (IMUs) are sensors that have become a crucial part of many
electronic devices ranging from consumer electronics to industrial applications. IMUs combine
accelerometers, gyroscopes, and sometimes also magnetometers in a single sensor unit. These
sensor clusters measure acceleration, angular velocity and magnetic field, typically along 3
axes. Common applications for IMUs can be particularly found in robotics: Aerial, ground, and
underwater vehicles, often unmanned, are using IMUs to improve navigation, maneuvering,
or complement other sensor systems e.g. GPS or camera. A strong advantage of IMUs is their
robust performance. Accelerometers and gyroscopes work almost independently from
environmental conditions and only need to be compensated for temperature-related
measuring errors. When calculating orientation angles, possible drifting errors can be well
compensated by implementing fusion algorithms (e.g. Extended Kalman Filter). IMUs are often
used in combination with other sensor systems that experience common measurement
problems, e.g., video camera data is suffering from bad lighting condition during navigation or
GPS signals cannot be received in tunnels or underground. With the miniaturization of IMUs
by using microelectromechanical systems (MEMS), IMUs have become widely available in
fields of consumer electronics (e.g. integrated in smartphones, image stabilization systems in
cameras, in the entertainment industry in game-controllers, or transportation vehicles such as
the Segway), automotive and manufacturing industry (airbags), industry (e.g. robot
monitoring). While IMU hardware continues to improve in accuracy, size, and production cost,
research has also been focusing on the data processing of IMUs. In 2013, Google launched the
ActivityRecognitionAPI for developers of their Android smartphone platform. Until today,
Android systems can automatically detect 5 activities from accelerometer and gyroscope data:
7

Activities of smartphone users can be classified as running, walking, driving, standing or biking
[1]. Electronic component manufacturers also started to integrate processors into the IMUs
for on-board data processing. One of the latest state-of-the-art IMUs is the ICM-20648, a 3-
axis gyroscope and 3-axis accelerometer. It has a Digital Motion Processor integrated that
locally performs step counts and basic activity classification. With a footprint of 3 x 3 x 0.9 mm
it distinguishes between 4 activities (running, walking, biking, still) [2]. With the strong
development progress of IMUs, the application field of this hardware has offered new research
opportunities. During my PhD, I focused on developing novel algorithmic approaches on how
to efficiently use IMU data collected from either a single sensor or networks of up to 36 IMUs
attached to humans or machines.

1.2 Measuring human motions with IMUs
There are two main sensor solutions on how to capture human motions: (1) External sensor
systems (e.g. camera-based systems) record motions (often by detecting markers on the
human body) or (2) On-body sensors attached to various body locations (e.g. IMU sensors)
record movements and orientation changes. IMU sensors have the advantage of eliminating
occlusion problems, operating independently from lighting conditions, and track motions in
high accuracy and sampling rates in the range of high-speed cameras [2]. In addition, IMU
sensors are not limited to a fixed location as cameras do, and allow flexible, location-
independent monitoring. This allows continuous human body monitoring, particularly when
humans frequently change locations. This provides new opportunities for new applications in
recovery health care (e.g., stroke rehabilitation), smart assistance (e.g., virtual fitness coach to
refine workout routines), or personal safety systems (e.g., monitoring elderly), performance
measurement (e.g., determining human concentration level at work or while driving) and
8

automated managing systems (e.g., auto-adjusting settings of smartphones based on
environmental conditions). Automated managing systems are particularly dependent on
situation awareness information, because of the limited options for user-input (e.g. wearables
such as smart watches or google glass often are equipped with only a few buttons or a limited
touchscreen). Because of the importance and application variety of body sensors and body
sensor networks, three of my PhD research projects focus on analyzing human motion data for
situation awareness and smart training.

1.3 Measuring machine motions with IMUs
Various types of machines benefit from situation awareness. With an increasing degree in
automation in various industries, smart monitoring is particularly crucial for predictive failure
systems to prevent costly machine down-times. These industries are ranging from
manufacturing floors (e.g., recognizing degraded states of assembly robots, CNC and Pick-and-
Place Machine), Transportation (e.g., vehicle monitoring, condition-based maintenance of
engines ranging from scooters to jumbo jets), Construction (e.g., Surveillance of swinging
behavior of Bridges, Utility Poles, and Buildings after earthquakes or similar natural disasters),
Offices (e.g., elevator surveillance), Retail (e.g., commercial washing machines), or high-end
consumer products (e.g., espresso machine, 3D printer). When monitoring a machine, IMU
sensors are often used in networks to monitor multiple independently moving components of
the machine. The extendable network design enables the large range and types of machines
that can be monitored. In addition, IMU sensors need to simply be attached to the machines,
which allows a non-invasive hardware design and a quick machine sensor installation.
In the following chapters, I will describe four research contributions that have been published
(IROS 2013, ICRA 2016, ICRA 2018) or recently submitted (IROS 2019).
9

2. Situation awareness via sensor-equipped eyeglasses
2.1 Abstract
New smartphone technologies are emerging which combine head-mounted displays (HMD)
with standard functions such as receiving phone calls, emails, and helping with navigation. This
opens new opportunities to explore cyber robotics algorithms (robotics sensors and human
motor plant). To make these devices more adaptive to the environmental conditions, user
behavior, and user preferences, it is important to allow the sensor-equipped devices to
efficiently adapt and respond to user activities (e.g., disable incoming phone calls in an
elevator, activate video recording while car driving). This paper hence presents a situation
awareness system (SAS) for head-mounted smartphones. After collecting data from inertial
sensors (accelerometers, gyroscopes), and video data (camera), SAS performs activity
classification in three steps. Step 1 transforms inertial sensor data into a head-orientation-
independent and stable normalized coordinate system. Step 2 extracts critical features
(statistical, physical, GIST). Step 3 classifies activities (Naive Bayes classifier), distinguishes
between environments (Support Vector Machine), and finally combines both results (Hidden
Markov Model) for further improvement. SAS has been implemented on a sensor-equipped
eyeglasses prototype and achieved high accuracy (81.5%) when distinguishing between 20
real-world activities.

Video Reference: https://www.youtube.com/watch?v=rZ1ZIIHUU2M
Published in: 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

10

2.2 Introduction
Head mounted displays (HMDs) embedded in eyeglasses are the next innovation along the
path of communication techniques. Such devices are hand-free systems. Although this is not a
new idea, currently released and commercially available products (such as the Project Glass by
Google) show the immense potential of this technology. They function as stand-alone
computers; their light glass frame is equipped with a variety of sensors; a projector displays
images and information onto the eye. While wearing these eyeglasses, the user is continuously
exposed to the displayed information. To avoid bombarding a user with unwanted clutter, a
management system is required. This management system identifies important messages
(e.g., phone calls, email popups) that are relevant to a user's situation. To evaluate the
importance of a message, however, knowledge of the user's activity is essential. Thus the
challenge of this paper is to present an effective approach to classify activities from sensor
data of the eyeglass frame. The data is recorded by a sensor cluster of IM U sensors
(accelerometers, gyroscopes) and a camera. Beyond wearable smartphones, the proposed
algorithms have potential applications for other types of embedded and robotics systems (e.g.,
sensors mounted on a robotic vehicle operating over different kinds of terrain). Although, the
functionality of eyeglasses is similar to that of a smartphone, the interaction is often only
limited to voice commands. Instead of, e.g., rejecting messages each time via voice commands,
the management system could automatically take over decision control in certain daily
situations [Fig. 1]. Adapting these settings and filtering information will guarantee a high
usability of the eyeglasses. Furthermore, logging classified activities throughout the day allows
support for many other fields. One example is delivering activity information to health care
programs for obesity prevention, treatment, or healthy life style recommendations [3].
11

Studies on activity classification have been demonstrated with a variety of sensors, mostly
inertial sensors (accelerometer, gyroscope), but also in combination with pressure sensors [3]
or a camera [11]. This paper also presents a multi-sensor approach that consists of an
accelerometer, a gyroscope and a camera. Research on activity classification has been done by
placing inertial sensors on various body parts: thigh, hip, waist, forearm, chest, lower back,
knee, ankle, neck, or foot [3], but rarely head [6][7]. Most work claims that placements around
the waist are the closest to the center of mass and thus better represent human motion and
activities [7]. Most studies for this scenario were performed with small databases ranging from
five to nine activities. Five activities were successfully recognized by a wrist-worn
accelerometer (94.13 %) [9] and a waist-worn accelerometer (99.5%) [8]. Six activities achieved
also good results with a pocket-worn (91.7 %) [12] and a belt-worn (82.8 %) [4] smartphone
accelerometer. An IMU placed at the front hip was able to differentiate between nine activities
(90%) [10]. Activities from large databases were performed with distributed inertial sensor
networks (up to six sensor modules) with sensor data simultaneously collected from multiple
body parts e.g. hip, wrist, arm, ankle, and thigh [7]. The most promising distributed sensor
networks scored with almost 84% accuracy on 20 different activities [6]. An inertial sensor
Figure 1: Sample decisions of an individual profile of an eyeglass user
12

network combined with a camera succeeded with 61% on 29 kitchen activities [11]. For fewer
activities (up to six) most of the sensor networks achieved results in the 85-95% range [6]. Head
placements are rare for activity classification and have only been used for detecting falls and
measuring balance during walking [7]. One possible reason is that sensor data is strongly
affected by variations of head orientation. However, when effectively compensated for head
movements, sensor data from eyeglasses have one significant advantage over other body
placements: People are wearing eyeglasses with only slight position variations. Smartphones,
however, can be put into the right or left pocket, in a purse or backpack. For daily use, sensor-
equipped clothes or shoes are less practical because they are often changed. Eyeglasses are
convenient particularly for people wearing prescription eyewear anyway. Furthermore,
eyeglasses are self-contained and do not need a wired or wireless connection to sensors
distributed all over the body, such as sensor networks do. The approach presented in this paper
is also an eyeglass solution, but it has a set of unique contributions: (1) a face-mounted design
of a sensor cluster to record sensor data, (2) an effective compensation technique for head
movements, (3) the identification of critical features, (4) a large database with previously
identified activities, (5) a classification technique to use the above ideas to classify and learn
future activities in a short time.

13

2.3 The SAS Approach
2.3.1 Placement and design of the sensor cluster
The head placement has certain benefits when recording data with a sensor cluster. From a
view point of inertial sensors, the head captures full-body motions (e.g. walking, jogging) with
similar results as other body parts. Part-body motions (e.g. washing dishes, eating) can only be
captured indirectly. When measuring part-body motions, the sensors actually record small
counter-motions of the head. These countermotions appear when the upper body
compensates for lower body movements (e.g. lifting a hand). This is the reason why also part-
body motions can be observed with inertial sensors. From the view point of a camera, the head
is an optimal location to collect activity data. The camera is mostly directed to where the eyes
pay attention to and do not have to deal with occlusion problems (such as smartphones in
backpack, pocket, or purse). From an ergonomic point of view, the eyeglasses allow a very
comfortable and easy daily use. Both IMU sensors and camera are integrated in the left
earpiece of our prototype IMU+Camera eyeglasses [Fig 2].

Figure 2: Design of the eyeglasses. The sensors are attached to the left earpiece
14

The design of the eyeglasses is reduced to two minimum necessary sensors to classify an
activity: A Pololu MinIMU-9 v2 Inertial Measurement Unit captures activities via accelerometer
and gyroscope. A 160x120 pixels camera records images of the environment to put classified
activities into the right contextual setting (e.g. indoors, outdoors).
2.3.2 IMU data transformation into stable coordinate system
One contribution of SAS is to extract features that are not affected by the orientation of the
head or the exact placement of the glasses on the head. When a person performs a head
movement, the local sensor coordinate system will change accordingly. The key idea of the SAS
approach is to keep the sensor data in a coordinate system that will stay stable even when the
head moves. Thus the sensor data needs to be transformed from its dynamic local sensor
coordinate system into a normalized coordinate system. This normalized coordinate system is
defined as the x-axis pointing out of the eye, the y-axis points out of the right ear and the z-
axis points vertically down. The normalized coordinate system compensates for any head
orientation variation in roll and pitch angle, but ignores the yaw angle. The reason for ignoring
the yaw angle is that the direction in which an activity is performed is not relevant. For
example, walking north-bound (yaw = 0°) should result in the same as walking south-bound
(yaw = 180°). When variations in roll and pitch angle happen, the accelerometer data will be
significantly affected by gravitational acceleration. One can walk with the head facing up,
down, or tilted to the side [Fig. 3]. The local sensor coordinate system will change its
orientation along with the head orientation. Particularly accelerometer data will be influenced
by different head orientations [Fig 4].

15

Figure 4: Sequences of walking with different head orientations
As a result, for example, statistical or physical features, such as mean, variance, or energy will
show very different results for each head orientation. Most likely it will also influence the
performance of the classifier. The transformation into a stable normalized coordinate system
will eliminate this effect and will compensate for any head orientation changes. The physical
idea behind the transformation is to use the gravity vector (g) as a static reference. This vector
always points vertically down and aligns with the z-axis of the normalized coordinate system.
The g-vector can be measured via the accelerometer. Furthermore, gyroscope input and fusion
Figure 3: Sensor data from any head orientation is transformed in a stable normalized coordinate
system
16

techniques (DCM Algorithm) are used to make the calculation of the g-vector very robust ±
even when measuring fast motion activities (e.g. jogging, running). When transforming, the z-
axis of the local sensor coordinate system is rotated until aligned with the g-vector. In other
words, it is now aligned with and transformed into the normalized coordinate system. As a
result, the roll and pitch angle of the new coordinate system will be 0° at all times independent
from the head orientation. Now the sensor data is completely invariant from the orientation
of the head and prepared for feature extraction.
2.3.3 Identification of critical features
Feature extraction is separately performed for inertial sensor data and video data. When
extracting features from inertial sensor data, SAS differentiates between statistical and
physical features. Statistical features are purely statistical operations (e.g. mean, variance) on
sensor data. Physical features, however, are calculated from an equation with physical
meaning (e.g. movement intensity, energy consumption). Previous research work uses either
pure statistical features [11] or both statistical and physical features combined [10]. Feature
extraction from video data is limited to one feature per image. Each feature is a large GIST
vector that describes the context of a scene.
2.3.4 Creating an extendable database
SAS must be trained with activities and types of environments in advance to classify them.
First, one minute of inertial sensor data was recorded for each of the 20 activities. 22-
dimensional feature vectors were extracted from six second windows and stored in the activity
database. Activities that are composed of many different motions (e.g. folding clothes) tend to
widely spread out. Activities with one clearly defined motion (e.g. brushing teeth) mostly
accumulate in small local areas [Fig.5]. In addition, SAS can easily extend the activity database.
17

Each new activity requires only one minute of inertial sensor data, to calculate sufficient new
features for classification. Second, 3000 images of video data of two types of environments
were recorded (2100 indoors, 900 outdoors). A 512-dimensional GIST vector was extracted
from each image and stored in the environment database [Fig. 6].

Figure 5: Dimension-reduced activity features displayed in 2D via principal component analysis.
Each color represents a different activity
Figure 6: Dimension-reduced environment database in 2D via principal component analysis with
two classes (inside, outside)
18

2.3.5 Classification technique for activities
First, a Naive Bayes classifier was trained with the activity database. It compares incoming
activity features with database features and outputs a list of probabilities of matching
activities. Second, a Support-Vector-Machine is trained with the environment database. It
compares incoming environment features with database features and outputs the matching
type of environment.

2.4 The Measurement in Detail
2.4.1 Overview
SAS records IM U data with a frequency of 50 Hz, camera images with 2.5 Hz. Furthermore, the
architecture can be structured in three major processing steps [Fig. 7]. Step 1 prefilters sensor
data and transforms it from a local dynamic coordinate system into the stable normalized
coordinate system. Step 2 handles the feature extraction for both IM U and camera data. IM
U data is segmented into windows, followed by the extraction of statistical and physical
features from each window. For each image in the camera data, one GIS T feature is calculated.
Step 3 is performing classification (activities, environments) by using a network of multiple
classifiers. The result of SA S is a list of activities with assigned probabilities.
19

Figure 7: Overview of the system architecture

2.4.2 Steps in Detail
2.4.2.1 Transformation (only for IMU data)
In the transformation step, the raw sensor data is first prefiltered (error-corrected, offset
compensated, and resampled to a constant 100 Hz). Then, the data is transformed into the
normalized coordinate system. The transformation is accomplished with the following two
equations:
𝐚 𝑛𝑜𝑟𝑚𝑎𝑙𝑖𝑧𝑒𝑑 = 𝐑 a
−1
∙ 𝐚 𝑠𝑒𝑛𝑠𝑜𝑟 (1.1)
𝛚 𝑛𝑜𝑟𝑚𝑎𝑙𝑖𝑧𝑒𝑑 = 𝐑 ω
−1
∙ 𝛚 𝑠𝑒𝑛𝑠𝑜𝑟 (1.2)
When recorded in the local coordinate system, acceleration data is stored in the 3x1 vector
a sensor, with a x, a y, a z representing the acceleration along the sensor axes. Gyroscope data is
stored in the 3x1 vector ω sensor, with ω x, ω y , ω z defined as the angular velocities along the
sensor axis.
𝐚 𝑠𝑒𝑛𝑠𝑜𝑟 = [
𝑎 𝑥 𝑎 𝑦 𝑎 𝑧 ] (1.3)
20

𝛚 𝑠𝑒𝑛𝑠𝑜𝑟 = [
𝜔 𝑥 𝜔 𝑦 𝜔 𝑧 ] (1.4)
In the normalized coordinate system, transformed acceleration data is stored in a normalized with
a x, a y, a z representing the acceleration along the normalized coordinate axes. Gyroscope data
is stored in ω normalized with ω x, ω y, ω z and is equal to the first derivative of the Euler Angles roll
φ, pitch θ, and yaw ψ.
𝐚 𝑛𝑜𝑟𝑚𝑎𝑙𝑖𝑧𝑒𝑑 = [
𝑎 𝑋 𝑎 𝑌 𝑎 𝑍 ] (1.5)
𝛚 𝑛𝑜𝑟𝑚𝑎𝑙𝑖𝑧𝑒𝑑 = [
𝜔 𝑋 𝜔 𝑌 𝜔 𝑍 ] = [
𝜙 ̇ 𝜃 ̇ 𝜓 ̇ ] (1.6)
To connect the sensor data vectors (a sensor, a normalized) and (ω sensor, ω normalized), two transformation
matrices R a and R ω are required. Both are composed from basic rotation matrices:
𝐑 x
(𝜙 )= [
1 0 0
0 cos(𝜙 ) −sin(𝜙 )
0 sin(𝜙 ) cos(𝜙 )
] (1.7)
𝐑 y
(𝜃 )= [
cos(𝜃 ) 0 sin(𝜃 )
0 1 0
−sin(𝜃 ) 0 cos(𝜃 )
] (1.8)
The transformation matrix Ra for transforming accelerometer data into normalized
coordinates is derived as following:
𝐑 a
= 𝐑 x
(𝜙 )𝐑 y
(𝜃 )= [
cos(𝜃 ) 0 −sin(𝜃 )
sin(𝜃 )sin(𝜙 ) cos(𝜙 ) cos(𝜃 )sin(𝜙 )
sin(𝜃 )cos(𝜙 ) −sin(𝜙 ) cos(𝜃 )cos(𝜙 )
] (1.9)
The transformation matrix RM to transform gyroscope data into normalized coordinates is
derived as following [15]:
21

𝐑 ω
= [[
1
0
0
],𝐑 𝒙 𝑻 (𝜙 )∙ [
1
0
0
]
𝒚 𝑻 ,𝐑 𝒙 𝑻 (𝜙 )∙ 𝐑 𝒚 𝑻 (𝜃 )∙ [
1
0
0
]] = [
1 0 −sin(𝜃 )
0 cos(𝜙 ) cos(𝜃 )sin(𝜙 )
0 −sin(𝜙 ) cos(𝜃 )cos(𝜙 )
] (1.10)
Once the transformation into the stable normalized coordinate system is performed, features
are extracted from sensor data.

2.4.2.2 Feature Extraction for IMU data
In total, 22 features are extracted from inertial sensor data [Fig. 8]. Seven features (energy,
periodicity) are created by transforming sensor data into the frequency spectrum via FFT
(Fast Fourier Transformation).

Figure 8: Statistical and Physical Feature distribution
22

Mean is used to measure the average acceleration and angular velocity for each sensor axis
over one window length. Mean is larger for activities with strong body motions. Variance
describes how far acceleration and angular velocity are spread out along an axis. Fast and wide
motions are larger. Movement Intensity (MI) specifies the intensity of motions. Mean and
variance of MI are calculated over one window [10].
𝑀𝐼 [𝑡 ] = √𝑎 𝑥 [𝑡 ]
2
+ 𝑎 𝑦 [𝑡 ]
2
+ 𝑎 𝑧 [𝑡 ]
2
(1.11)
Energy describes the motion quantity [8]. E i measures the energy for each axis; E is the energy
over the entire system. Parameters are N (number of samples per window length), M ai
(discrete FFT component magnitude of acceleration along the axis i), f (frequency), and F
(maximal frequency of window).
𝐸 𝑖 =
1
𝑁 ∑ 𝑀 𝑎 𝑖 [𝑓 ]
2
𝐸 𝑓 =0
(1.12)
𝐸 =
1
𝑁 ∑(𝑀 𝑎 𝑋 [𝑓 ]
2
+ 𝑀 𝑎 𝑌 [𝑓 ]
2
+ 𝑀 𝑎 𝑍 [𝑓 ]
2
)
𝐸 𝑓 =0
(1.13)
Energy Expenditure (EE) is also known as the normalized signal magnitude area [10] and
describes the amount of energy used for an activity. T is specified as the time of one window.
𝐸𝐸 =
1
𝑇 (∑|𝑎 𝑋 [𝑡 ]|
𝑇 𝑡 =1
+ ∑|𝑎 𝑌 [𝑡 ]|
𝑇 𝑡 =1
+ ∑|𝑎 𝑍 [𝑡 ]|
𝑇 𝑡 =1
) (1.14)
Periodicity (f peak) detects recurring motions. f peak determines the highest dominant frequency
for an axis i. Parameters are M ωi (discrete FFT component magnitude of angular velocity along
the axis i), c (minimum required magnitude threshold) to avoid noise peaks.
23

𝑓 𝑝𝑒𝑎𝑘 = MAX(𝑀 𝜔 𝑖 > 𝑐 ) (1.15)
2.4.2.3 Feature Extraction for video data
Camera data is processed with a GIST descriptor for scene recognition provided by A. Torralba
from MIT [13]. For each image, one 512 dimensional GIST feature is calculated
2.4.2.4 Classification
SAS uses the extracted features to classify a situation. A situation is specified by an activity and
an environment. For this reason, the classification process is split into three parts. First, a Naive
Bayes classifier distinguishes activities by analyzing inertial data. In parallel, a Support Vector
Machine (SVM) classifies between environments via video data. Finally, both results are
combined in a Hidden Markov model (HMM) to further improve the activity classification. The
Naive Bayes Classifier (activity classification) is trained with one minute of training data per
activity. This training data is divided into 10 training windows ± each 6 seconds. A Naive Bayes
Classifier only requires a relatively small amount of training data to estimate its two internal
parameters (mean, variance) for classification. The SVM (environment classification) is an
effective classifier for high dimensional data [10][14]. SVM is trained with GIST features of 1680
images (1260 indoors and 420 outdoors). The HMM combines both classification results
(activities, environment) and used as observation inputs in the HMM. The number of possible
observations (40) is the amount of activities (20) times the amount of environments (2) [Fig.
9]. Furthermore, HMM is used to model transition probabilities between activities. Therefore,
the matrices with transition and emission probabilities were designed based on real-world
behavior: Activity states are more likely to repeat themselves than switching into a different
activity state. Indoor activities are more likely to transfer into indoor activities. This applies to
outdoor activities accordingly. Overall, the walking state is a more likely transfer state, because
it connects most activities in the real world.
24

2.5 Test Results
2.5.1 Performance tests of Naive Bayes and SVM
SAS was tested with data chunked into 6s windows with 3s overlap. Each window was
classified. The training and testing was performed on different days by the same subject. Test
data only included activities which the database was previously trained for [Fig. 10]. The device
may need to be trained for each further subject in future (similar to speech recognition). First,
the classification performance for activities (Naive Bayes Classifier) was tested with sets of 5,
10, 15 and 20 activities. Then, the success rate of the SVM classifier for environments was
measured. Finally, the fusion performance of the HMM was determined.
Figure 9: HMM architecture (40 observation states, 20 hidden states)
25

Activity classification with inertial data: The performance of SAS was first tested with a small
set of 5 basic motion activities (lying down, walking, jogging, biking, running). Then, 10
activities were tested by adding mainly indoor tasks (vacuum cleaning, washing dishes,
brushing teeth, taking stairs, eating) followed by 15 activities by including hobby tasks (playing
cello, playing piano, playing table tennis, computer work, reading book). Finally, all 20 activities
were tested by adding complex activities (folding laundry, doing laundry, wiping cupboard) or
activities (driving car, taking elevator) that had a very similar head motion pattern to an activity
already existing in the database (computer work).
Results show that pure physical features and all features combined show a slightly stronger
performance than pure statistical features. For large databases (20 activities), pure physical
features (80.3%) even showed a slightly better performance than all features combined
(79.3%) [Fig. 11]: It turns out that statistical features do not improve the overall result of
physical features. Therefore, only physical features were used for further testing with HMM.
Figure 10: The 20 daily activities used for testing
26

Environment classification with video data: The SVM classifier over GIST data showed a
successful performance of 89.8% correct classification

2.5.2 Full system testing (including HMM)
The overall classification performance of SAS was tested. For this reason, both outputs of Naive
Bayes (activities) and SVM (environments) were assigned to the HMM. The HMM re-classified
activities to take environment information into account. The HMM classification performance
is shown in a confusion matrix (in %) [Fig. 12].
HMM achieved a classification result of 81.5%. Because of the transition probabilities and the
SVM input, the activities were recognized in a significantly more stable way than only using
Naive Bayes. This greater temporal stability was the main added value of the HMM. However,
the classification still fails in certain situations. For example, stopping a car at an intersection
will naturally discontinue the characteristic sensor data pattern for "driving car".
Figure 11: Test results of activity sets with pure inertial data
27

When the stop lasts for too long, the classified activity is likely to switch from "driving car" into
a better matching sensor data pattern, e.g. "computer work" or "taking elevator" (during
constant-speed travel, and discounting the initial / final vertical acceleration phases) [Fig. 12].
Under certain circumstances, classifying an activity can completely fail, if an activity can be
performed in too many different ways. While "wiping cupboard" in certain directions, head
counter motions apparently showed a too high similarity to "biking" and "taking stairs". As a
result, the HMM misclassified it. Furthermore, unstable and frequent switching of the Naive
Bayes classifier and low HMM transition probabilities were further contributing to the
Figure 12: Hidden Markov Model Results: Correct classification (green), wrong classification
(red), transition errors (orange), initializing error (blue)
28

misclassified state [Fig. 12]. However, SAS shows a very stable and promising overall result with
most activities being classified with 85% and higher success rate. Interestingly, this result was
obtained when using the physical features only, as we found that the contribution from
statistical features was negligible. A sample video was recorded for all 20 activities and
demonstrates the classification outcomes of all three classifiers
(https://www.youtube.com/watch?v=rZ1ZIIHUU2M).
2.6 Discussion and Conclusion
Activities without a distinct motion (computer work, taking elevator), or with too wide a range
of possible submotions (wiping cupboard, folding laundry), are easily misclassified. Besides
extracting physical features, the composition of these activities needs to be further studied. If
a complex activity such as folding clothes can be broken into all submotions, a probability
distribution (e.g. 40% shaking, 20% vertical movement, 40% horizontal movement) could
possibly lead to a more promising result. When extending the database with more activities,
the video data need to further be used to distinguish between significantly more environments
or even object recognition. Often, eyeglasses come along with microphones. Additional sensor
input could also contribute by measuring the surrounding noise level (e.g. car noises in street,
sound of music instruments, pure silence). SAS showed that head-mounted sensor systems can
be successfully used to measure 20 activities with an accuracy of 81.5% (15 activities with
91.3%). With the new eyeglass devices entering the market of communication technology,
situation-adapted controlling of system settings guarantees a high-usability for daily use. By
providing valuable activity classification, we hope that SAS made a contribution to the
development of situation awareness systems for eyeglasses.

29

3. Walking compass with head-mounted IMU sensor
3.1 Abstract
Emerging wearable technologies offer new sensor placement options on the human body.
Particularly, head-mounted glasswear opens up new data capturing possibilities directly from
the human head. This allows exploring new cyber-robotics algorithms (robotics sensors and
human motor plant). Glass-wear systems, however, require additional compensation for head
motions that will affect the captured sensor data. Particularly, pedestrian dead-reckoning
(PDR), activity recognition, and other applications are limited or restricted when head-
mounted sensors are used, because of possible confusion between head and body
movements. Thus, previous PDR approaches typically required to keep the head pointing
direction aligned with the walking direction to avoid positional errors. This paper presents a
head-mounted orientation system (HOS) that identifies and filters out interfering head
motions in 3 steps. Step 1 transforms inertial sensor data into a stable normalized coordinate
system (roll/pitch motion compensated). Step 2 compares walking patterns before and after a
rotating motion. Step 3 eliminates interfering head motions from sensor data by dynamically
adjusting the noise parameters of the extended Kalman filter. HOS has been implemented on
a Google Glass platform and achieved high accuracy in tracking a person’s path even in the
presence of head movements (within 2.5% of traveled distance) when tested in multiple real-
world scenarios. By eliminating head motions, HOS not only enables accurate PDR, but also
facilitates the task for downstream activity recognition algorithms.

Video Reference: https://www.youtube.com/watch?v=kUY9NpEMgaI
Published in: 2016 IEEE International Conference on Robotics and Automation (ICRA)
30

3.2 Introduction
The industry for smart personal devices with integrated sensors suitable for position tracking
and activity recognition has been emerging over the last few years. While this trend began first
with Smartphones, various devices such as smart watches, fitness trackers, and increasingly
head-mounted devices are now entering the market. Prominent examples are Google Glass,
Microsoft’s HoloLens and Oculus Rift. These head-mounted devices are equipped with inertial
sensors (IMU). While wearing them, the user is constantly recording valuable sensor data. This
information can be used to make this technology smarter by performing activity recognition
and creating situation awareness for the device, which can then adapt its behavior to the
current user activity. This could greatly enhance many device applications, e.g., music
streaming with Google Play Music: for different activities (e.g., working out, making food,
cleaning the house, or driving a car), the service adapts its music offerings; but currently users
have to manually indicate which activity they are engaged in. We showed in previous work that
head-mounted sensor systems can successfully distinguish between many daily activities
through online analysis and classification of the streaming IMU data, but this was when the
head was held pointing straight forward [25]. Head-mounted devices can also be used to track
motions without external reference systems in 3D Augmented Reality Virtual Rooms. This
allows users to create training simulations, exercise in environments or help with home
improvement by visualizing remodeling projects (e.g., Lowe’s Holoroom), with again the
current problem that head and body motions may be confused.
Tracking motions and activities is difficult. The head pointing direction may not always align
with the body orientation, which can lead to a wrongly classified activity or a positional error
in PDR. For this reason, we developed a stabilization algorithm that enables all these
applications. To test and validate our work, we use PDR as one of the most stringent
31

applications, and we focus on being able to recover walking direction and trajectory over time
even as the head occasionally turns side to side [Fig 13].

Figure 13: Possible head pointing directions (red arrows) and walking directions (green arrows).
Head-mounted pedestrian dead-reckoning systems need to compensate for head motions for correct
position tracking (green path) and to avoid false positional tracking (red path).
PDR has been well studied for various sensor systems and is mainly used to supplement other
navigation methods. PDR systems have been applied to many use-cases with different
placements of sensors on most body parts (helmet, foot, backpacks, pockets, wrist, and legs).
Most PDR systems work after the same principle: Identifying footsteps via accelerometer and
aligning them to the proper heading direction via orientation sensors such as gyroscope and
compass. A team of the University of Bremen developed a safety helmet for emergency
services. IMU sensors and a GPS receiver attached to the helmet were used for indoor/outdoor
navigation. Errors were as large as 5.4% of the total distance travelled with a cumulated model
step and 2% with a neural network prediction [16] [17]. However, the helmet needs to be
32

pointed in the direction of walking at all times and cannot compensate for head motions [16].
IMU sensors have also been attached to shoes in several projects.
The MiPN Project focused on supporting a soldier’s GPS system, particularly for indoor
tracking. The NavShoe System was developed for firefighters or other emergency first
responders. In both cases, the sensor data was merged with GPS data for long-term drift
compensation. In general, the position error for foot-mounted sensors varies between 0.3% -
2% of the distance traveled [19] [21], but this requires an additional battery-powered device
on the foot. Smartphone-based indoor navigation systems reached 0.19% - 2.92% accuracy
depending on the smartphone’s placement [20] [21] [23]. All tested PDR systems only work
reliably if the orientation of the sensors aligns with the orientation of the walking direction.
PDR systems have also assisted vision systems and decreased positional error to 0.5%.
Navigation applications can be found in the tourism industry, particularly guiding tourists
through museums, train and subway stations, and large buildings. Blind or visually impaired
people could also benefit from a navigation assistant [24]. For instance, the Walking Compass
tries to identify walking direction from accelerometer patterns [18]. Results show different
performances depending on the position the smartphone is carried in: Their accuracy is ±5º
(palm), ±3º (pant pocket), ±8º for swinging-hand. However, errors were reported when user
vary the phone’s orientation during operations. Instead of aligning footsteps with the heading,
one could also double integrate the acceleration. However, due to the complex composition
of acceleration data (gravitational acc., linear acc., coriolis acc., centrifugal acc., static noise,
and acceleration bias), double integration is not a suitable technique for low-cost IMU sensors
since it requires very accurate linear acceleration data. Overall, the performance of current
PDR systems is promising, however, many restrictions apply particularly for the orientation of
the sensors. Short-term errors are related to incorrect heading directions [17] that can be
33

caused by rotational motions (body, head, arm). Thus most applications that do not have
efficient compensation methods will be restricted to keep their sensor orientation in a fixed
orientation (e.g., perpendicular) to the walking direction [20]. This is why no head-mounted
PDR system has been presented yet, which actively compensates for rotational head-motions
and performs comparably to existing PDR systems. The HOS approach presented here is based
on head-mounted IMU data and has a head-motion compensation algorithm. This algorithm
can effectively compensate head-motions as long as they do not overlap with rotational body
motions. Synchronous rotational movements of head and body lead to a confusion and require
additional information or complementary sensor data (e.g., GPS tracking over time) to recover
the correct orientation information. The HOS approach is described in detail in the following
sections, and it has a set of unique contributions: (1) analysis of head-mounted IMU data
during walking and head movements, (2) effective segmentation technique of rotational
motion sensor data, (3) selection of critical features for filter tuning, (4) fusion algorithm with
dynamical tuning, (5) collection of test scenarios.

3.3 Analysis of IMU Data during walking activity
To understand the frequency composition of walking, 60 seconds of X-Y plane accelerometer
sensor data was recorded. During the recording, the head pointed in multiple directions within
a ±60º range from the walking direction [Fig 14]. The sensor data was then divided into multiple
3 sec windows and transformed into the frequency spectrum. Two frequencies showed a
critical magnitude [Fig 15]. The frequency value of the peaks depends on the walking speed
(and frequency of footsteps), but can be classified as:
34

Side Swinging Peak: Body swings to the right and left side for every step taken. The
accelerometer will capture the swing motion direction change. One full swing sequence
requires two steps.
Stepping Peak: Each step generates linear acceleration front-to-back (in addition to a vertical
component). When walking with the head straight (aligned with walking direction), steps are
dominantly recorded by the x axis, while swing is captured by the y axis. When walking with
the head pointed sideways, both x and y axes measure both stepping and swinging. With the
head turned 90º, the peaks would be measured by the reverse axes (swinging by x, stepping
by y).

Figure 14: X/Y axis detect swinging and stepping in different magnitude depending on head
orientation.
35

Figure 15: Frequency Spectrum of walking activity with head turned in multiple orientations.

3.4 The Measurement Detail
3.4.1 Overview
HOS records IMU data at 50 Hz. The Google Glass platform provides the accelerometer data as
well as an orientation estimator for Roll and Pitch. The architecture is structured around the 3
major processing steps summarized in Figure 16.

36

Figure 16: Overview of the system architecture
Step 1 pre-filters sensor data and transforms it into a stable normalized coordinate system to
compensate for roll and pitch changes. This is necessary to measure the accelerometer data in
a stabilized X/Y plane. Step 2 identifies walking patterns among the accelerometer axis x and
y. A step recognition algorithm determines the step duration from the walking patterns by
detecting peaks in the acceleration that correspond to footfalls. Head motions can be
identified by changes in the walking pattern described by a confidence parameter. Step 3 uses
this parameter to eliminate gyroscope data from head-motions. The result of the 3 steps is the
yaw angle which describes the heading direction of the pedestrian. If combined with step
counts of Step 2, the position of a person can be tracked.

3.4.2 Steps in Detail
3.4.2.1 Step 1: Compensation for Roll and Pitch Motions
In this step, the raw IMU sensor data of the Google Glass platform is first pre-filtered (offset
compensated to minimize gyro drift). Then, the data is transformed into the normalized
coordinate system. This transformation is necessary to measure two critical types of sensor
data during walking independently of the actual head orientation [25].
37

(1) The angular velocity ω z around the z-axis (vertical axis, perpendicular to the ground plane)
which is used to calculate the heading direction 𝜑 .
(2) The acceleration a x and a y along the x and y axis (in the plane parallel to the ground plane)
which is used to calculate the confidence parameter C gyro for the gyroscope data.
The transformation is accomplished by two equations (1) (3), where a is a 3x1 vector
representing the accelerometer sensor data and ω is a 3x1 vector representing the angular
velocities.
a
𝑛𝑜𝑟𝑚𝑎𝑙𝑖𝑧𝑒𝑑 = R
a
−1
∙ a
𝑠𝑒𝑛𝑠𝑜𝑟 (2.1)
𝐑 a
= 𝐑 x
(𝜙 )𝐑 y
(𝜃 )= [
cos(𝜃 ) 0 −sin(𝜃 )
sin(𝜃 )sin(𝜙 ) cos(𝜙 ) cos(𝜃 )sin(𝜙 )
sin(𝜃 )cos(𝜙 ) −sin(𝜙 ) cos(𝜃 )cos(𝜙 )
] (2.2)

𝛚 𝑛𝑜𝑟𝑚𝑎𝑙𝑖𝑧𝑒𝑑 = R
ω
−1
∙ 𝛚 𝑠𝑒𝑛𝑠𝑜𝑟 (2.3)
𝐑 ω
= [
1 0 −sin(𝜃 )
0 cos(𝜙 ) cos(𝜃 )sin(𝜙 )
0 −sin(𝜙 ) cos(𝜃 )cos(𝜙 )
] (2.4)
Note that this formulation using Euler Angles suffers from a singularity at 𝜃 = ±90° which is not
an issue here because it relates to an unrealistic head pose. The IMU data is now compensated
for roll and pitch motions. Only the normalized z-axis of the IMU is aligned with the gravity
vector. The normalized x- and y- axis measure sensor data without any gravity component. In
this gravity-compensated coordinate system, the sensor data is now used for the walking
pattern analysis.
38

3.4.2.2 Step 2: Walking Pattern Analysis
The HOS-Filter measures changes in the walking pattern as captured in the accelerometer data.
These changes are used to calculate a confidence parameter on how much the HOS-Filter can
trust the gyroscope data. This confidence parameter is the only critical feature to evaluate a
detected rotational motion.
To accomplish step 2, the transformed gyroscope is used to identify moments of rotational
motions. These timestamps are split at segmentation points, then analyzed in a Gyroscope
Activity Zone and an Observation Zone.
Segmentation Points: The segmentation points (red big arrow in Figure 17 and 18) are
segmented by a necessary minimum peak height (absolute value) of normalized gyroscope
data as well as necessary minimum peak distance.
Gyroscope Activity Zone: The gyroscope activity zone is an isolation zone around the
segmentation point that involves gyroscope activity and must not be used for evaluation of the
walking pattern. The algorithm extracts a time-window around a segmentation point until the
normalized gyroscope data falls below a set threshold. The gyroscope activity zone only tells
us if the gyroscope captured a rotational motion. It does not tell us if the head direction or the
walking direction changed.
Figure 17: Walking a straight line with changing head pointing direction. Gyroscope activities
with a peak threshold will be captured (red arrow) and the gyroscope activity zone segmented
(green area)
39

Observation Zone: The observation zone is necessary to evaluate the walking pattern before
and after the gyroscope activity zone. It will create the most critical feature to calculate the
“confidence” level of gyroscope data. Two fixed-length observation zones are extracted, one
before (B) and one after (A) a gyroscope activity zone [Fig 18]. These observation zones are up
to 180 samples large (3.6 sec), they are compared to each other and analyzed for similarity.
The similarity is shown as a confidence parameter (1 = highest similarity, 0 = no similarity). The
larger the similarity, the less likely a head motion occurred, since the alignment of the head to
the walking pattern remains unchanged between A and B. To calculate the similarity, we need
to first measure the Peak Ratio P ratio.
Peak Ratio 𝑃 𝑟𝑎𝑡𝑖𝑜 =
𝑃 𝑋 1
𝑃 𝑋 2
(2.5)
This value is calculated by diving the magnitude of P X1 by P X2. Both parameters P X1 and P X2
represent the two critical magnitude peaks of the FFT-transformed accelerometer data
window of the x-axis in the normalized coordinate system. Dividing P X1 by P X2 works amplifying
Figure 18: The Peak Ratio will be observed before (B) and after (A) the Gyroscope Activity Zone
and will provide valuable information if the walking pattern changed
40

on the final Peak Ratio. Once one magnitude increases (P X1 or P X2), the other one declines (P X2
or P X1). The Peak Ratio represents the ratio between swinging and stepping activity and is
directly related to the head pointing direction relative to the walking direction. Depending on
the head angle, the peak ratio changes accordingly [Fig 19].
Next, peak ratio data of the observation zones are selected. This data is necessary to evaluate
the walking pattern before and after the gyroscope activity zone. For both zones, the expected
value μ and the standard deviation σ are calculated and used to calculate the confidence
parameter. This parameter describes the trust in the gyroscope data. The confidence
parameter equation is derived from [26] and approximates the intersection area of the two
Gaussian Bell curves. The confidence parameter is calculated with the Complementary Error
Function erfc. The max confidence is 1 for a max intersection area. The min confidence is 0 for
no intersection area.
Confidence P arameter 𝐶 𝑔𝑦𝑟𝑜 = erfc(
|𝜇 1
− 𝜇 2
|
√2 ∙ (𝜎 1
2
+ 𝜎 2
2
)
) (2.6)

Figure 19: Calculation Example for Peak Ratio with Head Straight and Head Sideways
41

3.4.2.3 Step 3: Fusion Algorithm
The system uses a fusion algorithm to combine IMU measurement data (accelerometer,
gyroscope) with the derived features: step size (assuming const. 71.6 cm length), walking
velocity (step size / step duration), and the confidence parameter [Fig 21]. The confidence
parameter is the only feature to evaluate if the gyroscope data is counted as a head motion or
not. If the gyroscope detects a rotational motion, the system dynamically adjusts the
measurement noise matrix. For this step, the confidence parameter plays an important role
since it describes the ‘trust’ in the gyroscope data. A large parameter expresses high trust (no
head motions). Barely or no intersection show low ‘trust’ into the gyroscope data. This
information will then be used for the Extended Kalman Filter Fusion Algorithm [Fig 21].
Figure 21: Fusion Algorithm to combine IMU Data with derived features

Figure 20: The confidence parameter approximates the intersection area of the two Gaussian Bell
Curves. Both curves represent the Observation Zone before and after the gyroscope activity.
42

A Constant Velocity / Constant Angular Velocity Model (CVω–Model) was developed. The
model is described with two state dynamics equations. The first is called the state transition
model. This equation calculates the system state vector x k by the system’s dynamic function
f(x k-1,u k-1) with u k-1 as the control vector and added process noise ω k-1. The second equation
describes the observation model. The observation vector z k is calculated by the measurement
function h(x k) and the measurement error source v k. The Extended Kalman Filter uses two steps
to fuse the sensor data: The prediction step and the update step.
(1) Prediction Step
The state estimate x of the pedestrian includes the positional information x and y, the heading
angle 𝜑 , and the absolute velocity v. The CVω-Model does not require the heading angle 𝜑 to
be constant. This model only assumes that the angular velocity ω (turning rate of the
pedestrian), will stay constant. In addition the absolute velocity is measured via the step
detection and as well remains constant in the CVω-Model. The following equation describes
the implementation of the system’s dynamic function f(x k-1,u k-1).
𝑥 𝑘 =
[

𝑥 𝑦 𝜑 𝑣 𝜔 ]

𝑘 |𝑘 −1
=
[

𝑥 𝑦 𝜑 𝑣 𝜔 ]

𝑘 −1|𝑘 −1
−
[

𝑣𝑇𝑐𝑜𝑠 (𝜑 )
𝑣𝑇𝑠𝑖𝑛 (𝜑 )
𝜔𝑇
0
0
]

(2.7)
(2) Update Step
The first part of the update step is calculating the innovation ỹ 𝑘 by subtracting the predicted
measurement h(x) from the sensor measurement.
𝑦̃
𝑘 = [
𝑟 𝑟 𝑇 𝜔 𝑔𝑦𝑟𝑜 ]
𝑘 − [
√
(𝑣𝑇𝑐𝑜𝑠 (𝜑 ))
2
+ (𝑣𝑇𝑠𝑖𝑛 (𝜑 ))
2
𝑣 𝜔 ]
𝑘 |𝑘 −1
(2.8)

43

The measurement noise matrix R of the fusion filter is dynamic. It is adjusted in every update
cycle with the confidence parameter C gyro of the gyroscope data. Since the noise level should
decrease with increasing confidence parameter and vice versa, C gyro needs to be subtracted
from 1. It is multiplied with a tuning factor k to adjust its range for the fusion filter.
𝐃𝐲𝐧𝐚𝐦𝐢𝐜 𝐍𝐨𝐢𝐬 𝐞 𝐌𝐚𝐭𝐫𝐢𝐱 :𝐑 = 𝐑 𝑠𝑡𝑎𝑡𝑖𝑐 + 𝐑 𝑑𝑦𝑛𝑎𝑚𝑖𝑐 (2.9)

𝐷𝑦𝑛𝑎𝑚𝑖𝑐 𝑐𝑜𝑚𝑝𝑜𝑛𝑒𝑛𝑡 :𝐑 𝑑𝑦𝑛𝑎𝑚𝑖𝑐 = [
0 0 0
0 0 0
0 0 𝑘 ∙ (1 − 𝐶 𝑔𝑦𝑟𝑜 )
] (2.10)
𝑆𝑡𝑎𝑡𝑖𝑐 𝑐𝑜𝑚𝑝𝑜𝑛𝑒𝑛𝑡 :𝐑 𝑠𝑡𝑎𝑡𝑖𝑐 = [
𝜎 𝑚𝑟
2
0 0
0 𝜎 𝑚𝑣
2
0
0 0 𝜎 𝑚𝜔
2
] (2.11)
If the belief C gyro in the gyroscope data is high (EKF noise is low), the fusion filter will trust the
gyroscope data and validate gyroscope activity as changes in the walking direction. If the trust
in C gyro is low (EKF noise is high), the fusion filter will consider the gyroscope data related to
head-motions and will not take it into account. The output of the Extended Kalman Filter is a
vector of five entries including the position and heading angle of the pedestrian.
3.5 Test Results
3.5.1 Performance Tests of HOS
Two types of tests were performed on a university campus parking lot. The first test includes
a basic case test evaluating the performance of three base scenarios. The second test includes
a more diverse testing setup with multiple base scenarios combined. Each test scenario was
repeated three times. We analyzed a total of 12 datasets. We are particularly interested in the
short-term detection since of the nature of PDR systems assisting GPS systems and other
navigation systems that provide a long-term stability.
44

3.5.2 Basic Case Testing
HOS was tested with datasets of 30 sec of one subject. Each dataset counts around 50 steps
describing a specific case. In the case testing, the success rate of the Filter Algorithm was tested
for every case independently.

Setup
The performance of HOS was first tested on 3 basic cases [Fig 22]:
• Case 1 (Walking straight line, no head direction change)
• Case 2 (Walking straight line, with head direction change)
• Case 3 (Walking with turn, no head direction change)
45

Figure 22: Three basic case scenarios (Red Arrow: Head pointing direction, Green Arrow: Walking
Direction)
46

Results
Case 1 and Case 3 – no head motions involved
For Case 1 and Case 3, the results are expected to be similar to current PDR systems, since the
head pointing direction aligned with the walking directions at all times. Like previous PDR
systems we measure error as a percentage of distance traveled [Fig 23]. When a walking
direction change occurred, the filter algorithm successfully validated the rotational activity as
a change in walking direction and took all rotational motions into account.
Case 2 – head motions involved
For Case 2, the filter algorithm successfully detected head-motions and disregarded the
gyroscope activity. Standard PDR systems would fail in this case scenario since the change in
the head pointing direction would have been validated as a change in the walking direction.
The error for standard PDR systems would increase depending on the heading angle error Δ𝜑 :
PDR Error= √2 ∙ (1 − cos(∆𝜑 )) (2.12)

Figure 23: Results of the Basic Case Testing. (*) PDR error from system in [1] [2] [5] [6] [8].

47

3.5.3 Mixed Case Testing
Setup
HOS was tested with three datasets containing data of a walking subject of 30 sec. Each dataset
counts 50 steps (35 meters) and is a combination of multiple base cases [Fig 24]. In the
scenarios, the success rate of the Filter Algorithm was tested.

Results
Figure 25: Combination of Case 1, Case 2, and Case 3. The blue path represents the walking path,
the white arrows are the head pointing directions. The red markers show head motion activity, the
green marker represents a walking direction change
HOS successfully identified and filtered the head-motion from the recorded data [Fig 25]. HOS
performed an error lower than 2.5% of the traveled distance. Standard PDR systems would
have already gained an error of approximately 9.1 meters due to the left head turn which
Figure 24: Motion sequence for mixed case testing
48

would have been misclassified as a left body turn. Since this error would further grow over
time, navigation would not be possible [Fig 26].
Figure 26: Results of Mixed Case Testing (*) PDR error from system in [1] [2] [5] [6] [8].

3.6 Discussion and Conclusion
We have shown that interfering head motions can be successfully detected and compensated
for in PDR applications. However, in real-world scenarios, head rotations and body rotations
may occur at the same time or head motions could be unrecognized when performed gradually
over a longer time. For this work, the walking pattern provided critical information about
whether a head-motion occurred. To do motion tracking performed by both head and body at
the same time, more features besides the confidence parameter (e.g. rotational acceleration
may exhibit different patterns during head turns and body turns) need to be extracted from
the sensor data and walking pattern. These features would need to be tested on many subjects
who are both walking and moving their head in unconstrained real-world situations. In
addition, head tracking can be performed during many activities. In future research, we will
focus on showing that the HOS algorithm is also suitable for different activities with recurring
patterns such as jogging, running, biking, and skating. By providing a valuable approach for PDR
from a head-mounted sensor system, we hope that HOS made a contribution to the
development of navigation methods via wearable technology.

49

4. Inertial Machine Monitoring System for automated failure
detection
4.1 Abstract
Smart manufacturing technologies are emerging which combine industrial equipment with
Internet-of-Things (IoT) sensors to monitor and improve productivity of manufacturing. This
allows for new opportunities to explore algorithms for predicting machine failures from
attached sensor data. This paper presents a solution to non-invasively upgrade an existing
machine with an Inertial Machine Monitoring System (IMMS) to detect and classify equipment
failure or degraded state. We also provide a strategy to optimize the amount, placement
locations, and efficiency of the sensors. In experiments, the system collected data from 36
inertial sensors placed at multiple locations on a 3D printer. Normal operation vs. 10 types of
real-world abnormal equipment behavior (loose belt, failures of machine components) were
detected and classified by Support Vector Machines and Neural Networks. Using under 1
minute of recording while running a test print, a recursively discovered best subset of 4 to 9
sensors yielded 11-way classification accuracy over 99%. Our results suggest that even a small
sensor network and short test program can yield effective detection of machine degraded state
and can facilitate early remediation.

Video Reference: https://www.youtube.com/watch?v=dcm7vNEb8cA
Submitted to: 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

50

4.2 Introduction
Unplanned downtime of machines costs industrial manufacturers an estimated $50 billion
annually [Emerson, 2017]. Equipment failure is the cause of 42 percent of this unplanned
downtime. The results are excessive maintenance, time-consuming repair or equipment
replacement cost. However, worst case scenarios can be avoided by upgrading industrial
equipment with smart predictive machine failure systems. Such systems monitor equipment
with a sensor network and recognize abnormal machine behavior during run-time. In case of
an identified failure, the system could schedule on-demand maintenance service to avoid
larger economical damage to manufacturers. This is particularly important for factory
machines, of which more than 90% are not yet network connected, and the vast majority are
more than 15-years-old [Business Insider, 2016]. Upgrading equipment with predictive failure
systems could save cost and increase efficiency and productivity, not only for the
manufacturing industry. It is predicted that by 2025, the industrial IoT market will be worth
$11 trillion and predictive maintenance can help companies save $630 billion over the next 15
years [McKinsey Global Institute Report, June 2015]. Although predictive maintenance has the
most potential economic impact in factories (e.g. CNC, assembly robots, Laser Cutters, Pick-
and-Place Machines), there is a wide field of potential applications for predictive failure
systems. This ranges from Transportation (vehicle monitoring, condition-based maintenance
of engines ranging from scooters to jumbo jets), Offices (e.g., elevator surveillance), Retail
(e.g., commercial washing machines), but also could be implemented in high-end consumer
and hobby products such as espresso machines or 3D printers. Fig. 27 shows an example.

51

Figure 27: Sample decisions of a customized Machine Monitoring Setting (3D printer with belt-
driven print head that moves along 3 axes)

IoT-based, network-connected, and offline machines, could all benefit from a predictive
machine failure system. What are the critical challenges to predicting equipment failure? First,
abnormal behavior of a machine needs to be detected and classified. Critical for the success of
this step is the right combination of sensor hardware and data processing technique.
Particularly, two types of sensors have proven to be valuable for equipment failure detection.
First, microphones have been used to detect equipment failure with sound vibrations. One
example is the Fault Detection and Diagnosis of Railway Point Machines. A microphone records
moving noise of switchblades during operations of railway infrastructure. Statistical Features
are used to detect and classify three failure scenarios with an accuracy exceeding 94.1% [27].
52

Microphones have also been used for Acoustic Processing for Joint Fault Diagnosis of Legged
Robots [28]. Microphone-based approaches mainly focus on sound in the high-frequency
spectrum [29]. Our approach rather focuses on low-frequency vibrations (up to 25 Hz) which
covers the major frequency spectrum of motions in linear axis machines. For this reason, we
decided to design our hardware with a network of Inertial Measurement Units (IMUs). These
devices are sensor cluster which embed three-axes accelerometers and gyroscopes (and
sometimes magnetometers). They are well suited to capture low-frequency vibrations and
motions. IMU sensor networks have been used for many applications including Wind-turbine
Blade Condition Monitoring [30]. IMU-based approaches have shown to measure
deformations of a single linear axis by 5 µm (linear error) or 15 µrad (angular error) due to axis
degradation [31]. A research team from the Chongqing Technology and Business University
developed a system for fault identification and classification in gearboxes. Statistical features
of inertial vibration data were analyzed in a convolutional neural network to distinguish
between 12 discrete gear and bearing fault combinations. Although this approach performs
well on the selected fault patterns (96.8%), it is limited to a number of discrete fault
combinations (e.g., single gear failure, pair of gears failure). Our approach isolates each failure
as an independent class and thus does not require a machine failure to occur in a certain
combination of multiple faults [32]. With IMMS, we are building a universal sensor network
solution that also uses vibrations as critical signals. IMMS is flexible to apply to various
machinery, primarily focusing on linear and rotational motions. The approach presented in this
paper has a set of unique contributions: (1) a robust hardware design to record inertial sensor
data from 36 locations (2) an effective communication protocol to collect sensor data at 50 Hz
(3) the identification of critical features (4) a database with ten recorded failure scenarios (5)
53

a technique to classify and learn future failure scenarios in a short time and (6) a strategy to
optimize amount, placement, and efficiency of sensors.
4.3 The IMMS Approach
4.3.1 Robust Hardware Design of the sensor network
The network consists of two types of modules: The Sensor Module [Fig. 28] and the Master
Module. The network counts 36 daisy-chainable Sensor Modules that are connected over a
bus system to the Master Module. The sensor network is split in three independent bus
branches to limit bus noise and to provide a stable communication speed of 50 Hz [Fig. 29].

Figure 28: Design of the Sensor Modules to capture machine vibrations

Each Sensor Module consists of an IMU sensor (LSM9DS0), to collect gyroscope and
accelerometer data. The IMU sensor is connected via I2C to an on-board microcontroller
(ATmega328). The Sensor Module operates on stabilized 5V (LD1117AS33TR) and powered via
the 9V supply bus. The RS485 serial communication is half-duplex and performed with the
54

SP3485 IC. The IMU measures inertial motions as unsigned int16 (2 bytes), the Sensor Module
communicates the data bytewise to the Master Board. A total of 12 bytes is transmitted (3x
acceleration, 3x gyroscope) in each measurement cycle.
The Master Module collects data of all 36 Sensor Modules and forwards it to the PC for further
processing. We use the Arduino Due which is equipped with four UART (Universal
Asynchronous Receiver-Transmitter) ports as well as a native USB port that provides sufficient
bandwidth for the transmission of all sensor data. Three UART ports receive data from the
three sensor branches (250kHz Baud Rate), while the USB port forwards data to the PC
(4,000kHz Baud Rate).
4.3.2 Communication Protocol
Every 20 ms (50Hz), the Master Module collects IMU data of all 36 Sensor Modules (432 bytes),
adds a measurement timestamp (4 bytes), combines all data to a large package (436 bytes)
and passes it on to the PC. To collect the data efficiently, each Sensor Modules is programmed
with a unique ID. In every 20 ms cycle, the Master Module broadcasts a measurement
command. All Sensors Modules synchronously collect IMU data and consecutively send it to
the Master in ascending ID order. The serial transmission standard is RS485. The bus consists
of four wires: Two communication lines, 9V supply, and GND.
55

Figure 29: Wiring architecture of the sensor network. The green boxes represent Sensor Modules
with unique IDs

4.3.3 Identification of critical features
To reduce the dimension of the sensor data and to prepare it for classification, features are
extracted from the data. In this paper, we are testing our sensor network on a linear axes
machine and only select the accelerometer sensor data for feature extraction. Accelerometers
measure particularly linear motions and suit well for machines with movements along linear
axes. For machines with rotational motions, we recommend using gyroscope data as well. For
our experiments, we tested two approaches: (1) Statistical Features and (2) Vibration
Frequency Diagrams. Statistical Feature Vectors represent the sensor data as a vector of
length 1008. Each vector includes statistical features such as mean, kurtosis, skewness,
variance, range, min, max, interquartile range, median absolute deviation, median, and most
frequent value of the sensor data. Vibration Frequency Diagrams represents the sensor data
in the frequency domain with the Fast Fourier Transformation (FFT). The data is stored in a
matrix in which the vertical axis represents frequencies in the range between 0 and 25 Hz. In
consideration of the Nyquist sampling theorem, the maximum measurable vibration is 25 Hz
56

(half of the 50 Hz sensor sampling rate). The horizontal axis describes the numbers of sliding
FFT-windows. Each FFT-window represents a FFT over sensor data of fixed length. The color of
each cell in the Diagram describes the vibration frequency occurrence. Brighter colors mean
higher occurrence (FFT amplitude) [Fig. 30].

Figure 30: FFT-Diagrams describe vibration frequency occurrence over time. A 224 seconds
dataset was described with 112 FFT-windows of each two-second sensor data. This FFT-Diagram
displays the class “Tool scratches on surface” [Table 1].

4.3.4 Creating an extendable database
IMMS must have information about the failure scenarios of the machine, in advance, to classify
them. Therefore, ten failure or degraded machine state scenarios were created under real-
world settings, and sample data was recorded and stored in the IMMS database. This database
can easily be extended by recording sensor data of new failure classes.
57

4.3.5 Classification technique for machine failure
We tested two classifiers with the IMMS database. First, we trained a Support-Vector-Machine
in six different Kernel variations. Second, we measured the performance of a Feedforward
Neural Networks with various Hidden Layer Sizes.
4.3.6 Optimization Strategy for sensors
IMMS uses a greedy algorithm to identify the subset of sensors that lead to the best
classification result. This algorithm was applied to four independent sections of a machine test
program. This optimization approach provides information on required number of installed
sensors and placement location. It also gives suggestions on how a machine test program
should be composed for the optimized sensor cluster and can be optimized for duration.
4.4 The Measurement in Detail
4.4.1 Overview
The architecture of IMMS consists of three major processing steps [Fig. 31]. In Step 1, the
Master Module collects the IMU data of all Sensor Modules, combines it to one large package
and forwards it with a measurement timestamp to the PC. Step 2 prefilters the sensor data
and prepares it for feature extraction (Statistical Features or FFT Diagrams). All features are
stored in a feature vector. Step 3 performs classification between 11 classes (normal operation
+ 10 failure scenarios). The feature vector is processed by a classifier (Support Vector Machine
or Feedforward Neural Network). IMMS outputs the result of the classifier that determines if
the machine operates properly or shows one of the 10 abnormal behaviors.

58

Figure 31: Overview of the system architecture

4.4.2 Steps in Detail
4.4.2.1 Collection of IMU data
The Master Module collects 10,800 raw sensor values per second. For each collection cycle,
the Master Module broadcasts a measurement command to all three sensor branches. Once
received, each Sensor Module collects and locally stores the latest IMU data in its
microcontroller. After all Sensor Modules processed the measurement command, the
transmission process of the IMU data to the Master Module begins [Fig. 32].

Figure 32: The Master Module initiates a 20ms data collection cycle

59

All sensor modules have a unique identity (ID), and know their place in the transmitting queue,
which is being processed in ascending ID number. Each Sensor Module is counting the number
of transmitted bytes in the branch. Once it is the turn of the Sensor Module in the queue, it
releases the data and waits for the next measurement command of the Master Module to
arrive. Each cycle takes 20 ms. To guarantee a sampling rate of 50 Hz, the Master Board has
mechanisms to compensate for transmitting errors. In case the Master Module does not
receive all 432 bytes (36x12 bytes) of sensor data, it stops the cycle after waiting for 15 ms and
transmits only the received data to the PC. Should the transmission of data between the
Master Board and PC last longer than 5 ms, it interrupts the transmission and starts a new data
collection cycle on-time. Missing data is repaired or interpolated during prefiltering in Step 2.
4.4.2.2 Feature Extraction
In this paper, we record datasets of 224 seconds per class. This amount is equal 1.2M data
points of accelerometer data. We reduce the amount of sensor data significantly by extracting
features in two ways: (1) Extracting statistical features from the dataset reduces the sensor
data to 1008 data points. (2) FFT Feature Diagrams varies depending on the FFT-window size.
The amount of data points for FFT diagrams ranges from 350 (16 sec window) to 28,000 (0.2
sec window). To proceed with classification, FFT Feature Diagrams of size [N x M] will be
converted into one-dimensional FFT Feature Vectors of length N x M.
4.4.2.3 Classification
IMMS uses the extracted feature vector to detect and classify a potential machine failure. Two
classifiers were tested: First, we trained a Support-Vector-Machine (SVM). This classifier tends
to perform well for classification task in high dimensions. We tested 6 different Kernel types:
Linear, Quadratic, Cubic, Fine Gaussian, Medium Gaussian, and Coarse Gaussian. Second, we
used a Feedforward Neural Network. The Hidden Layer was modified and tested for sizes
60

between 1 – 100 Hidden Neurons. The output of the Network contained 11 Neurons, one for
each possible outcome class [Fig. 33].

Figure 33: The architecture of the Feedforward Neural Network

4.5 Test Results
4.5.1 Test Machine
To record real-world performance data, a linear axis 3D Printer was equipped with 36 IMMS
Sensor Modules. The sensors were placed in 3D-printed holders and epoxy-glued on the
machine [Fig. 34].
61

Figure 34: The testing machine is a 3D printer with motions along three linear axes

Locations and orientations of sensors are selected randomly but include all moving
components of the machine (X, Y, and Z axis). First, the 3D Printer was operated in normal
operation to record regular operation sensor data. Then, components were altered and
modified to create sensor data of 10 common machine failure scenarios [Figure 35, Table 1].
62

Figure 35: Altered machine components to record data of failure classes (as listed in Table 1)

For our experiments, we chose failure scenarios that not only occurred on 3D Printers, but can
be found on many linear axes equipment with prices ranging from low-cost consumer
equipment to expensive industrial machines. This includes machines for additive and
63

subtractive manufacturing (e.g. CNC, Router, 3D Printers), cutting (e.g. Waterjet, Laser Cutter),
or pick-and-place (e.g. electronic component placement machines).

Table 1: Common failure scenarios of a 3D Printer

4.5.2 Test Program
We composed and installed a test program for IMMS on the 3D printer to create motions along
all linear axes with varying speed and length. The total runtime is 224 seconds, but the program
can be described to pass four characteristic sections, each with a length of 56 sec [Fig. 36].
64

Figure 36: Test Program of 224 sec split into 4 sections. Each color shows the data of individual
Sensor Modules

The four sections can be described with a set of unique motion directions and frequencies
[Table 2].

Table 2: Composition of a machine program to cover various frequencies and motion directions

The total collected sensor data for the test program is 4.88 Mbytes. For each of the 11 classes,
the test program was recorded 30 times over multiple days and in varying failure order. With
a total of 330 test program recordings, we created a data package of 1.61 Gbytes. For our
experiments, we are only using accelerometer data (0.81 Gbytes). The acceleration data was
prefiltered, normalized and stored in double-precision format (8 bytes per number) which
65

created a total amount of 3.21 Gbytes acceleration data. This data package was split in 2/3
training data and 1/3 testing data.
4.5.3 Performance Tests
4.5.3.1 Classification Test (Statistical Features)
In our first experiment, we analyzed the performance of statistical features. Two scenarios
were tested: (1) Data from all 36 Sensors and (2) Data from four sensors, of which three were
attached to moving machine components (X, Y, Z axis) and one was placed on the inner metal
case.
For scenario (1), the Coarse Gaussian Kernel performed best with 27.3% error among all SVM
classifiers. The Feedforward Neural Network achieved a similar result. For scenario (2) the
overall performance error of SVM and Neural Network increased. This is an indication that
sensors attached to moving machine components are not sufficient enough to properly classify
all failure scenarios [Table 3].

Table 3: Failure Classification via Statistical Features

66

4.5.3.2 Classification Test (FFT Features)
First, FFT-feature vectors from various FFT-diagram sizes were created. FFT-windows ranged
from 0.2 sec. (Diagram dim. 25x1120) to 16 sec. (Diagram dim.: 25x14).
The SVM classifier achieved best results (4.5% error) using the Coarse Gaussian Kernel with 0.5
sec windows. Linear, Quadratic, and Cubic Kernels performed lower and scored slightly below
10% error. Fine and Medium Gaussian Kernel stayed above 25% error and are not listed in [Fig.
37].

Figure 37: Performance of the SVM Classifier using FFT Features for various window sizes

The best performing window size for the Forward Neural Network is between 0.5 sec - 4 sec.
The classifier achieved errors ranging between 0.7% - 0.0%. The 2 sec window with either 28,
31, 36, 38 or 42 Neurons in the Hidden Layer was best performing [Fig. 38].
67

Figure 38: Feedforward Neural Network classification error using various window sizes

In summary, it turns out that the Feedforward Neural Network is superior over the SVM
classifier for window sizes of 8 sec and shorter [Table 4].

Table 4: Best performing SVM and Feedforward Neural Network parameters

68

4.5.3.3 Optimization Strategy
In this experiment, the goal is to find a subset of highest performing sensors. Therefore, a
greedy algorithm is used to eliminate the least performing sensors. We temporarily remove
one sensor from a set of N sensors and measure the classification performance of the subset
N-1. A total of N subsets need to be tested. The temporarily removed sensor of the best
performing subset will be eliminated, because it contributes least to the classification success.
The algorithm starts with N = 36 and is repeated until N = 1. This greedy algorithm might not
strictly give us the best performing subset of sensors because we are only eliminating one
sensor at a time. However, the algorithm generates u-shaped performance curves [Fig 39].
When initially removing sensors from the set, the performance continuously improves until a
turning point is reached. At this point, the greedy algorithm minimized the error and all
redundant sensors have been removed (e.g. sensors with redundant vibration measurements,
because of similar placement location). After passing the turning point, the error increases
again. This is because valuable sensor data is now being removed that is necessary for reliably
distinguishing between failure scenarios. The experiment was performed on sections (half
duration, quarter duration) and the whole test program. The Feedforward Neural Network was
used with a best performing hidden layer size of 42 Neurons.
69

Figure 39: Applying a Greedy Algorithm to find the best performing subset of sensors. Section 4 is
not displayed, since it only reached a performance of 7.65% error

Table 5: Classification results of best performing subsets

This optimization strategy shows that machine failures can reliably be detected by a small
subset of sensors (four to nine). Below is a picture showing the placement location of the
subset sensors of section 3 [Fig 40]. We observed that sensors of the best performing subset
are commonly found on the inside and outside of the sheet-metal case of the 3D printer, but
not necessarily on moving components. It also turned out that the highest performing sections
of the test program contained combined motions over all machine axes. This observation
should be considered when designing further machine testing programs. This experiment also
70

proved that sufficient performance can be achieved with test programs lasting less than 1
minute. Because our neural network is very small, it could easily run on a low-cost embedded
processor.

Figure 40: The subset of best performing sensors consists of sensors placed on the outside case (5
red arrows), inside case (3 white arrows) and the lid (1 orange arrow).

4.6 Conclusion
IMMS showed that inertial sensor networks can be successfully used to reliably detect and
classify between 10 machine failures or degraded states of operation. With an increasing
number of IoT sensors entering the manufacturing floor, predictive maintenance of industrial
equipment guarantees higher efficiency and productivity. By providing valuable failure
classification and early detection of machine degradation, we hope that IMMS made a
contribution to the development of smart monitoring systems to help minimizing machine
downtime.

71

5. Inertial-based Motion Capturing and Smart Training System
5.1 Abstract
Smart coaching platforms are emerging which combine Body-Sensor-Networks with AI-based
training software to monitor and analyze body motions of athletes, workers, or medical
patients. This allows for new opportunities to explore algorithms to interpret body sensor data
and provide analytical feedback for learning a physical task, refining body motions, or to
protect from work-related injuries. This paper presents a solution to non-invasively equip a
person with sensors of a Smart Training System (STS) to improve training efficiency during
sport activities. Our system calculates the significance of each body part during physical
activities and provides targeted feedback on which body locations are under-performing. In
experiments, the system collected data from 13 inertial sensors attached to the entire body of
inexperienced golf learners. Us-ing an indoor golf training net with a central target with 3 con-
centric zones, 1,080 real-world golf swings of 11 participants were analyzed. During the first
30 swings of each participant, the system learned distributions of motions from each sensor,
conditioned on swing performance reported by users from their hitting location on the target.
In the later 70 swings, feedback was provided to a subgroup of 8 participants, by computing,
for an optimal set of features determined during training, the largest discrepancy. The
remaining 3 (control) participants received no feedback. From only 100 golf swings for each
par-ticipant, our system led to significantly improved scores by on average 3.7x (t-test, p
<0.0001) over the latter 70 swings. Our results suggest that the combination of motion sensors
and processing developed here was able to yield significantly im-proved golf swing training.
Video Reference: https://www.youtube.com/watch?v=ZZZ6Wh27yp0
Submitted in: 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)
72

5.2 Introduction
Smart wearable device sales will double by 2022 worldwide, becoming a $27 billion+ market
with 233 million unit sales according to industry analyst CCS Insight (2018) [33]. Monitoring
human body parts will be more accessible than ever before. In addition, full-body haptic smart
suits are entering the market that allow users to receive tactile information through their skin
via electrostimulation or vibrotactile actuators [34][35][36]. Their fabrics are stretchable,
breathable, washable and durable and can be used in a wide range of applications, particularly
in healthcare, sports and industry. With the growing accessibility of full-body motion data and
availability of novel feedback suits, algorithms for smart trainer systems need to be developed.
These systems will process sensor data and provide targeted instructions to hu-mans. These
virtual trainer systems will effectively help learning physical tasks, refining body motions, or
protecting employees from work-related injuries.
Smart trainer systems could be particularly important during technical refinement to an
already long practiced, well-established motion skill. A team of researchers from the University
of Central Lancashire (UK) has shown that when “a [golf] performer focuses their attention on
one aspect in isolation, that aspect would be constrained and re-duce in variability across
trials” [37]. A detailed instruction on a specific body part e.g. “focus on your left arm before
you hit the golf ball” will lead to a more consistent arm rotation and thus to a more uniform
club swing. To prove their hypothesis, the team equipped 3 golf professionals with 17 IMU
sensors (inertial measurement units) and recorded golf swing motion data. Motion variability
was reduced when the golf players focused their conscious motion control on a specific aspect
of their swing technique. In this work, we not only encourage our subjects to focus on a body
part, but also provide goal-directed feedback (e.g. “rotate your left arm faster before you hit
the golf ball”), not only to reduce variability of the golf swing, but also to improve swing
73

performance. Our assumption is that goal-directed feedback could be applied to many more
sport applications: Swinging tennis rackets or baseball bats, kicking soccer balls, throwing
basketballs, and many others. Real-time posture correction and motion change instructions
could ultimately optimize training routines, bridge gaps of human trainer absence, and monitor
humans for training injuries caused by excessive motions and bad postures. Besides sports,
smart trainer systems could be used by humans in workplace environments with high risk of
injuries or negative long-term health effects. Overexertion injuries is the most common and
most expensive workplace injury related to pulling, lifting, pushing, holding, carrying, and
throwing activities at work [38]. The resulting musculoskeletal disorders, such as strains or
sprains, account for 33 percent of total occupational injuries according to a 2015 news release
by the Bureau of Labor Statistics of the U.S. Department of Labor. A feed-back-based system
could remind workers to correct their posture when moving heavy items, and could avoid
overexertion of certain body parts. This concept of injury prevention could also be applied in
rehabilitation. A smart trainer system could, e.g., guide a stroke patient through the daily
recovery exercises by automatically adjusting motion feed-back to the patient’s healing
progress. In case of an abnormality, the system could modify the rehabilitation therapy in real-
time and notify the doctor with analytical body motion data.
Human coach feedback is often subjective, and could sometimes be faulty from observation
errors (e.g., hidden body parts), false assessment of a person’s skill, the inability to track all
body motions at the same time, or lack of attention. Smart Training Systems, however, will
reliably keep all body parts monitored, identify a person’s capabilities by analytical data, and
provide quantitative, objective feedback. However, Smart Training Systems are still in an early
development stage. Commercially available solutions are often limited in the complexity of
motion patterns they can analyze. The AI coaching platform company ARDA has developed a
74

smart training software that monitors and interprets sports activities to refine a training
routine. The ARDA platform fuses inputs from multiple data sources (accelerometer,
barometer, GPS, heart-rate monitor) and, e.g., notices if strides are too long. It then will
provide actionable advice to athletes such as “you are pushing too hard and are now showing
signs of muscular fatigue”. However, ARDA currently only supports sports activities with
repetitive motion patterns, such as running, cycling and walking [39]. Trajectory tracking,
segmentation and interpretation for complex body motions such as tennis or golf are not yet
supported and require complex software architectures for interpretation of the sensor data,
often with domain-specific features. Smart Training Systems often lack detailed feedback for
athletes to refine their body motions. A team from the University of Science and Technology
of China has developed an IMU-based Online Serve Performance Evaluation System for tennis
players [40]. The tennis player is equipped with two IMU sensors (serving wrist and lower leg).
A Hidden Markov Model (HMM) is segmenting the serving process into eight stages. Unique
motion features relating to rhythm, gestures, and power are calculated for a regression model
to calculate a performance score. In this case, however, the athlete needs to interpret the
performance score without any actionable advice and change her/his body motions by trial-
and-error to achieve a higher performance score. To complement a smart training platform,
detailed feedback to the athlete is required to refine body motion. The evaluation system
would need to analyze previous score results of the tennis player to provide further guidance
on how to improve the body motions during servings. Smart Training System need to learn
from small datasets. Datasets for athletes are relatively small. Sharing sensor recordings from
individual athletes and combining data of multiple training sets is difficult, due to the individual
body characteristics of each subject, and variability in sensor placement from day to day. In
the referenced literature, the average datasets to measure performance or train classifiers
75

consists of, e.g., 20 golf swings [37], or 86 tennis serves [40]. Training a virtual coach requires
a fast-learning, well generalizing architecture. Deep-learning approaches are less suit-able in
this case, and alternative fast learning architectures need to be explored and tested. Optimal
body motions vary from athlete to athlete. Coaching experts seem to have no unanimous
consensus on the trajectory and body motions for a perfect golf club swing, and different
textbooks on the topic provide a wide range of recommendations. Therefore, professional golf
players show a very diverse body motion profile, e.g., golf club swings in one plane vs. multi-
planes [41]. The selected swing trajectory will ultimately depend on the individual body
structure, training technique, and possibly other factors [42]. Thus, transferring training
profiles and comparing sensor data of 3rd party athletes might not lead to an improvement of
the trainee’s performance.
In this paper, we introduce STS (Smart Training System). STS is a universal sensor network
solution to train and refine body motions with detailed feedback targeting underperforming
body parts. STS can be applied to various training routines, covering linear and rotational body
motions. The approach presented in this paper has a set of unique contributions: (1) a robust
hardware design to record inertial sensor data from 13 body locations, (2) an effective
communication protocol to collect sensor data at 95 Hz, (3) an optimized placement location
for all sensors, (4) a robust seg-mentation algorithm to detect critical body motions, (5) the
identification of critical features, (6) an effective technique to calculate feedback from limited
available training data, and, (7) an audio-based feedback system to provide coaching advice
for body motion refinement.

76

5.3 The STS Approach
5.3.1 Hardware Design of the IMU Body Sensor Network
The IMU Body Sensor Network consists of two modules: The Sensor Module [Fig. 41] and the
Master Module. The hardware used in this work has improved features (Version 2), and it is
based on the 2016 sensor network that was developed to monitor machines for automated
failure detection [43]. All sensors in the network are connected to only one bus to optimize for
cable length and weight [Fig. 42].

Figure 41: Sensor Module to capture body motions (V2)

Each Sensor Module consists of the latest IMU sensor (Version 1: LSM9DS0, Version 2:
LSM9DS1), to collect gyroscope and accelerometer data. The IMU sensor is connected via I2C
to an on-board microcontroller (ATmega328) with stable clocking (Version 1: Internal
Oscillator, Version 2: External crystal oscillator with improved clocking precision to support
higher UART communication speed). The Sensor Module operates on stabilized 3.3V
77

(LD1117AS33TR) and powered via the 9V supply bus. The RS485 serial communication is half-
duplex (SP3485 IC). The IMU measures inertial motions as unsigned int16 (2 bytes), and the
Sensor Module communicates the data bytewise to the Master Board. A total of 18 bytes are
transmitted (3x acceleration, 3x gyroscope, 3x magnetometer) in each measurement cycle.

Figure 42: Wiring architecture of the sensor network. The yellow circles represent Sensor Modules
with unique IDs

The Master Module collects data of all 13 Sensor Modules and forwards it to a PC for further
processing. We use the Arduino Due which is equipped with four UART (Universal
Asynchronous Receiver-Transmitter) ports as well as a native USB port that provides sufficient
bandwidth for the transmission of all sensor data. We only use one UART port to receive data
from the body sensor chain (1 Mbps), while the USB port forwards data to the PC (480 Mbps)
[Fig. 42].

78

5.3.2 Communication Protocol
Every 10.5 ms (95 Hz), the Master Module collects IMU data of all 13 Sensor Modules (234
bytes), adds a measurement timestamp (4 bytes), concatenates all data to a large packet (238
bytes) and passes it on to the PC. To collect the data efficiently, each Sensor Module is
programmed with a unique ID. In every 10.5 ms cycle, the Master Module broadcasts a
measurement command. All Sensors Modules synchronously collect IMU data and
consecutively send it to the Master in ascending ID order. The serial transmission standard is
RS485. The bus consists of four wires: Two communication lines, 3.3V supply, and GND.
5.3.3 Optimized Placement of sensors
Sensors are placed on all moving body parts to capture motions of the entire human body. The
13 sensors are placed on: 1x head, 1x back, 2x upper arm, 2x lower arm, 1x left hand, 2x thigh,
2x leg, 2x shoes [Fig. 42] [Fig. 43]. The right hand is holding the golf club along with the left
hand and is performing redundant motions. We intentionally did not equip the golf club with
sensors, because our research goal was to develop a generic approach to refine body motions
independently from the used tool (e.g. golf club, racket, bat, etc.).

79

Figure 43: All 13 moving body parts of the golf player are equipped with sensors

5.3.4 Segmentation of critical swing phases
STS records datasets of 10 seconds per swing. To use the sensor data for further processing,
critical motion phases and timestamps need to be extracted to evaluate the golf swing. For this
reason, the captured sensor data is segmented into 3 phases (Backswing, Swing before impact,
Swing after im-pact) as well as 4 critical timestamps which describe the beginning, transition
and end of these phases (Start of Backswing, Start of Swing, Impact, End of Swing) [Fig. 44].
For validation of the segmentation algorithm, the golf swings are captured by a high-
speed camera (240 Hz) and synchronized via a displayed Aruco-Marker on the PC Screen during
sensor data recordings [Fig. 45]. The average time for recorded golf swings is approximately
1.5 seconds. The segmentation process discards the non-relevant sensor data (85%) that is not
part of the actual golf swing
80

Figure 44: Segmentation of a golf club swing by dividing the gyro-scope data into 3 phases and 4
timesteps

5.3.5 Identification of critical features
Once the data of the critical phases and timestamps has been extracted, features are
calculated for each phase and timestamp to evaluate the swing performance. For this work,
features selection focuses on capturing mainly rotational motions from the gyroscope to
optimize for the rotational nature of golf swings. Other sports (e.g., basketball throw) might
involve more translational motions captured by the accelerometer and the feature selection
would have to be adjusted accordingly. In STS, features extracted for all 3 golf swing phases
consist of rotational speed and rotational angle range. STS also calculates a tilt angle of each
81

body part to measure the body posture during two timestamps: “Backswing Start” and
“Impact”.
5.3.6 Effective training of virtual coach with small datasets
STS identifies body parts that play a critical role in successful golf swings. This allows feedback
to only be targeted towards body locations with highest need of improvement. To identify the
timing (which phase or timestamp) as well as the motion type (motion change, posture
change), current swing features will be compared to a set of optimal motion features [Fig. 45].
This set is estimated by calculating a mean weighted average of features from previous swings.
Weights are adjusted according to previous performance scores which are manually recorded
during each swing [Fig 46].

Figure 45: Rotational Speed of the left arm during a golf swing. The difference between the latest
swing and the weighted mean of previous swings will affects the selection of feedback
82

In our experiments, scores were obtained
directly from the participants and checked
by the experimenter. They measured how
well the participant was able to hit the
center of a bullseye target. Hence, with this
scoring system, participants were to
improve in their aiming accuracy. Different
scoring measures could be used to achieve
different training goals (e.g., one could score
shot distance in yards to help participants
improve on their driving range), without
modifications of our approach. The optimal motion features are re-calculated after every swing
and will only take the previous 30 swings into account. Therefore, STS dynamically adjusts to
positive or negative training progress and always provides detailed feedback on relevant body
parts.
5.3.7 Audio-based feedback system
STS has 26 pre-recorded feedback components: 5 time components (3 phases and 2
timestamps), 2 motion type components (4 x motion and 2 x posture), and 13 body parts. STS
combines these components to compose and deliver verbally a valid feedback command.

Figure 46: The golf swing experiment setup
83

5.4 The Measurement in Detail
5.4.1 Overview
The architecture of STS consists of 3 major processing steps [Fig. 47]. During the sensor data
recording process, the Mas-ter Module collects the IMU data of all 13 Sensor Modules,
combines it to one large packet and forwards it with a measurement timestamp to the PC. Step
1 prefilters the sensor data, segments critical motion timestamps and motion phases, and
extracts features.

Figure 47: Overview of the System Architecture

In Step 2, STS adds the latest motion features and performance score to its memory of all
previously recorded data. The Optimal Motion Feature Estimator uses current and previous
performance and swing motion data to calculate an optimal motion feature set. This set will
be used by the Critical Body Parts Classifier to identify important and less important body parts
for future feedback instructions. In Step 3, the feedback creator algorithm selects feedback
components from a pool of generic instructions. It merges them together to feedback
commands that were suggested by the critical body parts classifier to improve the golf swing
84

motion. The feedback trigger will decide whether to provide feedback to the subject or not
based on previous performance and feedback score.

5.4.2 Steps in Detail
5.4.2.1 Collection of IMU data
The Master Module collects 11,115 raw sensor values per second. For each collection cycle the
Master Module broadcasts a measurement command to all three sensor branches. Once
received, each Sensor Module collects and locally stores the latest IMU data in its
microcontroller. After all Sensor Modules processed the measurement command, the
transmission process of the IMU data to the Master Module begins [Fig. 48].

Figure 48: The 10.5 ms data collection cycle

All sensor modules have a unique identity (ID), and know their place in the transmitting queue,
which is being processed in ascending ID number. Each Sensor Module is counting the number
of transmitted bytes on the bus. Once it is the turn of the Sensor Module in the queue, it
releases the data and waits for the next measurement command of the Master Module to
85

arrive. Each cycle takes 10.5 ms. To guarantee a sampling rate of 95 Hz, the Master Board has
mechanisms to compensate for transmitting errors. In case the Master Module does not
receive all 234 bytes (13 x 18 bytes) of sensor data, it stops the cycle after waiting for 7.5 ms
and transmits only the received data to the PC. Should the transmission of data between the
Master Board and PC last longer than 3 ms, it interrupts the transmission and starts a new data
collection cycle on-time. Missing data is repaired or interpolated during filtering in Step 1.

5.4.2.2 Segmentation of Swing Motion
The segmentation process is solely performed on the gyroscope data because of the rotational
nature of golf swings. First, the absolute sum of all gyroscope sensor values is created and
median filtered, resulting in a one-dimensional vector. This preparation step increases the
segmentation reliability by minimizing the effect of outliers, error measurements, and falsely
classified minima. Second, the seg-mentation process begins by searching for the highest gyro-
scope activity, which represents the “Impact”. From this anchor point, the closest local minima
before and after the impact describe the “Swing Start” and “Swing End”. The “Backswing Start”
is located by finding the closest mini-mum prior to local minimum “Swing Start” that exceeds
a minimum gyroscope activity threshold [Fig. 49].
86

Figure 49: Segmentation of a golf swing on gyroscope data

This segmentation process is ideal for sports activities that involve an impact swing with an
optional backswing, e.g., golf, tennis, soccer. For other physical tasks, the segmentation
algorithm needs to be adjusted accordingly to identify critical phases and timestamps.
5.4.2.3 Feature Extraction
The feature set F extracted from the segmented data contains three components [Fig. 50].

Figure 50: The extracted feature set from the motion data

87

The rotational speed feature is measured along all 39 sensor axes (3 local axes on 13 sensors)
over all 3 segmented phases. The phases have different length from swing to swing, therefore
the rotational speed trajectory is time-warped to a fixed length of 400 timesteps for a full
recording (200 timesteps for the backswing, 100 timesteps each for the swing before and after
impact). The rotational angle range feature is also measured long all 39 axes over all 3
segmented phases. The tilt angle feature is measuring the sensor deviation from a global axis
that aligns with the vertical gravity vector. It describes the posture during 2 critical timestamps:
the “Backswing Start” and the “Impact”. This tilt angle is measured by an accelerometer and
stabilized with gyroscope and magnetometer data (optionally). We implemented the
MadgwickAHRS filter for fusion of all sensor data [44].
5.4.2.4 Optimal Motion Feature Estimator
The performance of each golf swing is measured by comparing its features F current to a set of
optimal features F optimal. F optimal is calculated by the weighted mean of features. Direct target
hits have highest weight, close target hits have lower weights and swings hitting far-off the
target have zero weight [Fig. 46]. During a training session, the optimal features are calculated
from data limited to the last 30 swings.
𝐅 𝐨𝐩𝐭𝐢𝐦𝐚𝐥 =
1
∑ w
Target
𝑖 𝑛 𝑖 =1
∙ [∑w
Target
𝑖 ∙ 𝐅 Target
𝑖 𝑛 𝑖 =1
] (4.1)
This number of swings is sufficient to approximate an optimal set of features, and to also
quickly adjust the optimal feature set to positive or negative training progress. F optimal is
calculated for all 3 feature components and requires that not all swings were off-target [Eq.
4.1][Fig. 51].
88

Figure 51: Example of the Rotational Speed Feature component of Foptimal

5.4.2.5 Critical Body Parts Classifier
Each sensor represents a body part. The goal of this classifier is to figure out which body parts
play an important role in swings that hit the target. Therefore, the last 30 recorded feature
sets are split into two groups: F off contains all m feature sets with swings not hitting the target.
F on represents all n feature sets with swings hitting one of the three target circles in our
experiments [Fig. 46]. The distance between F optimal and F off was calculated and stored in ΔF off
[Eq. 4.2]. The distance was calculated for F optimal and F on and stored in ΔF on accordingly [Eq. 4.3].
Next, we calculated the critical value array V critical that is the quotient of the variance ΔF off and
variance ΔF on [Eq. 4.4]. The values in V critical of the 39 rotational axes are clustered by sensor
and only the maximum value per sensor per feature is selected. The dimension of V critical is 13
body parts x 8 ratio dimensions. The 8 ratio dimensions consist of on/off variance ratios for
rotational speed for each of the 3 swing phases, for angular range for each of the 3 phases,
and for tilt angle at two critical time points: beginning of backswing and impact. The values of
89

V critical explain how much of a role each body part plays during swings during each phase of a
swing and at each critical each timestamp.
Δ𝐅 𝐨𝐟𝐟 = [𝐅 𝑜𝑝𝑡𝑖𝑚𝑎𝑙 − 𝐅 𝐨𝐟𝐟 𝑖 ]
𝑖 =1
𝑚 (4.2)
Δ𝐅 𝐨𝐧
= [𝐅 𝑜𝑝𝑡𝑖𝑚𝑎𝑙 − 𝐅 𝐨𝐧
𝑖 ]
𝑖 =1
𝑛 (4.3)
𝐕 𝐜𝐫𝐢𝐭𝐢𝐜𝐚𝐥 =
var(Δ𝐅 𝐨𝐟𝐟 )
var(Δ𝐅 𝐨𝐧
)
(4.4)

Two cases are analyzed:
Case 1 (V critical ≤1 ): Body part does not play an important role in distinguishing between swings
with target hits and no target hits, because it did not deviate from optimal significantly more
in off-target shots compared to on-target shots. Any feedback provided for this body part will
likely not affect the performance.
Case 2 (V critical>1): Body part plays an important role in distinguishing between target hits and
misses, because its variability from optimal was higher for missed shots than for hits. Feedback
provided for this body part will likely affect the performance by reducing the variability of this
body part from optimal.
The 13 values of the rotational speed ratio dimension of V critical are presented for all 3 swing
phases in Figure 52.
90

Figure 52: The higher the value, the more important is the body part for swings that hit the target.
The displayed values represent the Vcritical for the rotational speed feature.

5.4.2.6 Feedback Creator and Trigger
Feedback is created by combining a time element, a motion type element, a body part, and a
motion direction. A total of 208 feedback combinations are possible [Fig. 53].
91

Figure 53: The pool of generic instructions shows all possible feed-back combinations

The information about critical body parts acquired in the previous step plays an important role
in selecting effective feedback. First, the distance of the current swing features and the optimal
features are calculated for each sensor and stored in FB Score_i for each swing i. The maximum
feedback score FBMax_Score is calculated by adding up the elements of FB Score_i of the last 3 swings
with n as the latest swing. This represents the body part that had the highest deviation from
the optimal feature set over the last 3 swings. If indeed the critical body parts classifier
identified this body part as important, the feedback will be passed on to the feedback trigger
module [Eq. 4.5]. If not, the second highest FB Max_Score that was classified as important is
selected.
𝐅𝐁
𝐌𝐚 𝐱 𝐒𝐜𝐨𝐫𝐞 = max[∑ 𝐅𝐁
𝐒𝐜𝐨𝐫𝐞 𝒊 𝒏 −𝟐 𝒊 =𝒏 ] (𝟒 .𝟓 )
92

The feedback trigger only provides feedback if the latest swing did not perfectly hit the highest
target (the red circle in our experiments). The Feedback is provided as an audio output. All 26
elements in the pool of generic instructions have been voice-recorded. The elements are
appended to an audio stream according to the outcome of the FB Max_Score.

5.5 Test Results
5.5.1 Test Setup
To record real-world performance data with and without system feedback, 8 human golf
novices were equipped with 13 IMU Sensor Modules. For reference, 3 additional sub-jects
performed golf swings without the sensor equipment attached to their bodies. The sensors
were glued into 3D-printed holders, epoxy-glued on anti-slipping sport bands and strapped
around 13 body parts. Placement of the sensor did not need be accurate or reproducible,
thanks to our algorithmic approach described above. However, it was important to limit
slippage during a particular session (hence the use of sports bands). Further, no calibration was
necessary. Participants could start practicing as soon as the sensors and computers were on.
None of the subjects had previous golfing experience at the time of the recording. All subjects
were right-handed. In our experiments, we focus on learning and refining swing body motions
for short-distance golf swings. A golf practice net was setup with three target circles. The
hitting location in these circles represent the level of accuracy of the golf swing: The red circle
(3 points), the white inner circle (2 points) and the white outer circle (1 point) [Fig 46]. If the
ball did not hit any circle, no points were given. The subject was standing approximately 4
meters in distance to the target.

93

5.5.2 Dataset and T-Tests
5.5.2.1 Test Details Overview
Each subject performed 80 – 100 golf swings. 8 of 11 sub-jects received feedback after the 30th
golf swing. The audio feedback was provided via a loudspeaker next to the subject. To
statistically evaluate the recorded datasets, we performed two-sample t-tests on the
performance data [Fig. 57]. We aimed to test two hypotheses: a) Did the system increase
score, overall for our population of participants? b) Did the system increase score for every
participant who used it?
In addition to these two baseline tests, and because some improvement of score may occur
just by virtue of practicing even without our coaching system, we compared the slopes of
cumulative score plots (an indication of expected score on each shot) between the first 30 and
the later swings. This allowed us to evaluate whether participants experienced a higher score
expectation after our system was turned on at the 30th swing.

5.5.2.2 The overall 2-sample t-test
Test Description: The overall 2-sample t-test was performed on two groups: The first group
include subject 1 to 8 (received feedback after swing 30), while the second group includes
subjects 9 to 11 (received no feedback at all). For each group, the performance results were
concatenated in two samples: Sample A contained all scores of the first 30 swings. Sample B
contained all scores after the first 30 swings. The two-sample t-test was performed on Sample
A and B. Our null hypothesis claims that there is no statistically significant difference between
Sample A and B. The alternative hypothesis suggests the opposite.
94

Test Results: The results show a very low p-Value of Group 1 (Subjects 1 to 8) [Fig 54]. This
demonstrates that there is a
statistically significant
difference in score between
swings that received no
feedback (Sample A) and
swings that received
feedback (Sample B). Ultimately, the null hypothesis was rejected, and the alternative
hypothesis was suggested. In comparison, Group 2 (Subjects 9 to 11) showed a high p-Value
which aligns with the expectation that there is no statistically significant difference in score
within the Samples A and B. This aligns with the experiments, in which neither Sample A nor B
received feedback.
Figure 54: The overall t-test results confirm a swing performance
change when receiving feedback
95

Figure 55: Average score per swing shows a significant improve-ment in the second period for
subjects who received feedback vs. subjects who did not receive feedback. On average, score of
participants who received feedback improved 3.7x.

In summary, the overall t-test confirms that swing performance was improved by feedback
from STS. Even though the control participants of Group 2 improved slightly after the 30th
swing, that improvement was too small to be statistically significant; in contrast, the
participants receiving feedback from STS in Group 1 improved much more dramatically and
significantly [Fig. 55].
5.5.2.3 The individual 2-sample t-test
Test Description: The individual 2-sample t-test splits the performance results for each subject
in two samples: Sample A contains the first 30 swings. All following swings are assigned to
Sample B. None of the subjects received feedback on the first 30 swings. However, subjects 1
to 8 received feedback during the following swings while subjects 9 to 11 continued swings w
96

Test Results: For 7 of 8 subjects that received feedback, the null hypothesis was rejected [Fig.
56]. For 3 of 3 subjects that received no feedback, the null hypothesis was suggested. The
results show that the feedback yielded a statistically significant performance difference in
87.5% of cases. In summary, the individual t-test shows that not every individual subject
improves performance from receiving feedback during swings. Hence, it is expected that,
although useful for most people, this system may not work for all people. (Note that Bonferroni
correction is not warranted here since we perform 11 unrelated tests).

Figure 56: The individual t-test shows that there can be a p-Value outlier: Not all subjects who
receive feedback experience a swing performance change.

Possible reasons for the performance outlier (participant 4) are that feedback was not
understood properly by the subject. Another explanation could be that the subject was not
able to put the feedback into practice. As a result, we claim that providing feedback during golf
swings might not guarantee for every individual to experience a change in their swing
performance.
5.5.3 Learning analysis
Every subject with or without receiving feedback during their experiments showed a
performance increase, though it was small and not significant for the control Group 2 and
subject 4. The score curve for each subject, however, varies, as shown by fitting a line to the
cumulative score results for the first 30 swings, another for the later swings, and com-paring
97

their slopes [Fig. 57]. We made multiple observations from the collected data: (1) The
cumulative score curve among subjects receiving feedback is significantly steeper than the
subjects not receiving feedback [Fig. 57]. (2) During the first 30 golf swings, subjects 1 to 8
performed on average lower than the subjects 9 to 11. Subjects 1 to 8 had to get used to
performing golf swings with sensors and cables attached to their body, which may have
lowered their baseline performance. Subjects 9 to 11 did not wear additional equipment. (3) If
the average point gain is too small for the first 30 swings, the task might be too difficult [Fig.
16, Subject 5]. (4) Training improvements sometimes come with slight delay. Subject 2, 6, 8 did
not show instant improvement results when feedback was provided starting at swing 31.
98

Figure 57: Learning results. An increase in slope between the black dashed line (fitted to cumulative
scores of swings 1-30) and the red dashed line (fitted to cumulative scores of later swings) indicates
that enabling the system improved a participant’s performance

99

5.6 Conclusion and Future Work
In this work we demonstrated that body motions for a specific task can be successfully refined
by providing body-part-specific feedback. The feedback in this work is based on analytical data
measured on all moving body parts. Although the feedback represents the highest variation
from an optimal set of body motion features, it is not evaluated on how it is received by a
person. To evaluate the efficiency of a feedback (e.g., do we need to provide this feedback for
every other swing only, does certain feedback show no improvement, etc.), we need to create
a feedback evaluation module. This could be a shallow recurrent neural network that quickly
learns from the previous scored performances and provided feedback and decides if a future
feedback in-deed will lead to an improvement [Fig. 58].

Since this feedback module is only dependent on performance score and feedback given and
does not rely on the same exact sensor placement locations, it could be used throughout
multiple training sessions. By providing a valuable approach on improving the swing
performance of golf players using a full-body sensor network, we hope that STS made a
contribution to the development of Smart Teaching Systems with wearable technology.
Figure 58: A Recurrent Neural Network (RNN) would evaluate if feedback is efficiently received
by the person
100

6. Discussion

The previous four research projects show the development and testing of various applications
with IMUs attached to humans and machines. The last two projects demonstrate the high
flexibility of placement locations and non-invasive designs of IMU sensor networks. Based on
the promising research results, a possible next research challenge could be to explore
capturing IMU data from humans and
machines in parallel. A complex example
is the monitoring of cars [Fig. 59]. Smart
car safety systems analyze data from
IMU networks to ensure proper
functionality of all mechanical car
components but also activate and
deactivate car functionalities based on
normal vs. abnormal driving behavior.
While mechanical failures could be
displayed as warning signals on the
dashboard, the car safety system could
also adjust car setting based on driver
characteristics: Limiting maximum
speed, detecting possible distractions
(e.g. playing on radio buttons), or even
verifying user identity by analyzing acceleration / deceleration patterns. A challenging research
problem is the parallel segmentation of sensor data for classification of driver state and car
Figure 59: IMU sensor network monitor humans
(e.g. driver) and machines (e.g. car) in parallel
101

performance. To solve this complex problem, IMU data could be complemented with sensor
data from additional on-board sensors such as eye-trackers or cameras.
The increasing level of automation in future transportation (particularly self-driving cars and
delivery drones), will substitute human operators and drivers to navigate vehicles, but not their
monitoring skills to recognize degraded states of mechanical components with abnormal
vibrations or noises. This development emphasizes the need for smart monitoring systems that
will need to be an integral part of any autonomous transportation vehicle in future.
The preventive healthcare industry will also fundamentally transform. Sensor suits with
embedded body sensors combined with smart algorithms will enable long-term body
monitoring. Currently, low-cost, energy-efficient high-tech suits with less obtrusive sensor
networks are still in an early development stage. However, sensor suits will continuously
measure our health data by monitoring body activity, temperature, blood pressure or heart
beat in various day-to-day situations and will provide a deeper analysis of our health status
over time rather than a single timestamp health check at an occasional doctor’s visit. A slight
change in body state and motion patterns measured over time by continuous monitoring could
indicate a possible disease. An early phase discovery however might allow an effective
treatment. Large amount of collected data processed with deep learning networks will enable
such complex body analysis. In work safety, employees with physically challenging tasks (e.g.
construction job sites) will be protected from injuries by combining body suit data with
sophisticated data analytics. Motion data of workers will be analyzed in real-time and proactive
feedback will be given to prevent health-related long-term damage from excessive body
movements or false postures.
This variety of application for smart IMU sensor networks comes along with partially unsolved
technical challenges. During my PhD research, I successfully developed and tested systems for
102

non-invasive sensor design with IMU motion sensors, developed algorithms for optimization
of sensor amounts and placements on humans and machines, created effective data fusion
algorithms, identified best feature extraction methods, and selected and parametrized
classifier architectures. By providing valuable approaches during my PhD research on
improving automated situation awareness of humans and machines, I hope that my PhD
research made important contributions to the development of smart monitoring and
autonomous situation classification systems.

103

7. Acknowledgements

I would like to thank my advisor Prof. Dr. Laurent Itti for his strong support and advice over the
years. I would also like to thank all members of the USC iLab and those who have supported,
inspired, and entertained me during my research.
The work in this thesis was supported by the National Science Foundation (grant number
CMMI-1235539, CCF-1317433, CNS-1545089), the Defense Advanced Research Projects
Agency (N65236-12-C-3884), the Army Research Office (W911NF-11-1-0046 and W911NF-12-
1- 0433), U.S. Army (W81XWH-10-2-0076), the Office of Naval Research (N00014-13-1-0563),
C-BRIC (one of six centers in JUMP, a Semiconductor Research Corporation (SRC) program
sponsored by DARPA), Google, and the Intel Corporation.

104

8. References

[1] Google References on Developer APIs for Android, accessed on Feb 28th, 2019:
https://developers.google.com/android/reference/com/google/android/gms/location/DetectedActivity

[2] Technical Datasheet of ICM-20648 IMU sensor, accessed on Mar 15th, 2019:
http://www.invensense.com/wp-content/uploads/2017/07/DS-000179-ICM-20648-v1.2-TYP.pdf

[3] T. Zhang, W. Tang, "Classification of Posture and Activities by Using Decision Trees", 34th
Annual International Conference of the IEEE EMBS , San Diego, CA, USA, 2012

[4] S. Zhang, P. McCullagh, C. Nugent, H. Zheng, "Activity Monitoring Using a Smart Phone's
Accel. with Hierarchical Classification", 6th Intl. Conf on Intellig. Environments, Kuala
Lumpur, Malaysia, 2010

[5] X. Long, B. Yin, R.M. Aarts, "Single-accelerometer-based daily physical activity
classification", 31st Annual Int. Conf of the IEEE Eng. in Med. and Bio. Society (EMBC
2009) 6107-6110 Minneapolis, MN, USA, 2009

[6] L. Bao, S. Intille, "Activity Recognition from User-Annotated Acceleration Data",
PERVASIVE 2004, vol. LNCS 3001, A. Ferscha, F. Matterns, Eds. Springer-Verlag Berlin
Heidelberg, 2004, pp. 1-17

[7] C. Yang, Y. Hsu, "A Review of Accelerometry-Based Wearable Motion Detectors for
Physical Activity Monitoring", Sensors vol. 10, 2010, pp. 7772-7788

[8] M. Lee, J. Kim, K. Kim, I. Lee, S.H. Jee, S.K Yoo, "Physical Activity Recognition Using a
Single Tri-Axis Accelerometer", World Congress on Engineering and Comp. Sc. 2009 Vol
I, San Francisco, USA, 2009

[9] S. Chernbumroong, A.S. Atkins, H. Yu: "Activity classification using a single wirst-worn
accelerometer", 5th IEEE Intl. Conf. on Software, Knowledge, Information Mgmt and
App., Benvento, Italy, 2011

[10] M. Zhang, A.A. Sawchuk, "A Feature Selection-Based Framework for Human Activity
Recognition Using Wearable Multimodal Sensors," Intl. Conference on Body Area
Networks, Beijing, China, 2011

[11] E.H. Spriggs, F. Torre, M. Hebert, "Temporal Segmentation and Activity Classification
from First-person Sensing", IEEE Workshop on Egocentric Vision, Miami, FL, 2009, pp.
17-24

[12] J.R. Kwapisz, G.M. Weiss, S.A. Moore, "Activity Recognition using Cell Phone
Accelerometers", 4th Intl. Workshop on Knowledge Discovery from Sensor Data,
Washington, DC, USA, 2010

105

[13] A. Oliva, A. Torralba, "Modeling the Shape of the Scene: A Holistic Representation of the
Spatial Envelope", Intl. Journal of Comp. Vision 42(3), Kluwer Academic Publishers.
Netherlands, 2001, pp. 145-175

[14] C-W. Hsu, C.C. Chang, C.J. Lin, "A Practical Guide to Support Vector Classification",
Technical Report, Computer Science Department, National Taiwan University, 2003

[15] M. Cook, "Flight Dynamics Principles: A Linear Systems Approach to Aircraft Stability",
2nd Edition, 2007, Butterworth-Heinemann, Chapter: "Angular velocities
transformation", p. 23

[16] S. Beauregard, “A Helmet-Mounted Pedestrian Dead Reckoning System”, 3rd
International Forum on Applied Wearable Computing (IFAWC), Bremen, Germany, 2006,
pp. 1 – 11

[17] Y. Jin, H.-S. Toh, W.-S. Soh, W.-C. Wong, “A Robust Dead-Reckoning Pedestrian Tracking
System with Low Cost Sensors”, IEEE International Conference on Pervasive Computing
and Communications (perCom), Seattle, 2011

[18] N. Roy, H. Wang, R.R. Choudhury, “I am a Smartphone and I can Tell my User’s Walking
Direction”, Proceedings of the 12th annual international conference on Mobile systems,
applications, and services (MobiSys), 2014, PP. 329-342

[19] E. Foxlin, “Pedestrian tracking with shoe-mounted inertial sensors”, IEEE Computer
Graphics and Applications, 2005, pp. 38 – 46

[20] W. Kang, S. Nam, Y. Han, S. Lee, “Improved Heading Estimation for Smartphone-Based
Indoor Positioning Systems”, IEEE 23rd International Symposium on Personal, Indoor and
Mobile Radio Communications – (PIMRC), 2012

[21] D. Pai, M. Malpani, I. Sasi, P. S. Mantripragada, N. Aggarwal, “Padati: A Robust Pedestrian
Dead Reckoning system on smartphones”, IEEE 11th International Conference on Trust,
Security and Privacy in Computing and Communication, 2012

[22] J. Bird, D. Arden, “Indoor Navigation with Foot-Mounted Strapdown Inertial Navigation
and Magnetic Sensors”, IEEE Wireless Communication, 2011

[23] A. R. Pratama, Widyawan, R. Hidayat, “Smartphone-based Pedestrian Dead Reckoning
as an Indoor Positioning System”, International Conference on System Engineering and
Technology, 2012

[24] M. Li, B. H. Kim, A .I. Mourikis, “Real-time Motion Tracking on a Cellphone using Inertial
Sensing and a Rolling-Shutter Camera”, ICRA, 2013

[25] J. Windau, L. Itti, “Situation awareness via sensor-equipped eyeglasses”, IROS, 2013

[26] L. Itti, “Models of Bottom-Up and Top-Down Visual Attention”, PhD Thesis, Caltech,
Pasadena, CA, USA, 2000, pp. 149
106

[27] J. Lee, H. Choi, D. Park, Y. Chung, H. Kim, S. Yoon, “Fault Detection and Diagnosis of
Railway Point Machines by Sound Analysis”, Sensors, April 2016

[28] S. Chattunyakit, Y. Kobayashi, T. Emaru, “Joint Fault Diagnosis of Legged Robot based on
Acoustic Processing”, IEEE/SICE International Symposium on System Integration (SII),
Meijo University, Nagoya, Japan, December 2015

[29] J Li, W. Dai, F. Metze, S. Qu, S. Das, “A comparison of Deep Learning methods for
environmental sound detection”, IEEE Int. Conference on Acoustics, Speech and Signal
Processing (ICASSO), 2017

[30] O. Esu, J. Flint, S. Watson, “Static Calibration of Microelectromechanical Systems (MEMS)
Accelerometers for In-Situ Wind Turbine Blade Condition Monitoring”, Proceedings of
33
rd
IMAC, A Conference and Exposition on Structural Dynamics, Special Topics in
Structural Dynamics, Volume 6, pp 91-98, 2015

[31] G. Vogl, M Donmez, A. Archenti, B. Weiss, “Inertial Measurement Unit for On-Machine
Diagnostics of Machine Tool Linear Axes”, Annual Conference of the Prognostics and
Health Management Society, 2016

[32] Z. Chen, C. Li, R. Sanchez: “Gearbox Fault Identification and Classification with
Convolutional Neural Networks”, Research Article, Shock and Vibration, Hindawi
Publishing Corporation, Volume 2015, Article ID 390134, 10 pages

[33] CCS Insight, Market Forecast Wearables Worldwide, 2018 – 2022,
https://www.ccsinsight.com/research-areas/wearables/market-forecast/

[34] H. Uematsu, et. al., “HALUX: Projection-based Interactive Skin for Digital Sports”, ACM
SIGGRAPH, Emerging Technologies, Article No. 10, 2016

[35] Y. Konishi, “Synesthesia suit: the full body immersive experience”, SIGGRAPH '16 ACM
SIGGRAPH, VR Village, Article No. 20, 2016

[36] Teslasuit, https://teslasuit.io/, Accessed Feb 28th, 2019

[37] H. Carson, et. al, “Intra-individual movement variability during skill transitions: A useful
marker?”, European Journal of Sport Science 14(4), 327-336, 2014

[38] Bureau of Labor Statistics of the U.S. Department of Labor, News Release USDL-16-2130,
2015

[39] AI Coaching Platform ARDA – www.arda.ai, Accessed Feb 28th, 2019

[40] D. Yang, et. al, “TennisMaster: an IMU-based online serve performance evaluation
system”, Proceedings of the 8th Augmented Human International Conference, Article
No. 17, 2017

107

[41] A. Morrison, et. al, “Analysis of the delivery plane in the golf swing using principal
components”, Proceedings of the Institution of Mechanical Engineers Part P Journal of
Sports Engineering and Technology, 2018

[42] P. Hume, et. al, “Handbook of Human Motion“, Chapter 97 “Movement Analysis of the
Golf Swing”, 2018

[43] J. Windau, L. Itti, “Inertial Machine Monitoring System for automated failure detection”,
IEEE International Conference on Robotics and Automation (ICRA), 2018

[44] S. Madgwick, et. al, “Estimation of IMU and MARG orientation using a gradient descent
algorithm”, IEEE International Conference on Rehabilitation Robotics, 2011

Abstract (if available)

Abstract Emerging wearable and cloud-connected sensor technologies offer new sensor placement options on the human body and machines. This opens new opportunities to explore cyber robotics algorithms (sensors and human motor plant) and smart manufacturing algorithms (sensors and manufacturing equipment). These algorithms process motion sensor data and provide situation awareness for a wide range of applications. Smart management and training systems assist humans in day-to-day living routines, healthcare and sports. Machines benefit from smart monitoring in manufacturing, retail machinery, transportation, and construction safety. During my PhD Research, I have developed several approaches for motion analysis and classification. (1) A situation awareness system (SAS) for head-mounted smartphones to respond to user activities (e.g., disable incoming phone calls in elevators, activate video recording while car driving), (2) a filter for head-mounted sensors (HOS) to allow full-body motion capturing by removing interfering head-motions, (3) an Inertial Machine Monitoring System (IMMS) to detect equipment failure or degraded states of a 3D-Printer, and (4) a “Smart Teaching System” (STS) for targeted motion feedback to refine physical tasks. To capture real-world sensor data, we designed hardware prototypes or used state-of-the-art wearable technology. We developed novel sensor fusion algorithms, implemented feature extraction methods based on gist, statistics, physics, frequency diagrams and validated classifiers: SAS achieved high accuracy (81.5%) when distinguishing between 20 real-world activities. HOS reduced the positional error of a traveled distance below 2.5% with head-mounted sensors for pedestrian dead reckoning applications. IMMS yielded 11-way classification accuracy over 99% when distinguishing between normal operation vs. 10 types of real-world abnormal equipment behavior. STS demonstrated that combining motion sensors and providing targeted feedback yield significantly improved golf swing training (3.7× increased performance score).

Linked assets

University of Southern California Dissertations and Theses

Conceptually similar

PDF

Magnetic induction-based wireless body area network and its application toward human motion tracking

PDF

Data scarcity in robotics: leveraging structural priors and representation learning

PDF

Learning to adapt to sensor changes and failures

PDF

Transfer learning for intelligent systems in the wild

PDF

Modeling dynamic behaviors in the wild

PDF

Pretraining transferable encoders for visual navigation using unlabeled datasets

PDF

Scalable machine learning algorithms for item recommendation

PDF

A framework for research in human-agent negotiation

PDF

Predicting and planning against real-world adversaries: an end-to-end pipeline to combat illegal wildlife poachers on a global scale

PDF

Toward situation awareness: activity and object recognition

PDF

Artificial intelligence for low resource communities: Influence maximization in an uncertain world

PDF

Neuroscience inspired algorithms for lifelong learning and machine vision

PDF

Algorithm and system co-optimization of graph and machine learning systems

PDF

No-regret learning and last-iterate convergence in games

PDF

Design of cost-efficient multi-sensor collaboration in wireless sensor networks

PDF

Effective graph representation and vertex classification with machine learning techniques

PDF

Latent space dynamics for interpretation, monitoring, and prediction in industrial systems

PDF

Interaction between Artificial Intelligence Systems and Primate Brains

PDF

Fair Machine Learning for Human Behavior Understanding

PDF

Building straggler-resilient and private machine learning systems in the cloud

Asset Metadata

Creator Windau, Jens Paul (author)

Core Title Smart monitoring and autonomous situation classification of humans and machines

School Viterbi School of Engineering

Degree Doctor of Philosophy

Degree Program Computer Science

Publication Date 06/28/2019

Defense Date 04/18/2019

Publisher University of Southern California (original), University of Southern California. Libraries (digital)

Tag anomaly detection,artificial intelligence,classification,Hardware,machine learning,OAI-PMH Harvest,sensors,software,wearable

Format application/pdf (imt)

Language English

Contributor Electronically uploaded by the author (provenance)

Advisor Itti, Laurent (committee chair)

Creator Email jenswindau@gmail.com,windau@usc.edu

Permanent Link (DOI) https://doi.org/10.25549/usctheses-c89-179145

Unique identifier UC11662646

Identifier etd-WindauJens-7524.pdf (filename),usctheses-c89-179145 (legacy record id)

Legacy Identifier etd-WindauJens-7524.pdf

Dmrecord 179145

Document Type Dissertation

Format application/pdf (imt)

Rights Windau, Jens Paul

Type texts

Source University of Southern California (contributing entity), University of Southern California Dissertations and Theses (collection)

Access Conditions The author retains rights to his/her dissertation, thesis or other graduate work according to U.S. copyright law. Electronic access is being provided by the USC Libraries in agreement with the a...

Repository Name University of Southern California Digital Library

Repository Location USC Digital Library, University of Southern California, University Park Campus MC 2810, 3434 South Grand Avenue, 2nd Floor, Los Angeles, California 90089-2810, USA