University of Southern California Dissertations and Theses

A non-invasive and intuitive command source for upper limb prostheses

(USC Thesis Other)

A non-invasive and intuitive command source for upper limb prostheses

PDF

Download a page range

Download transcript

Copy asset link

Request this asset

Transcript (if available)

Content

A NON-INVASIVE AND INTUITIVE COMMAND SOURCE
FOR UPPER LIMB PROSTHESES
by
Rahul Reddy Kaliki

A Dissertation Presented to the
FACULTY OF THE USC GRADUATE SCHOOL
UNIVERSITY OF SOUTHERN CALIFORNIA
In Partial Fulfillment of the
Requirements for the Degree
DOCTOR OF PHILOSOPHY
(BIOMEDICAL ENGINEERING)

December 2009

Copyright 2009 Rahul Reddy Kaliki
ii

Tabl e of Contents
List of Tables iv
List of Figures v
Abstract vi
Chapter 1: General Introduction 1
Available solutions for amputees 2
Next-generation prosthetic limbs and hand 5
Areas of active research in prosthesis control 6
Available solutions for quadriplegics 10
Active areas of research for spinal cord injured patients 13
Postural synergies: a unified approach for transhumeral amputees and C5/C6 quadriplegics
15
Evidence for postural synergies 16
Related previous work 20
Research hypotheses and assumptions 22
Chapter 2: Prediction of elbow trajectory from shoulder angles using neural networks 27
Introduction 28
Clustering of joint angle synergies 28
Methods 29
Two-dimensional point-to-point reaching experimental setup 29
Reaching and grasping vertically- and horizontally-oriented targets on a plane 31
Data acquisition and pre-processing 31
Neural network training 32
Evaluating neural networks in real-time tasks 34
Evaluating neural network models in virtual reality tasks 34
Results 36
Neural network off-line performance 36
Real-time deployment of trained neural network 38
Virtual reality reach and grasp task 39
Discussion 39
Chapter 3: Prediction of distal arm posture in 3-D space from shoulder movements for control
of upper limb prostheses 42
Introduction 43
Methods 45
Reaching and grasping task 45
Target locations 45
Data acquisition 48
Data preprocessing and partitioning 50
Neural network training 51
Results 54
Discussion 57
iii

Chapter 4: Evaluation of non-invasive command scheme for upper limb prostheses in a virtual
reality reach and grasp task 60
Inferential command scheme 64
Methods 68
Motion tracking procedure 68
Training data acquisition and processing 69
Artificial neural network training 70
Virtual reality testing 72
Results 76
Offline artificial neural network performance 77
Comparisons in trial mean times between groups 77
Analysis of spatial variability in performance 80
Identifying sources of differences in performance between groups 83
Discussion 84
Generalization between ANNs 84
Speed and accuracy 86
Potential improvements 89
Chapter 5: General Discussion 95
Implications for future studies 97
Implications for prosthetic control 98
Real world Implementation 105
Bibliography 109

iv

List of Tabl es
Table 1. Comparison of neural networks’ performances on the validation set. 37
Table 2. Summary data for ANNs trained with 3 shoulder angles predicting elbow angle. 55
Table 3. Summary data for ANNs trained with 5 shoulder angles predicting elbow angle 56
Table 4. Summary data for ANNs trained pedicting forearm angle and gripping pressure. 56
Table 5. Summary data for ANNs trained with 4 shoulder angles predicting elbow, forearm, and
shoulder protraction-retraction angles. 57
Table 6. Offline analysis of both intra- and inter-subject artificial neural networks. 77
v

List of Figures
Figure 1. Example shoulder/elbow synergies. 29
Figure 2. A schematic drawing of the experimental workspace. 30
Figure 3. A screen capture of our virtual reality environment. 35
Figure 4. Offline performance vs. number of training targets. 37
Figure 5. The performance of the neural network trained with the minimized set. 37
Figure 6. Plots of real-time neural network performance. 38
Figure 7. Plot of task completion time vs. number of successful task completions. 39
Figure 8. The robotic gantry that was used to present targets in 3D extrapersonal space. 46
Figure 9. Extrapersonal space definition. 47
Figure 10. A stem plot of all the locations of the targets in extrapersonal space. 48
Figure 11. A schematic of the subject during experimentation. 49
Figure 12. A typical example of joint angles recorded during a reach to a target. 55
Figure 13. Schematic of inferential command scheme (ICS). 67
Figure 14. The virtual reality reach and grasp task. 73
Figure 15. Scatter plot of Distance vs. Time for the second phase of VRE experiments in the tenth
session. 76
Figure 16. Plot of completion percentages of trials over sessions for Group A subjects. 79
Figure 17. Completed trial mean times over sessions. 80
Figure 18. Color-scaled plots of Subject 3’s mean times over blocks of the subject’s workspace
over ten sessions. 82
Figure 19. Breakdown in mean time to complete task by phase for subjects on the tenth trial. 83
Figure 20. The results of sensitivity analyses of the ICS. 93
Figure 20. Sketch of portable motion tracking system for a transhumeral amputee. 107
vi

Abstract
C5/C6 tetraplegic patients and transhumeral amputees may be able to use voluntary shoulder
motion as command signals for a functional electrical stimulation (FES) system or a
transhumeral prosthesis. Such prostheses require the control of endpoint position in three-
dimensions, hand orientation, and grasp. Stereotyped relationships, termed “postural
synergies,” exist between the shoulder, forearm, and wrist joints emerge during goal-oriented
reaching and transport movements as performed by able bodied subjects. Thus, the posture of
the shoulder can potentially be used to infer the posture of the elbow and forearm joints during
reaching and transport movements. To fit these synergies we utilized three-layer artificial
neural networks (ANNs). In contrast to previous work in this field, we initially trained ANNs with
three rotational angles at the shoulder to predict the elbow angle during reaches in a horizontal
plane. We found that the ANNs could predict elbow angle remarkably well across the entire
horizontal workspace during offline and online analysis. In the subsequent works, we extended
this paradigm to include shoulder translation movements in addition the shoulder rotational
angles to predict forearm angle and to control grasping in 3D extrapersonal space.
The complete inferential command system (ICS) was deployed for use in a virtual reality reach
and grasp task. In order to examine whether the ANNs generalized across subjects, we
alternated the use of ANNs trained on the subject’s own data and ANNs trained with a novel
subject’s data. Furthermore, we compared the performance of subjects using the ICS with
subjects operating the simulated prosthesis in virtual reality according to complete motion
tracking of their actual arm and hand movements. Subjects using the ICS were able to complete
the task at a high percentage and with low spatial variability across the workspace. Mean task
completion times of subjects using the ICS compared favorably to subjects using full motion
vii

capture regardless of which set of ANNs were used. Inferring the desired movement of distal
joints from voluntary shoulder movements appears to be a relatively simple, intuitive and non-
invasive approach to obtaining command signals for prostheses to restore distal arm and hand
function.
1

Chapter 1: General Introduction
From shaking a friend’s hand to catching a football to eating a sandwich, our upper limbs are
essential in our day-to-day affairs. We do these things without even giving a moment of
consideration as to how much force to apply so as to not crush our friend’s hand, or when to
reach outward to correspond with the arrival of a football, or which gripping pattern to apply so
that the sandwich does not fall from our grasp. Yet for each situation millions of neurons in the
brain and spinal cord must take into account the task goals, the current and future states of the
body, and current and future states of the environment to coordinate the activity of numerous
muscles actuating several joints across the whole body, all with minimal conscious effort.
Perhaps it is when these abilities are lost or not yet developed that the complexity of motor
control is appreciated.
Impairments of upper limb function can be caused by a variety of pathologies including
neuromuscular disorders, congenital defects, and acquired injuries. Amputees and
quadriplegics typically fall under the latter category. In the case of amputees, injuries are
sustained to a portion of the limb and irreparable segments are surgically removed leaving as
much of the nerves, musculature, and bone intact as possible. Typically, amputations are
unilateral (Dudkiewicz, et al. 2004). Conversely, quadriplegics retain intact limbs but lose the
ability to control all or parts of the limb bilaterally due to a lesion at the spinal cord. The
inability to perform day-to-day activities for both these patient classes depends on the level of
impairment.
It has been stated that the loss of hand functionality is a loss of 90% of the upper extremity
function and a 54% loss of overall body function (Blair, et al. 1987). While these figures appear
to be high considering the obvious differences in functionality between bilateral and unilateral
2

deficits, it is nevertheless clear that the hand is important in day-to-day use. A loss of the hand
results in the inability to grasp and manipulate objects. Also lost is feedback from sensory
receptors on the fingers and palm, which help identify objects, detect pain, and sense
temperature. In addition to the loss of hand function, a loss of function at the forearm or wrist
results in the inability to orient the hand. Moving proximally up the arm, the next major loss of
functionality occurs at the elbow. The elbow, coupled with shoulder, position the hand in three
dimensional space. Lastly, impairment of the shoulder joint leads to complete loss of function
of the limb.
Available solutions for amputees
Amputees can be fitted with either cosmetic prostheses or functional prostheses to restore
some use of the upper limb. Unfortunately, the available functional prostheses are very limited
in both their control and function. In fact, the most commonly used prosthetic limbs today still
rely on a mechanical control paradigm invented nearly 150 years ago. Amputees currently only
have two options for control of their prostheses: body power and myoelectric control.
The first body-powered prosthesis was invented by a German dentist named Peter Baliff in 1818
following the Napoleonic wars (Meier 2004). Baliff’s prosthesis required transradial amputees
to move their trunk and shoulder girdle to flex and extended prosthetic fingers through a
mechanical linkage. This principle was later extended for transhumeral amputees and in 1860 a
shoulder harness and split hook terminal device (TD) was introduced. This design is
fundamentally the same as the popular cable-powered split-hook prosthesis available today.
The current design includes a harness worn around the upper torso with a cable attached
between the harness and a spring-loaded split hook or hand. To open the terminal device, the
amputee must increase tension in the connected cable by moving either the torso, shrugging
3

the shoulder, or flexing the shoulder, depending on the amputation level and harness design.
Tension in the cable also provides the wearer with limited but useful sensory feedback of
gripping force. High level amputees (transhumeral and shoulder disarticulation amputees) can
be fitted with a body-powered prosthesis with additional degrees-of-freedom. The additional
joints can either be passive friction-controlled, mechanically locked, or body-powered (Stark and
LeBlanc 2004). In unilateral amputees both passive and lockable joints have to be moved to the
desired position by moving them with the able hand. Obviously, this is not an option for
bilateral amputees. Body-powered joints are flexed with similar systems to that used for the
split-hook. To move the body-powered elbow and shoulder joints, amputees can tension the
cable attached to joint appropriate for the movement. Simultaneous movement of multiple
joints cannot be done for all parts of the workspace and in fact, most high-level amputees opt
for mechanically lockable elbow and/or shoulder joints (Stark and LeBlanc 2004). Aside from
their practical use and functionality, the body-powered prostheses are not visually appealing,
especially when using the split-hook TD. Despite these limitations, body-powered prostheses
are the most commonly used prostheses worldwide due to their relative low cost, reliability,
high durability, and functionality (Kruger and Fishman 1993).
Following World War II, the United States established the National Research Council which later
became the Prosthetic Research Board, with the mission of funding projects to create better
prostheses. Supported by these government grants, Samuel Alderson at IBM created the first
externally powered prostheses complete with motorized joints in 1949 (Wilson 1992).
Unfortunately, the technology was ahead of its time and the electronics could not be
miniaturized to meet the project’s requirements and commercial development. In 1958, a
Russian team of engineers created a similar electronic hand but was controlled by electrical
4

activity generated by intact muscles. Otto Bock commercialized this design; heralded as the
world’s first commercially available myoelectric hand (Meier 2004).
Current myoelectric prostheses are lighter weight, made with more durable materials, and
operate faster than the model first sold over 50 years ago, but the basic mode of control and
functionality has remained the same. Electromyography (EMG) sensors are placed on the belly
of functional muscle of the stump. Typically, one sensor is placed above flexor muscles and
another is placed over extensors. The EMG signals are amplified, digitized and passed to a
microcontroller. The direction and speed or force of the motor is proportional to the relative
amplitude of the EMG signals, subject to special states defined by thresholds and coactivation
criteria. Amputees with higher level amputations must co-contract their flexor and extensor
muscles simultaneously to switch between the additional degrees-of-freedom, making control
of multiple degrees-of-freedom cumbersome during daily use. An even bigger problem for
myoelectric control as a whole is the very low amplitude resolution of EMG, which is an
inherently noisy signal arising from the random combination of overlapping action potentials in
many motor units. Because the amplitude of the EMG signal cannot be used reliably to set a
desired position, it is used as a velocity command.
With these technical issues it is no wonder that many myoelectric users have abandoned these
prostheses altogether (Biddiss and Chau 2007). The most common complaint has been a lack of
functionality and the difficulty of use. Amputees have also complained of the poor durability,
reliability, and power of myoelectric prostheses (Biddiss and Chau 2007). These concerns are
limitations of myoelectric prosthetic devices in general and are not directly related to the
control paradigm. On the other hand, myoelectric prostheses do appear more life-like and do
5

not require large movements of the trunk or shoulder to move the limbs. Thus, these devices
are worn more often in public (van Lunteren, et al. 1983).
As a compromise, some amputees opt for a combination of body-powered and myoelectric
prostheses termed “hybrid prostheses,” but these prostheses are subject to the same
limitations as previously discussed. Surgical procedures can also restore some functionality to
patients but are far more common in developing countries where prostheses are unavailable or
too expensive (for a review see Wilson 1992).
As mentioned earlier, the poor functionality and comfort of currently available prosthetic
solutions has led many amputees to abandon their prostheses. A recent study has shown that
anywhere from 23-47% of all body-powered and myoelectric prostheses users eventually
abandoned prostheses use altogether, citing “problems involving comfort, lack of function, and
durability (Biddiss and Chau 2007).” Another study surveying amputees who did not wear any
prostheses found that 89% of the surveyed population felt they were more functional without
them (Melendez and LeBlanc 1988). It is obvious that considerable improvements need to be
made to prostheses. Specifically, amputees desire a more intuitive command mechanism to
control a durable and lifelike prosthetic limb.
Next-generation prosthetic limbs and hand
Recently, there have been significant advances in mechatronic prosthetic hands and limbs. The
first of the next-generation prosthetic hands is the i-LIMB (Touch Bionics, Inc.) introduced in
2007. This hand features an anthropomorphic design with four motorized fingers and a passive
thumb covered with a realistic skin-like cosmesis. The i-LIMB also features a compliant grasp
which takes the shape of the object being held. Despite these added features, the i-LIMB is
slower and less durable than other myoelectric hands. Otto Bock has also developed a
6

motorized, multi-articulated hand called the Michaelangelo hand set to release in 2010. This
hand features two motorized fingers and a motorized thumb. Lastly, a group in Italy has also
developed a multi-articulated hand called the CyberHand, which includes many of the same
features as the Michaelangelo hand and i-LIMB but also plans to introduce sensory feedback
into the peripheral nervous system (Carrozza, et al. 2006). While all of these hands are slightly
different, each hand will be able to generate an unprecedented number of grip patterns and
individual finger movements.
Two groups are also looking beyond building next-generation hands and are working on entire
mechatronic arms. The DEKA research group founded by Dean Kamen is developing an
anthropomorphic prosthetic limb with 18 degrees of freedom (10 powered) dubbed the “Luke
Arm.” The arm only weighs around 9lbs but can lift up to 20lbs. Another group at the Applied
Physics Laboratory at Johns Hopkins University is seeking to commercialize an arm that
replicates the entire 22 degrees-of-freedom present in the human limb. Clearly, realistic,
anthropomorphic prosthetic hands and limbs are on the horizon and will soon be commercially
available. Unfortunately, control for these prostheses is limited to body-powered and
myoelectric control. In fact, the state-of-the-art i-LIMB, despite being capable of six different
grip patterns and individual finger movement, still uses a pair of EMG electrodes to open and
close the hand. It is obvious advances need to be made in the control of prostheses as well.
Areas of active research in prosthesis control
Currently, there are several areas of prosthetic control that are actively being researched.
Several groups are developing advanced signal processing algorithms to extract more
information from EMG signals (Hudgins, Philip and Scott 1993, Huang, et al. 2005, Chu, Inhyuk
and Mu-Seong 2006). These algorithms use machine learning techniques to classify patterns of
7

EMG activity as grip postures, individual finger movements, or wrist movements. Yet because
these algorithms are EMG-based they have the same limitations as other EMG-based devices, as
previously discussed. Furthermore because these algorithms require arrays of surface
electrodes, they could be sensitive to movement artifacts, changes in skin impedance during
daily use, and crosstalk from muscles.
One promising technique for prosthetic control is targeted muscle reinnervation (TMR) (Kuiken,
Dumanian, et al. 2004). TMR is a surgical solution where residual nerves from the amputated
limb are relocated to innervate muscles rendered non-functional by the amputation. Targeted
muscles are denervated and separated into sections to create electrically isolated substrates for
the residual nerves. Nerves that originally innervated the lost muscles are attached to the
isolated sections of the targeted muscle. The muscle fibers serve as “biological amplifiers” of
the signals from the nerve, which can be picked up by conventional EMG sensors. After several
months of healing and rehabilitation, patients need only to think about moving the lost degrees-
of-freedom and the sensors should be able to detect the desired movement intention. Thus,
TMR allows amputees to intuitively move multiple joints simultaneously. Furthermore, residual
sensory nerves also reinnervate the targeted muscle and overlying skin and can be used as a
means of providing sensory feedback to the patient.
TMR is a significant leap in the control of prostheses but is still not widely deployed. The surgery
is quite extensive and is not ideal for older amputees. Risks of the surgery include permanent
paralysis of the targeted muscle, recurrence of phantom limb pain, and the development of
painful neuromas (Kuiken, Miller, et al. 2007). Some amputees have pre-existing damage to the
residual nerves, often due to complications of the original amputation surgery, and are unable
to benefit from TMR. Additionally, the healing and rehabilitation process typically takes a year
8

before patients can regain the use of a prosthetic limb. The entire surgery and rehabilitation is
costly, both monetarily and in terms of stress on the amputee, and may not be valuable for
unilateral, lower-level amputees.
A group at the University of Utah has developed a similar technique for intuitive prosthesis
control using electrodes recording directly from the residual nerve endings (Dhillon, Lawrence
and Horch 2004, Dhillon and Horch 2005). The group has implanted longitudinal intrafascicular
electrodes (LIFEs) to directly interface with the severed fascicles of the residual peripheral
nerves. After two days following implantation, subjects were able to control specific peripheral
nerves associated with missing limb degrees-of-freedom in a computer-based task environment.
Furthermore, the group has been able to selectively stimulate afferent sensory fibers with in
fascicles of peripheral nerves (Dhillon and Horch 2005). In this study, stimulation of specific
nerves elicited sensations of touch and movement of a patient’s phantom hand. While these
results are promising, there are few technological hurdles to overcome. First, implanting wired
electrodes through the skin might not be robust to movement and general daily use leading to
problems in reliability of the control paradigm. Furthermore, long term studies have not been
conducted to determine longitudinal signal efficacy. Additional studies are needed to determine
the effects of activity of stump muscles and other functioning muscles on the signal measured
by the LIFEs. Also, the success of this system, like TMR, depends on the ability of the amputee
to generate discrete commands through the nerves and the ability of the residual nerve stumps
to conduct action potentials. If either component is lacking this technique is not an option.
Perhaps the most exciting of the recent advances in neuroprosthetics has been the promise of
direct cortical control through brain-machine interfaces (BMIs). The earliest work in this field
was done by Evarts in 1971, when he found that certain cells in motor cortex showed bursts of
9

activity correlated with and preceding movement of the arm in primates (Evarts and Thatch
1971). In 1982 Georgopolous et al. found that some neurons in the motor cortex have
“preferred directions” and fire with the highest probability when the hand moved in the same
direction as the preferred direction (Georgopoulos, et al. 1982). Other studies have shown that
instantaneous activity in the motor cortex can be correlated with hand speed (Fu 1995) or
planned movement distance (Schwartz 1993). Furthermore, these correlations were not limited
to the motor cortex and could be found in other brain areas (Kalaska, Caminiti and
Georgopoulos 1983, Caminiti, et al. 1991, Fortier, Smith and Kalaska 1993, Cohen and
Prud'homme 1994). Thus, these works opened up the possibility of placing arrays of electrodes
directly into the brain to obtain motor commands for prosthetic limbs. Several groups have
shown that nonhuman primates could use cortical activity to control BMIs for cursors on a
screen (Musallam, et al. 2004) and robotic arms in real-time (Chapin, et al. 1999).
Despite these advances, there are still some major technical challenges to overcome. Advances
in BMIs have been driven by the fundamental question in motor control: how does the brain
plan and control movement? Several of the aforementioned studies relate activity in the brain
to the hand trajectory in extrinsic coordinates, under the assumption that the brain or the spinal
cord can solve the fairly complex inverse kinematics problem. Some suggest that the brain plans
and executes motion in intrinsic coordinates (Soechting and Flanders, 1992), while others argue
that cortical activity is directly related muscle activity (Scott, 2003), yet no one theory has been
able to explain brain behavior for every situation (for a complete discussion on this topic see
reviews by Scott, 2004 and Shadmehr and Krakeur, 2008). In order for a BMI to be robust for
daily use, it is likely that a correct interpretation of brain activity is required. Another problem
related to BMIs is chronic implantation of electrode arrays in the brain. Studies in nonhuman
primates have shown that useful recordings from cortical arrays typically last no longer than a
10

year (Kipke, et al. 2008). This is primarily due to the immune system’s response to the foreign
electrode array which results in capsulation and a degradation of the recorded signals. Chronic
implantation in one human has lasted over three years (Donoghue 2008) but results can vary
widely and may degrade more rapidly in subjects that are more physically active. Clearly, BMIs
will not be practical if patients will need to have brain surgery regularly to implant new arrays,
but this is an active area of research with promising results (see review by Kipke et al., 2008).
Available solutions for quadriplegics
Quadriplegics typically do not enjoy the same quality of life as amputees. All four limbs of
quadriplegics are completely or partially disabled. In addition to problems with mobility, these
patients also have difficulties with bowel, bladder, and sexual function. When asked about the
greatest needs for quadriplegic paints a recent study found that arm and hand manipulative task
took priority over all other functions (Kilgore, Scherer, et al. 2001, Snoek, et al. 2004). Current
solutions for high-level (C5-C6) quadriplegics are surgical interventions or neuroprostheses.
Surgical interventions typically involve a technique called tendon transfer. This technique calls
for a functional muscle’s tendon to be relocated to a new attachment point so as to restore
movement to a paralyzed joint. Tendon transfers were first introduced by Erik Moberg in 1975
to restore elbow extension and hand grasping functions to quadriplegics (Moberg 1975). To this
day many surgeons replicate Moberg’s method of using the posterior and all or parts of the
middle deltoid to replace the triceps to extend the elbow. Over the last 30 years this method
has been applied to other types of movement and has been successful in restoring wrist
extension, forearm rotation and hand grasp and release (Mulcahey 2008). While this surgery
has become common, it is still an invasive procedure and patients face risks typically associated
with elective surgery. Additionally, some patients can rupture the tendon or permanently
11

stiffen the tendon if the transferred muscle is moved too early or too late following the surgery,
respectively. Not all C5-C6 patients opt for tendon transfer surgeries because these patients
typically only have voluntary control of the shoulder and elbow flexion allowing positioning of
their hand in space but leaving them without any grasping abilities. Functionally, patients with
elbow extension tendon transfers have been shown to be generally uncoordinated and slow
during reaching and pointing (Wierzbicka and Wiegner,1992; Wierzbicka and Wiegner,1996;
Laffont, et al. 2000). It should also be noted that rehabilitation and training is required for
patients to learn to use a previously unrelated muscle to move the formerly paralyzed joint,
which results in unintuitive control.
Neuroprostheses are a class of devices that can restore motor, sensory, or cognitive functions
damaged by injury or disease. In the 1960s, the first upper extremity neuroprostheses were
developed using surface electrodes to open and close the hand (Vodovnik, et al. 1965). Soon
after, functional electrical stimulation (FES) was used in a variety of applications including
bladder control (Brindley, Polkey and Rushton 1982), respiration (Glenn, Gee and Schachter
1978), coughing (Linder 1993), standing (Triolo, et al. 1996) and walking (Sharma, et al. 1998).
Hunter Peckham and colleagues pioneered the use of FES in C5-C6 quadriplegics with their
externally powered FES system (Smith, et al. 1987). Their design was the foundation of the
FreeHand
TM
system (NeuroControl Corp.) used today to restore grasping function to C5-C6
quadriplegics. The first generation Freehand system maps the residual voluntary
protraction/retraction of the contralateral shoulder to stimulate implanted electrodes in the
hand for opening and closing (Kilgore, et al. 1997). The dependence on contralateral shoulder
movements makes it unsuitable for extension to bimanual tasks and deprives users of the
proprioceptive feedback that might be derived from the interaction of the command signals
with the actual arm being moved. This system, while a little awkward in its method of control, is
12

able to restore grasping function, but does not lend itself to control of the equally important
functionality of reaching.
The second generation Freehand system extended the control to restore reaching and forearm
pronation function in addition to grasping ( Kilgore, et al. 2005). Patients were implanted with
at total of 12 stimulating electrodes in the triceps, muscles of the forearm, and muscles in the
palm. A stimulator-telemeter was also implanted which communicated wirelessly with an
external control unit (ECU). Control of the FES was provided by a pair of implanted EMG
electrodes placed on a pair of agonist-antagonist muscles. To control a specific degree-of-
freedom, patients needed to either voluntarily contract the agonist or antagonist muscle
depending on the required function. To cycle between degrees-of-freedom, patients needed to
co-contract the agonist and antagonist muscles until the desired degree-of-freedom was
selected. Implanted patients regained the ability to control various degrees-of-freedom and
were able to accomplish some activities of daily living, such as eating and drinking. This method
is a great improvement in terms of functionality for paraplegic patients, but the number of
controlled outputs is limited by the number of viable voluntary muscles available. The limited
number of inputs allowed users to move only one DOF at a time. Furthermore, if some of the
controller muscle groups were involved in other functions required by the patient then the
muscle group could not be used as a reliable command source. A prosthesis that relies on
command sources unrelated to natural reaching movements forces the user to learn unnatural
reaching control strategies. Such a system could require ungainly and cumbersome movements
to restore functionality.
An alternative control strategy is also being tested for the second generation Freehand system.
It is comprised of a magnet and an array of Hall effect sensors implanted in the wrist to detect
13

motion (Johnson, et al. 1999). Stimulation is proportionally controlled by the position of the
wrist. Unfortunately, due to the limited number of control inputs, this strategy is prone to the
same limitations as the EMG-based strategy.
Active areas of research for spinal cord injured patients
Researchers have also begun working on BMIs for quadriplegics. John Donoghue and colleagues
took advantage of the previously discussed correlation between gross cortical activity and hand
position to create the BrainGate
TM
neuroprostheses (Cyberkinetics Neurotechnology Systems,
Inc.) (Hochberg, et al. 2006). The BrainGate is a revolutionary system which allows previously
completely paralyzed (“locked-in”) patients to communicate with the outside world through the
use of a PC. The system consists of an implanted array in the primary motor cortex (M1)
attached to a personal computer. The activity from M1 is processed and interpreted as the two-
dimensional position of a cursor on a computer screen visible to the patient. While this
technology is incredibly valuable for locked in patients it is not as useful for incomplete
quadriplegics (C5 and below) who desire the ability to reach and grasp.
A recent study by Moritz et al. has shown promise for C5-C6 quadriplegics by bridging the gap
between FES and BMIs (Moritz, Perlmutte and Fetz 2008). In the study, nonhuman primates
were first trained on modulating the discharge rates of recorded cells in M1. Next, peripheral
nerves controlling the wrist were blocked with a local anesthetic and stimulating electrodes
were implanted. Discharge rates were then scaled to stimuli delivered to the paralyzed muscles.
The monkeys learned to control their wrists almost immediately. Clearly, these results provide
evidence that BMIs for the restoration of upper limb function in quadriplegics is possible. In
fact, these results have led some experts in the field to state that “the technology has reached
sufficient maturation that the proof-of-concept clinical demonstration could be accomplished
14

within the next 5 years (Pancrazio and Peckham 2009).” While, these results are indicative of
progress, it must be noted that BMIs still have many technological hurdles to overcome before
they are ready for clinical use as previously discussed. Additionally, it is still unclear how
intuitive this control scheme is. In this case, the monkeys were trained to consciously modulate
brain activity unrelated to the volitional movement of the wrist. It remains to be seen whether
this control scheme can be used to generate coordinated movement of multiple degrees-of-
freedom and if it will be intuitive for daily use.
The ultimate goal for any spinal cord injured patient is regeneration or replacement of damaged
nerve tissue. In spinal cord injuries, severe trauma to the spinal cord can cause the death of
many different cell types primarily in the gray matter leading to cavities and cysts, which in turn
disrupt the descending and ascending axonal tracts. Following the injury, secondary processes
such as apoptosis (Crowe and Beattie 1997), demyelination (Totoiu and Keirstead 2005), glial
scarring (Silver and Miller 2004), and inflammatory cell responses (Jones, McDaniel and
Popovich 2005) cause additional permanent damage to the structure and function of the spinal
cord. Many cellular and molecular therapies which seek to limit components of primary injuries
and secondary responses are actively being researched by many investigators and some studies
have reached or are approaching clinical trials (see reviews by Thuret et al., 2006; Lim and Tow,
2008). Some of these therapies have had success in restoring function in non-primate animal
models (Cheng, Cao and Olson 1996, Koda, et al. 2005) but, have failed to achieve similar results
in humans (Amador and Guest 2005, Reier 2004). Furthermore, many regenerative therapies
have been shown to succeed only when the therapy is applied immediately following the injury
(Karimi-Abdolrezaee, et al. 2006, Hall and Springer 2004, Faulkner and Keirstead 2005). Patients
with pre-existing spinal cord injuries may not be suitable candidates for these therapies.
Perhaps the biggest issue is that while some therapies promote axonal growth, all of these
15

approaches face difficulties of properly restoring functional circuitry in the spinal cord (Campos,
R.T. and Martin 2004). Thus, while the promise of nerve regeneration is attractive, effective
solutions for quadriplegics remain elusive.
Postural synergies: a unified approach for transhumeral amputees and C5/C6
quadriplegics
At first glance, upper limb amputees and quadriplegics do not appear to share many similarities.
The morphology of injury, quality of life, and overall functional abilities are not similar across the
two patient classes. Yet, upon closer inspection there are several distinct parallels between the
two. As described above, there is a dearth of solutions available to restore limb functionality to
these patients. The current solutions are also limited in control and functionality. Furthermore,
current research and potential future solutions have significant biological and technological
hurdles to overcome before they are made available.
In terms of isolated upper limb function there clear parallels based on the level of injury.
Considering this, our solution focuses on a subset of these two patient classes: C5/C6
quadriplegics and transhumeral amputees. Patients in both these classes have deficiencies in
function at and below the elbow, yet retain partial to full control of their shoulders. Under the
assumption that patients are able to fully articulate their shoulders, we have developed a
solution that uses the residual shoulder motion to recover control of some of the lost degrees of
freedom. Specifically, by identifying spatiotemporal kinematic relationships (which we term
postural synergies) between the proximal and distal joints of the upper limb, motion of the
shoulder can give rise to an intuitive and non-invasive solution for restoring some of the lost
degrees freedom for reaching and grasping. To understand how such a solution can provide
intuitive control for these patients, one must first understand the characteristics of motor
16

control that give rise to these postural synergies between the proximal and distal joints of the
limb.
Evidence for postural synergies
The coordination of the multiple degrees-of-freedom present in the upper limb to reach and
grasp is not trivial. The human arm has more degrees-of-freedom than required to specify end-
point position and orientation. The additional degrees-of-freedom provide the brain with
flexibility is selecting reaching strategies but make it difficult to understand how the brain is
solving the equations of motion governing the planning of movement. Consider the case of
pointing in three-dimensions: in theory, due to the redundancy problem, a point in space can be
reached with nearly an infinite combination of joint postures. This issue was first addressed by
Bernstein in 1967 and termed the “redundancy problem (Bernstein 1976).” So how does the
brain solve the redundancy problem? While theories of motor control are not the topics of
discussion in this thesis, it is clear that the brain is able to solve the redundancy problem and as
a result some interesting stereotypical behavior emerges.
A study by Fitts in 1954 was the first published work to show evidence of recurring patterns of
movement (Fitts 1954). In this task subjects were asked to move a pen from a starting position
to a “goal region” as quickly and accurately as possible. He found that subjects produce
movements whose duration varied logarithmically with distance. Furthermore, the relationship
between distance and time was modulated by constraints on accuracy and by variations in the
weight of the pen. This study provided evidence that regularities in motor planning were the
result of strategic planning of movements in the brain which included considerations of
constraints of accuracy, speed, and load.
17

In 1981, Morasso found further invariances in motor planning in simple point-to-point reaching
in a two-dimensional, horizontal plane (Morasso, 1981). Specifically, he found that subjects
moved their hands in roughly straight lines between the starting and ending positions and with
bell-shaped velocity profiles invariant to initial and target positions. These features were
consistent within and across subjects. The straight hand paths and normalized bell-shaped
velocity curves were preserved when the speed of the reaches were varied (Flash and Hogan
1985), for reaches in three dimensions (Nishikawa, Murray and Flanders 1999; Morasso, 1983)
and under various loads (Hong, Corcos and Gottlieb 1994).
Soechting and Lacquaniti found similar invariances in joint angle and joint velocity space
(Soechting and Lacquaniti, 1981). In this case, they examined the shoulder and elbow flexion-
extension as subjects moved in a two-dimensional sagittal plane and pointed to various targets
at different heights. They found that the phase angle (angle-angle) relationships were
consistent across trials for a given target in space and invariant to the movement speed. Unlike
straight hand paths, postural synergies varied across the workspace. Additionally, they reported
a linear relationship between shoulder and elbow angular velocities during the deceleration
phase of the reaching movement invariant to target position in space and speed of reach.
Additional studies supported the existence of these relationships in reaches under varying loads
and in reaches with artificially extended limbs (Lacquaniti, Soechting and Terzuolo, 1982).
In 1982, Lacquaniti and Soechting decided to follow up their previous work with postural
synergies for reaching by examining kinematic relationships between the shoulder, elbow, and
wrist joints during a reaching and prehension task (Lacquaniti and Soechting, 1982). The task
called for subjects to make pointing movements with a specified rotation component while
keeping the arm in the sagittal plane. They found no consistent relationship between the
18

proximal (elbow and shoulder) joints and the wrist joint, leading the authors to conclude that
planning of prehension was computed separately from the planning of reaching. Yet because
the task confined motion to a 2D sagittal plane and inherently required a decoupling of the
prehension and reaching movements the results were considered inconclusive (Desmurget, et
al. 1995).
Soechting and Flanders revisited this problem in 1993 and found that the sensorimotor
transformation between the extrinsic representation of object orientation to the intrinsic
representation of hand orientation was not independent of the transformation for object
location (Soechting and Flanders, Parallel, 1993). In other words, planning of hand orientation
was not independent of the planning of reaching. These results were supported by Desmurget
et al. who found a stereotyped relationship between elbow and wrist trajectories (Desmurget,
et al. 1995). In a follow up study Desmurget et al. found consistent relationships between
object orientation and arm trajectory during reaching and prehension movements (Desmurget,
et al. 1996). Furthermore, that final posture of the entire arm was consistently dependent on
the orientation of the target. The existence of relationships between transport movements and
grasp were also examined (Jeannerod 1984, Gentilucci, et al. 1991), but no such relationships
were found.
The above studies show that stereotyped relationships clearly exist between the kinematics of
shoulder and elbow joints during pointing and reaching movements resulting in straight hand
paths and bell-shaped hand velocity curves. In addition, the planning of movement is also
dependent on the target’s orientation therefore requiring a simultaneous coordination of
shoulder, wrist, and elbow joints. Grasping movements appear to be controlled by a separate
mechanism.
19

The existence of stereotyped relationships is the result of functional coupling among the joints
of the upper limb. It is likely that this coupling emerged from the internal constraints of
coordinating volitional limb movement. One such constraint is inertial coupling between
segments of the arm. For example, the inertia of the forearm is coupled to the inertia of the
upper arm, meaning torque applied at one joint produces acceleration at the other via Coriolis
forces (Hollerbach and Flash 1982). Functionally, this could reduce the complexity of the control
problem because joint torque calculations become the function of a single variable. Another
constraint could be the previously described redundancy problem. The number of independent
variables could be reduced if the coordination of multiple joints is optimized based on the task
constraints (Todorov and Jordan 2002).
The anatomy of the arm also provides support for the control of coordinated movements of the
limb. The upper limb contains several biarticular muscles which simultaneously act on two
joints (Basmajian 1978). In the forearm, for example, most of the muscles involved in pronation
or supination also act as flexors and extensors of the elbow. In fact, it has been shown that the
activities of these muscles during simultaneous movement of the forearm and elbow were less
than when subjects moved one joint individually; indicating that simultaneous movement of two
joints is possibly more energetically efficient (Sergio and Ostry 1994). Thus, the coordination of
limb movements is aided by biarticular muscles.
Further evidence for the anatomical basis of synergies can be seen in the connection between
the brain and muscles. Descending commands from the brain are transmitted to the muscles via
two types of pathways: direct and indirect pathways. Direct pathways are generally believed to
be relatively new adaptations, in terms of evolution, in primates (both human and non-) to
provide for dexterous control of the hand (Isa, et al. 2007). Indeed, direct cortico-
20

motorneuronal (CM) pathways account for the majority of the descending connections to the
muscles of the primate hand (Rathelot and Strick 2006)ck, 2006) and are not present in non-
primates (Illert, Lundberg and Tanaka 1976, Kuypers 1982, Gugino, Rowinski and Stoney 1990,
Yang 2003, Alstermark, Ogawa and Isa.T., 2004, Alstermark and Ogawa, 2004). These
connections can account for the independent control of grasping from reaching. Indirect
corticospinal (CS) pathways, on the other hand, send information from the brain to the motor
neurons via subcortical or spinal interneuronal systems. The interneurons can synapse onto
multiple synergistic agonists and/or antagonists (Alstermark and Lundberg, 1992, Porter and
Lemon 1993, Maier, et al. 1998) (Porter and Lemon 1993). While relatively little is known about
the function of these pathways in humans, cat studies have shown that these indirect pathways
mediate commands for forelimb “target reaching (Alstermark and Lundberg, 1992).” Thus, it is
possible that reaching commands are communicated via indirect pathways through the spinal
cord, which recruit groups of synergist muscles and synchronize the movement of multiple joints
during reaching. It should be noted, though, that both indirect and direct pathways exist for
both proximal and distal muscles of the upper limb, so it is likely that pathways are recruited
based on a given task’s requirements.
Related previous work
Clearly, spatiotemporal synergies exist between proximal and distal joints of the arm during
reaching movements. Thus, it may be possible to take advantage of these synergies to restore
function to transhumeral and/or C5-C6 quadriplegics. Popovic and colleagues have developed
various synergy-based upper limb neuroprosthetic controllers that predicted elbow
flexion/extension from shoulder flexion/extension based on simple scaling rules (Popovic and
Popovic 1998), through inductive learning (Popovic and Popovic 2001), and from training a radial
basis function network (Iftime, Egsgaard and Popovic 2005). In the latter experiment, the
21

investigators employed radial basis function artificial neural networks (RBF ANN) to extract the
joint acceleration synergies between proximal and distal upper limb joints. Specifically, the
predictive abilities of five different synergies were examined:
• shoulder flexion/extension vs. elbow flexion/extension angular accelerations (Sy1)
• shoulder abduction/adduction vs. elbow flexion/extension angular accelerations (Sy2)
• elbow flexion/extension vs. forearm pronation/supination angular accelerations (Sy3)
• forearm pronation/supination vs. thumb flexion/extension angular accelerations (Sy4)
• thumb flexion/extension vs. index finger flexion/extension angular accelerations (Sy5)

Subjects were to complete a task separated into four sequences: reaching and grasping a
“finger food” target on a horizontal workspace (Sequence 1), bringing the food to their mouths
(Sequence 2), returning the food to its original position and releasing it (Sequence 3), and
returning to the initial position (Sequence 4). The RBF ANNs were trained on a set of sequences
for each target location. Then, each RBF ANN was evaluated for three different scenarios: the
standard scenario, in which the RBF ANN’s ability to generalize inter-subject reaches to the
same target positions was examined, the distance scenario, in which the networks were
evaluated on their ability to predict the sets of sequences for targets located distally to those
with which the network was trained, and the laterality scenario, in which the networks were
evaluated on their ability to predict sets of sequences for targets located lateral to the target
used to train the network.
For the standard scenario, the RBF ANNs trained on Sy1, Sy2, and Sy 3 were able to predict
Sequences 1 and 4 accurately. In the distance scenario, RBF ANNs trained on Sy1 and Sy3 had
high correlation coefficients for all four sequences. Finally, for the laterality scenario, no RBF
ANN was able to accurately predict any sequence. This indicated that the synergy rules changed
across the two-dimensional workspace. Therefore, the user would need to manually select
among several synergy rules based on where he or she wanted to reach. While some reaching
22

motion could be restored to these subjects, having to switch manually between synergy rules
would become cumbersome during everyday activities, especially if the user wanted to make
movements across the boundaries of these regions.
It should also be noted that the task presented in this study was highly stereotyped. Reaching
to and grasping a finger food target only represents a very small subset of possible reaching and
grasping tasks. Intuitively, reaching to and grasping objects positioned at various orientations
could yield quite different relationships between elbow flexion/extension and forearm
pronation/supination, thereby, possibly reducing the predictability of forearm
pronation/supination from the elbow angle.
In contrast to Popovic et al., we hypothesize that all five degrees of freedom at the shoulder
might contribute useful information that would generalize better over the whole workspace,
thus, eliminating the need for manual switching. Furthermore, our initial studies suggested that
the spatiotemporal relationships between joint angles are relatively stable and therefore more
likely to result in a user-controllable system as compared to joint velocity and joint acceleration
relationships. Finally, as opposed to training synergy models with data from only a single target
location, we wanted to use a set of reaches to a wide range of target locations in order to
account for variations in the reaching strategy across the workspace. Data from the entire
reaching trajectory were used, rather than just the end-points.
Research hypotheses and assumptions
Available control schemes for prosthetic and FES limbs are underactuated. The number of
degrees-of-freedom available for control is far less than the number of degrees-of-freedom
required to move the arm smoothly. In addition, current control paradigms are not intuitive and
require a great amount of conscious efforts to reach, grasp and manipulate objects. Any
23

successful control paradigm must allow amputees to reach smoothly and intuitively.
Restoration of basic functionality to an upper limb of a transhumeral amputee or a C5-C6
quadriplegic requires at least five degrees-of-freedom: the specification of endpoint position in
three-dimensions, the orientation of the hand determined by the forearm pronation/supination
angle, and the control of the opening and closing of the hand. We propose mapping five
degrees-of-freedom at the shoulder to the required five degrees-of-freedom outputs. Because
the outputs of the proposed command scheme are predicted from stereotyped relationships
between the proximal and distal joints of the upper limb, we hypothesize that able-bodied
subjects can learn to use the command scheme to reach and grasp objects.
The work presented here operates under one basic assumption: spatiotemporal postural
synergies exist between the shoulder and distal joints for reaches in three-dimensions. As
previously discussed, Lacquaniti and Soechting have shown that consistent relationships exist
between shoulder and elbow joint angles and velocities during reaches in two dimensions
(Soechting and Lacquaniti, 1981). Experimental evidence has also suggested that movements of
the wrist (specifically forearm pronation-supination) are consistently synchronized with motion
of the shoulder and elbow joints during reaches to targets of varying orientation (Soechting and
Flanders, 1993; Desmurget, 1995; Desmurget, et al. 1996). Thus, we believe it is fair to
hypothesize that these kinematic relationships are also present during reaches in three-
dimensions.
The primary hypothesis to be tested by this work is that able-bodied subjects can learn to use
volitional shoulder movement to control elbow flexion-extension, forearm pronation-
supination, and hand-opening and closing movements in a 3D virtual reality reach and grasp
task. We also wish to answer two secondary questions throughout this work:
24

1) Can artificial neural networks fit these postural synergies? The angle-angle relationships
exploited in this work are non-linear and non-linear regression machine learning
algorithms may be used to fit them. In the work presented here, three-layer artificial
neural networks (ANNs) trained with back propagation were used to fit the postural
synergies. ANNs are mathematical models which can be used for non-linear regression
applications. ANNs were first developed in 1954 by Minsky (Minsky 1954) and became
popular during the 80s and 90s in the field of robotics, artificial intelligence, and
computer science, among others (Hecht-Nielsen 1990). While there are other more
modern non-linear regression models that can be used, we decided to use ANNs due to
their relative ease-of-use and speed of training. We did evaluate other techniques such
as support vector machines and Gaussian process regression, but found the training
time to be too long and improvements in the results vs. ANNs to be insignificant.
2) Can able-bodied subjects learn to use novel synergy-based neural networks? Lacquaniti
et al. artificially extended the lengths of subjects’ forearms by attaching a pole that
effectively doubled the length of the forearm (Lacquaniti, et al. 1982). When asked to
move between an initial and target position with no other instructions, subjects
immediately were able to reach the target position while maintaining a straight end-
point path. Likewise, the subject automatically adjusted the angular trajectories to
accommodate the added length of the arm. It is possible that postural synergies differ
across subjects, perhaps related to anatomical differences in arm dimensions or
muscularity or in prior motor experience. Additionally, there is variability in the
calibration of joint angles due to variations in the position of sensors from session to
session. Thus, when an operator uses a novel neural network derived from another
subject, it is likely that the output of the neural networks will differ from the expected
25

distal joint angles. This scenario will be typical when fitting a prosthetic system for a
patient whose original motor synergies are unknown. We hypothesize that subjects can
quickly learn to modify the movements of their shoulder to generate reliable motion at
the endpoint, much as they learned to compensate for physical alterations such as the
extended arm experiment.
The following thesis is separated into three chapters. In chapter two we investigate whether
artificial neural networks can learn to fit the postural synergies between the shoulder/elbow
angles and shoulder/forearm angles recorded during reaches to targets in a horizontal 2D plane.
We also examine the real-time errors between predicted and actual elbow angles while using
both intra-subject and inter-subject trained neural networks. In chapter three, we extend the
training of neural networks to data recorded during reaches to a target located in 3D
extrapersonal space and with varying orientations. In this chapter we arrive on set of networks
which can predict elbow angle, forearm pronation-supination, and hand opening and closing
offline. In chapter four, we deploy the entire inferential command scheme in a 3D virtual
environment. We examine whether subjects can learn to use the command scheme for a simple
reach and grasp task with neural networks trained on their own data and a separate novel
subject’s data. We also compare their performances with subjects using only motion capture to
complete the task. In the final chapter, we discuss the implications of this work and potential
applications for prosthesis control.
The chapters in this thesis are based on published or future submitted journal articles:
Chapter two is based on work that has been published:
Kaliki, R.R., Davoodi, R. and Loeb, G.E. “Prediction of elbow trajectory from shoulder angles
using neural networks.” Int. J. Computat. Int. Apl. 7(3): 333-349, 2008.
26

Chapter three includes unpublished results and work that has been published:
Kaliki, R.R., Davoodi, R., Loeb, G.E. “Prediction of distal arm posture in 3-D space from shoulder
movements for control of upper limb prostheses.” Proc. IEEE. 96(7): 1217-1225, 2008.
Chapter four is based on work that is unpublished but will be submitted
27

Chapter 2: Prediction of elbow
trajectory from shoul der angles using
neural networks
Kaliki, R.R., Davoodi, R., Loeb, G.E.
CHAPTER 2 ABSTRACT: Patients with transhumeral amputations and C5/C6 quadriplegia may be
able to use voluntary shoulder motion as command signals for powered prostheses and
functional electrical stimulation, respectively. Spatiotemporal synergies exist between the
shoulder and elbow joints for goal-oriented reaching movements as performed by able bodied
subjects. We are using a multi-layer perceptron neural network to discover and embody these
synergies. Such a network could be used as a high level controller that could predict the desired
distal arm joint kinematics from the voluntary movements of the shoulder joint of an able-
bodied subject. We evaluated this for a task that involved reaching to 16 targets in a horizontal
plane. After reaching reasonable offline prediction accuracy for our neural networks we then
deployed the best network to make real-time predictions of the elbow angles and examined its
performance on both inter- and intra-subject trials. Finally, we extended the model to utilize
the five degrees-of-freedom at the shoulder to control the five degrees-of-freedom required for
a prosthetic arm and hand to reach and grasp variously oriented objects in the extrapersonal
workspace. Such a system, although very simple, was readily controllable for a reach and grasp
task presented to the subject in a virtual reality environment.
Contains sections from:
Kaliki, R.R., Davoodi, R. and Loeb, G.E. “Prediction of elbow trajectory from shoulder angles
using neural networks.” Int. J. Computat. Int. Apl. 7(3): 333-349, 2008.
28

Introduction
It is clear from previous work that synergies between upper and lower arm segments can be
used, to some extent, to predict goal-oriented reaching movements (Popovic and Popovic 1998,
Popovic and Popovic 2001, Iftime, Egsgaard and Popovic 2005, Mijovic, Popovic and Popovic
2008) .
In this paper, we describe a multi-layer feed-forward neural network used to model
shoulder/elbow and shoulder/forearm synergies. Unlike Mijovic et al., who used the predicted
elbow angle to predict forearm pronation/supination angle, we used the shoulder angles as
predictors for forearm pronation/supination angle. We believe that using a predicted value to
predict another variable could introduce unnecessary errors. In order to better characterize
movements across a two-dimensional workspace, all three joint angles present at the shoulder
joint were used as inputs to the network, leaving translational motion from sternoclavicular
motion for mapping to hand orientation and grasp. Preliminary work suggested that the spatio-
temporal relationships between joint angles are relatively stable and simple in shape and,
therefore, more likely to result in a user-controllable system compared to joint acceleration
relationships. Finally, as opposed to training the networks with only a single reach, we use a set
of reaches to different targets across the workspace in the network’s training set to train a
network. Once the network was trained, we examined the network’s ability to predict joint
angles during intra-subject and inter-subject real-time deployment.
Clustering of joint angle synergies
Postural synergies change across a workspace. As the subject moved to different areas of the
workspace, the shape and orientation of the shoulder/elbow angle varied greatly (Fig. 1A). In
29

certain areas of the workspace, target reaches exhibited similar spatio-temporal synergy
relationships. An example of the similarly shaped synergies is shown in Fig. 1B.
Due to the “clustering” of synergies in the workspace, we hypothesize that not all targets in the
workspace are necessary to train the neural network. In fact, it is possible that including all
targets in the workspace in the training set may hinder the neural network training because
synergies that occur at a greater probability will be weighted more strongly by the network. To
examine this supposition we trained feed-forward multi-layer perceptron (MLP) neural networks
with incrementally added target reaching data and determined the significance of using a
reduced training set as opposed to a complete training set.
-0.2 0 0.2 0.4 0.6 0.8 1 1.2
-1
-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
S
FE
angle
E
FE
angle
8P
6D
1P
3D
5P
4P
3P
2P
A B

Figure 1. Example shoulder/elbow synergies. A. Shoulder/elbow joint synergies of target reaches 1P, 8P, 3D, and
6D. B. Shoulder/elbow joint synergies of target reaches 2- 5P. Clearly there is a cluster of synergies for EFE vs. SFE
synergies.
Methods
Two-dimensional point-to-point reaching experimental setup
An experimental workspace was designed such that hand position was kept in a horizontal plane
during a reach. Sixteen elongated, rectangular-shaped targets (four and half inches in length and
one inch wide) were placed in two concentric arcs on a pegboard as shown in Fig. 2. The distal
30

target set was placed on a circular arc near 80% of the maximal reach of the subject while the
proximal target set was placed at the midpoint between the distal target set and the initial
position. In each set, eight targets were spaced 22.5° from each other on the arcs.

Figure 2. A schematic drawing of the experimental workspace.
A Plexiglas® cover was placed over the workspace to allow the subject to see the targets while
keeping their hand in the x-y horizontal plane during reaching. This helped reduce errors due to
variance in movement in the z-direction. An able-bodied subject was seated in an armless, high
back chair and an elastic restraint was placed underneath the subject’s arms and around their
torsos to prevent any contributions from the subject’s trunk. The experimental workspace was
placed at a height just below the subject’s elbow while held at 90° with respect to the body’s
longitudinal axis. The workspace’s initial position was placed along the body’s midline. The edge
of the workspace was placed at a distance of one third of the subject’s maximal reach along the
body midline away from the subject’s iliac crest.
Once in position, a subject was asked to make self-paced reaches from the initial position to and
from the target position. To successfully reach to a target, the subject had to place their index
31

finger within the boundaries of the rectangular-shaped target. To simplify offline analysis, the
subject was asked to pause at the target and initial positions for three seconds. The subject
reached to targets in a sequence from 1P to 8P and then 8D to 1D to complete one trial of the
experiment. For this experiment, one subject was tested.
Reaching and grasping vertically- and horizontally-oriented targets on a plane
To examine the strength of synergies between shoulder joint angles and forearm
pronation/supination, we designed an experiment in which subjects would reach and grasp
objects in both horizontal and vertical orientations. The targets were cylindrical in shape (five
inches in height, two and half inches in diameter) and were placed directly over the same target
locations as used in the point-to-point reaching experiment (Fig. 2). The subject was asked to
reach to the target, grasp the target for three seconds, and return to initial position. The target
was first placed in the horizontal orientation and then was placed in the vertical orientation.
The subject was instructed to reach to the reset target between each target reach. This
sequence was repeated at all 16 target locations. To complete the experiment, the subject was
asked to finish three trials. Neural networks trained with this data used the three joint shoulder
joint angles as inputs to predict elbow flexion/extension and forearm pronation/supination at
the outputs.
Data acquisition and pre-processing
A Flock of Birds® magnetic motion tracking system (Ascension Technology Corp., Burlington, VT)
was used to track the motion of segments of the subject’s arm at a sampling rate of 100 Hz.
Sensors were attached to the clavicle, humerus, and ulna segments. Each sensor measures the
position of the attachment point and the orientation of the segment as a nine-element rotation
matrix relative to a transmitter. The motion tracking system was calibrated to each subject prior
32

to any experiment. Euler coordinate transformations were used to calculate joint angles in
clinical coordinates from the rotation matrices (for detailed calibration and angle extraction
methodology refer to Hauschild et al., 2007). The calculated shoulder joint angles were shoulder
abduction/adduction (S
ABAD
), the angles about the x-axis of the fixed reference frame (Fig. 2),
shoulder flexion/extension (S
FE
), the angle about the y-axis of the moving frame, and internal
external rotation (S
IER
), the angle about the z-axis of the moving frame. The other recorded
angles were sternoclavicular depression/elevation (SC
DE
), sternoclavicular protraction/retraction
(SC
PR
), elbow flexion/extension (E
FE
) and forearm pronation/supination (F
PS
). The recorded data
were filtered offline at three hertz cutoff frequency using a third order Butterworth low-pass
filter. The filter was used to remove four hertz noise present in the recorded motion data. This
noise, unrelated to the reaching motions, originated from our motion tracking system as
evidenced by the presence of the noise when the sensors were removed from the subject. The
filter had no discernible effect on the recorded trajectories. Lastly, the data were normalized to
fit within a range of negative one to one based on the minimum and maximum of the joint
angles measured during an entire experiment.
Neural network training
Two trials of the experiments were used as inputs to the neural network in order to promote
generalization. The data was uniformly distributed into these two sets: 70% of the data were
used as training data while 30% were used as a test set. Data from the third trial of the
experiments were used as a validation set independent of the training data with which the
trained neural network’s performance was evaluated and compared to other networks’
performances. For the point-to-point reaching experiment, the inputs to the neural networks
(NN1) were the angular positions of S
FE
, S
IER
, and S
ABAD
, while the output was E
FE
. For the reach
and grasp experiment, the inputs to the neural networks (NN2) were the same as the previous
33

neural networks but the outputs were E
FE
and F
PS
. Three-layer (input, hidden, and output layers)
MLP neural networks were created in NeuralWorks Predict® (NeuralWare). This software
employs an adaptive gradient backpropagation algorithm to tune the weights of the MLP to
maximize the correlation between the model predictions and the recorded data. In addition,
hidden units with hyperbolic tangent (tanh) activation functions in a single hidden layer were
added incrementally to improve the performance of the network. The output units were logistic
sigmoid activation functions. NeuralWorks Predict also uses a cascade learning algorithm
created by Fahlman and Lebiere (Fahlman and Lebiere 1990). This algorithm adds hidden units
incrementally to the hidden layer until performance is no longer improved. The software also
uses a method of early stopping to prevent the MLP from overfitting. Early stopping examines
the performance of the MLP during training by examining its performance on the validation set.
If the network’s performance on the validation set is no longer improved then training is
stopped.
Networks were trained with incrementally added target reaching data. Initially, a network was
trained with a single target reach. Due to the redundancy in joint angle synergies in the
workspace, the networks trained on certain target reaching data would be able to predict for
other non-trained target reaches. The single target reach data that trained a network that was
able to estimate the most reaches accurately was used as the basis for the network’s training
set. After each training cycle was completed, the network’s performance on the validation set
was tabulated. Specifically we measured the mean squared error (MSE) and final angle error
(FAE). The FAE was designated as the absolute difference between the joint angles predicted by
the neural network and the recorded joint angles at final position of the arm after a reach to a
specified target. The target which was predicted the worst, based on the FAE, was added to the
training set and the network was re-built and retrained. This process was continued iteratively
34

until all target reaches in the workspace were added to the training set. Finally, we used the
paired t-test to determine the significance of the performance of the network trained with the
minimized training set as compared to the network trained with the complete training set.
Evaluating neural networks in real-time tasks
After neural network training was completed, we deployed the best performing neural network
in real-time to determine its real world ability to predict joint angles. The networks were tested
both on the subject with whose data the network was trained (intra-subject test) and on a
subject whose data was not used to train the network (inter-subject test). Like the previous
experiment, sensors were attached to a subject and the two-dimensional workspace was scaled
to the subject and moved into place. Subjects were asked to make point-to-point reaches to
eight targets spaced 22.5º apart on an arc located at the 75% of their maximal reach in the
target direction. The trained neural network was presented with input data measured from the
subject in real-time. The predicted neural network output and the actual joint angles were
recorded and analyzed offline. The MSE and FAE were tabulated for both tests.
Evaluating neural network models in virtual reality tasks
To further evaluate the utility of the synergistic models in upper limb neural prostheses, we
designed a simple controller which maps the five degrees-of-freedom at the shoulder to the five
degrees-of-freedom at the distal portion of the upper limb. The three shoulder rotational angles
were designated as inputs to a neural network which predicted the E
FE
. SC
DE
was used to control
the F
PS
angle. The forearm would supinate when the subject elevated his/her shoulder above a
threshold. The threshold was designated to be eight degrees. As long as the shoulder elevation
was above threshold the forearm supination angle would continue to increase at a rate of 20
deg/s. To pronate the forearm, the shoulder would need to be depressed beyond a negative
35

threshold of negative three degrees. Below this threshold the forearm pronation angle would
increase at a rate of 20 deg/s. When the SC
DE
level was below both thresholds the F
PS
angle
would remain fixed until the SC
DE
crossed a threshold. To control grasp, SC
PR
was mapped to the
closing and opening of the hand. If the subject made a movement with which the SC
PR
crossed a
threshold of eight degrees, the grasp would turn on and the hand would close. When the SC
PR

went below a negative threshold of negative three degrees, the grasp would turn off and the
hand would be opened.
To examine a patient’s ability to use such a simple controller, we deployed the controller in a
virtual reality environment (VRE) (Davoodi, et al. 2007). In this environment, the subject views a
skeletal model in computer screen, which responds to the joint angles recorded by the motion
tracking system and those outputted by the controller (Fig. 3).

Figure 3. A screen capture of our virtual reality environment.
In our VRE, the subject attempted to reach to and grasp a small virtual block on a virtual table.
The object was randomly placed in one of eight locations on the table within the reaching space
of the subject and the subject was asked to control the degrees-of-freedom of the arm to reach
36

to the object, match the orientation of the object, and grasp the object. To grasp the object
several conditions had to be met: the virtual hand should be close enough to the object, the
orientation of the hand should match the orientation of the object, the velocity of the hand
should be near zero, and the hand should be closed around the object. Once the object was
grasped, the subject was asked to bring the object back to a box near the torso of the skeletal
model. To complete the task, the subject needed to drop the object in the box. After the object
was placed in the box, the object’s was placed in a random location away from the box within
the subject’s reaching space. The object’s position away from the box was held constant but
varied in the x- and y- directions. During the experiment, we measured the time to complete
the task. The experiment was completed after the subject had successfully completed the task
16 times. The subject was not previously familiar with the task and was only given a brief
demonstration on how to control the various degrees of freedom. Additionally, the neural
network was not trained on this subject’s reaching data.
Results
Neural network off-line performance
The final angle errors and mean squared errors as a function of number of training targets for
NN1 are shown in Fig. 4. The network with the best performance was one trained with a
minimized training set (NN1[minimized]) containing 10 target reaches. This set included target
reaches: 1P, 5P, 8P, 1D, 2D, 3D, 4D, 5D, 7D and 8D. This network had an average FAE of 2.1° and
a MSE of 5.6 and had 13 hidden layer units. The performance of this network on the validation
set is shown in Fig. 5. The network trained with the complete set of training data (NN1[all])had
an average FAE of 3.0° and a MSE of 7.0 and had 14 hidden layer units.
37

0
2
4
6
8
10
12
14
16
0 2 4 6 8 10 12 14 16
Number of Training Targets
Final Angle Error (° )
0
20
40
60
80
100
120
140
0 2 4 6 8 10 12 14 16
Number of Training Targets
Mean Squared Error
A B

Figure 4. Offline performance vs. number of training targets. A. Plot of final angle error as a function of number of
training targets. B. Plot of mean squared error as a function of training targets.
0 20 40 60 80 100 120 140 160 180
10
20
30
40
50
60
70
80
90
100
110
Time (s)
Elbow Angle ( °)
Output of Network Trained with Minimized Training Set

Recorded EFE
Model EFE

Figure 5. The performance of the neural network trained with the minimized set. Output of neural network is
shown on the validation set.

Network Output FAE MSE
NN1
[minimized]
E
FE 2.14 5.61
NN1
[complete]
E
FE 3.00 7.08
E
FE
5.62 32.82
F
PS
26.48 594.37
NN2

Table 1. Comparison of neural networks’ performances on the validation set.
38

We used a paired t-test to determine the significance of the performance of the network trained
on the minimized training set. We used the FAE information from the 16 targets in determining
the t-value. We obtained a t-value of 1.93 and a probability of null hypothesis to be 0.073. This
p-value is not below the conventional criteria of p = 0.05 to reject the null hypothesis.
Networks (NN2) trained with the reach and grasp data did not reach a minimum in terms of
error prior to adding all 16 training targets to the training set. The FAE and MSE for the E
FE
output were 5.6º and 32.8, respectively. For the F
PS
output the FAE and MSE were 26.5 º and
594.4. The results of all three neural networks are shown in Table 1.
Real-time deployment of trained neural network
The neural networks deployed in real-time in the intra-subject test and inter-subject test are
shown in Fig. 6. The FAE and MSE for the intra-subject test were 7.4º and 40.8, respectively.
The FAE and MSE for the inter-subject were 10.4 º and 80.4, respectively.
0 10 20 30 40 50 60
50
60
70
80
90
100
110
Time(s)
E
FE
Angle(°)
Intra-Subject Real-Time Neural Network Performance
0 5 10 15 20 25 30
30
40
50
60
70
80
90
100
Time(s)
E
FE
Angle(°)
Inter-Subject Real-Time Neural Network Performance

recorded E
FE
predicted E
FE
B
A

Figure 6. Plots of real-time neural network performance. A. Plot of intra-subject real-time neural network
prediction in the point-to-point reaching task. B. Plot of neural network performance during real-time deployment
on inter-subject test.

39

0
5
10
15
20
25
30
35
40
1 3 5 7 9 11 13 15
Number of Successful Task Completions
Time to Complete Task (s)

Figure 7. Plot of task completion time vs. number of successful task completions.
Virtual reality reach and grasp task
The subject was able to reach and grasp to all targets presented in the virtual workspace. The
average task completion time was 21 seconds. The plot of the time to complete a reach vs. the
number of task attempts is shown if Fig. 7.
Discussion
Contrary to the previous work by Popovic et al we have shown that one trained MLP neural
network was able to predict reaches to targets across the entire workspace, as opposed to
having several neural networks for different areas of the workspace and having to switch
between them. The addition of the two remaining rotational angles at the shoulder, S
IER
and
S
ABAD
, in the input layer and the addition of more varied reaching data in the training set
accounted for the differences in performance. It should be noted that we did not conduct a
comparison of learning techniques. We chose MLP neural networks because of their abilities to
generalize data well and their high level of performance on this task. We have yet to examine
other learning techniques and compare their performance to the MLP neural networks.
The results from the point-to-point reaching experiment indicate that a neural network is able
to accurately predict the E
FE
from the three shoulder angles for reaches across a two-
40

dimensional workspace. Using a subset of target reaches in the training set resulted in an
overall less average FAE and MSE as compared to a network trained with all target reaches.
While not significant, this result indicates that not all the reaching data in the workspace is
necessary to train the neural networks. From Fig. 3, we see that both the FAE and MSE fall
dramatically after adding up to four targets in the training set, then gradually decrease and
reach a minimum with ten targets in the training set. After this point, the errors gradually
increase until all targets are added to the training set, indicating that the errors could potentially
worsen as more and more targets are added. Therefore, in three-dimensional reaching, using a
subset of the reaching data may prove to be significantly more accurate due to the increase in
target numbers and workspace size.
From the reach and grasping experiment, it is clear that shoulder angles alone cannot provide a
good prediction for the forearm pronation-supination angle. The neural network trained to
simultaneously predict EFE and FPS resulted in larger errors for the prediction of EFE as
compared to the previous networks and irrecoverably large errors for FPS prediction. It may be
possible that other inputs such as sternoclavicular depression/elevation and sternoclavicular
protraction/retraction can be used to predict forearm pronation/supination. Furthermore, it is
likely that the simultaneous prediction of two outputs by one neural network may result in
degraded performance on both outputs. It is also possible that the 2D task used in this work
forced a decoupling between reaching and orientation of the hand similar to studies by
Soechting and Lacquaniti (Lacquaniti and Soechting, 1982; Soechting, 1984). These results do
indicate that the previous work, in which forearm pronation/supination was predicted by elbow
flexion/extension, by Iftime et al. examined a task that was probably overly stereotyped (Iftime,
Egsgaard and Popovic 2005). By presenting a task in which the orientation of the object varies
we see that the predictability of the forearm pronation/supination is greatly decreased.
41

Examining the performance of the neural networks in real-time shows that there are some
errors between the predicted and actual EFE angles during deployment. The errors measured
during the intra-subject test were decidedly larger than those obtained in the offline evaluation.
These can be attributed to variations in sensor position as compared to when the data to train
the networks was acquired or improper tracking of the joint angles. The larger errors in the
inter-subject test are due to the potential variability between subjects and their reaching
strategies. This seems specifically evident at the edges of the workspace where the errors seem
the greatest. But, these errors are expected and the question is not whether the controller can
accurately match the actual elbow angle but instead the question is if the errors are small
enough that the subject can learn to use the controller with relative ease.
From the final experiment we see that the subject was able to learn to use the controller to
reach, grasp, and manipulate a virtual object. As expected there was a slight learning curve to
use the virtual controller. Initially, the task completion time was around 33 seconds and then by
the end of the experiment was around 11 seconds. After eight trials the subject learned to
reduce the time required to complete the task to consistently below 20 seconds. The subject
remarked that for certain target locations they had difficulty determining the hand’s three-
dimensional position relative to the object because the virtual environment was presented on a
two-dimensional display.

42

Chapter 3: Prediction of distal arm
posture in 3-D space from shoul der
movements for control of upper limb
prostheses
Kaliki, R.R., Davoodi, R., Loeb, G.E.
CHAPTER 3 ABSTRACT: C5/C6 tetraplegic patients and transhumeral amputees may be able to use
voluntary shoulder motion as command signals for a functional electrical stimulation (FES)
system or a transhumeral prosthesis. Such prostheses require, at the most basic level, the
control of endpoint position in three-dimensions, hand orientation, and grasp. Spatiotemporal
synergies exist between the proximal and distal arm joints for goal-oriented reaching
movements as performed by able bodied subjects. To fit these synergies we utilized three-layer
artificial neural networks. These networks could be used as a means for obtaining user intent
information during reaching movements. We conducted reaching experiments in which
subjects reached to and grasped a handle in a three-dimensional gantry. In our previous work,
the three rotational angles at the shoulder were used to predict elbow flexion/extension angle
during reaches on a two dimensional plane. In the present study, we extended this model to
include the two translational movements at the shoulder as inputs and an additional output of
forearm pronation/supination. Counter-intuitively, as the complexity of the task and the
complexity of the neural network architecture increased the performance also improved.
Contains sections from:
Kaliki, R.R., Davoodi, R., Loeb, G.E. “Prediction of distal arm posture in 3-D space from shoulder
movements for control of upper limb prostheses.” Proc. IEEE. 96(7): 1217-1225, 2008.
43

Introduction
Typically, quadriplegic patients have substantially intact voluntary shoulder motion because the
shoulder muscles are usually innervated above the level of the spinal cord injury. Specifically
patients with C5-C6 level spinal cord injury retain nearly normal control of their shoulder and
elbow flexion but have no control of elbow extension or wrist or finger movement. Similarly,
patients with transhumeral amputations retain most of their shoulder functionality.
Both C5-C6 quadriplegics and transhumeral amputees require the restoration of the ability to
position the hand in three-dimensional space, orient the hand to match an object, and close and
open a hand to grasp and release an object. Positioning the hand in three-dimensional space
requires determination of the shoulder and elbow joint angles. Interestingly, stereotypical
behavior between the shoulder and elbow joints emerges during reaches to positions in space.
Specifically, the angular motions of the shoulder and elbow joints are consistently synchronized
to give rise to straight hand paths and bell-shaped hand velocity curves during reaches to a
given target in space (Soechting and Lacquaniti, 1981). In our previous work we showed that
the three rotational joint angles at the shoulder are sufficient to predict the elbow joint angle
during reaches to targets on a horizontal 2D plane over multiple trials. Additionally, because the
arm was not constrained to movement in the horizontal plane it can be reasonably assumed
that elbow angle can also be predicted during reaches to targets in 3D extrapersonal space.
Furthermore, neurophysiological evidence has shown that the straight hand paths and bell-
shaped velocity curves are preserved during reaches to targets in three-dimensions (Morasso,
1983; Soechting et al., 1995; Nishikawa et al., 1999), indicating the possibility of stereotyped
relationships between the shoulder and elbow joints. In the latter study, Soechting et al. found
44

that final posture of the arm during 3D point movements was a function of the initial posture.
Therefore, in the work presented here we strictly controlled the initial position of the hand.
Studies have also shown that shoulder, elbow, and forearm motion are stereotypically
coordinated during reach and grasps to targets of varying orientation in the frontal plane
(Desmurget et al., 1995; Desmurget et al., 1996). In the previous chapter we found that artificial
neural networks trained with three shoulder rotational angles to predict forearm angle were
insufficient, either due to the lack of relevant information in the motion of the shoulder angles
or due to the constraints of the 2D task. Thus, in this work we extend the set of inputs to
include the translational movements of the shoulder and record joint angles during reaches to
targets with varying orientations in a subject’s 3D extrapersonal space. We hypothesize that
artificial neural networks can fit synergies between the shoulder and forearm
pronation/supination angle. Lastly, hand opening and closing must be specified to have a
complete system. We investigated whether or not shoulder joint angles could predict the onset
of hand grasping.
In the previous chapter, we found that using a subset of all the reaches to targets as a training
set resulted in equally good performance as networks trained with the entire training set. We
hypothesized that using a subset of training data could remove redundant or weak synergies
and improve performance of artificial neural networks. In this study, we tested this hypothesis
by investigating the effect of training data on neural network performance.
45

Methods
Reaching and grasping task
We constructed a large robotic gantry (Parker Hannifin, Co.) to automate the presentation of
targets in 3D workspace of the arm. The gantry was able to reach anywhere in a 2m x 1m x 1m
workspace (Fig. 8A). The working end of the gantry included a handle which was able to rotate
to one of four designated rotations (0°, 45°, 90°, or 135°) in the frontal plane. The handle was
composed of a primer bulb sealed on one end and pressure sensor capable of sensing from 0 –
30 psi at the other end (Fig. 8B). The gantry was controlled through LabVIEW (National
Instruments Corp.) and indicators and a graphical representation of gripping pressure were
displayed on screen in front of the subject. Prior to experimentation target locations were
tailored to the subject’s physical measurements. Targets were presented in pseudo-random
positions in 3D extrapersonal space and with random orientations. Motion tracking of the
relevant joint angles was performed as the subject reached and grasped the handle. Once the
handle was grasped, the subject needed to generate sufficient gripping pressure and then
return to a designated initial position. The initial position required the subject to position the
hand in a designated orientation in order to maintain a consistent initial posture during the
experimentation.
Target locations
Target locations were expressed in shoulder-centric spherical coordinates: α, β, and f
(corresponding to the fraction of the subject’s entire arm length). Measurements were taken of
the subjects upper (L
1
) and lower arm (L
2
) segments, as well as the height of the subject’s line of
sight above the shoulder center of rotation (L
3
) to constrain the target locations and tailor them
to the subject (Fig. 9). We first defined a set of targets in X-Z plane which was then rotated
46

about the z-axis to produce target locations in three dimensions. We then applied the
constraints to the target set to ensure that all the targets were in a reasonable working space. L
3

and L
1
limited the maximum and minimum height of targets in the y- direction, respectively. To
ensure the safety of the subject during experimentation we kept the subject’s head and body
outside of the reachable area of the gantry arm. This limited the location of the front-most
targets in our workspace. The distance between the front-most possible target and the
shoulder center of rotation along the x-axis was measured (L
4
) and was used to determine the
minimum angle of α at the subject’s maximal reach (f = 1.0) in the two-dimensional plane.

Figure 8. The robotic gantry that was used to present targets to a subject in 3D extrapersonal space. 1B: Close-up
of gantry handle.
A B
47

-15 -10 -5 0 5 10 15 20 25
0
5
10
15
20
25
z axis (inches)
x axis (inches)
α
min
L
1
L
2
L
4
z
x
A
L
1
L
3
x
y
β
B

Figure 9. Extrapersonal space definition. A top down view of the two-dimensional workspace boundary. The angle
α is shown as well as significant physical measurements taken of the subject. This plane is rotated about the z-axis.
B: A side view of the workspace.
When the upper arm crosses the sagittal plane at shoulder joint (
o
90 ≥ α ), the maximal reach
length is no longer equivalent to the subject’s arm length due to the restriction of movement as
the humerus makes contact with the torso. In order to reduce undesirable contributions from
the trunk in this area of the workspace, the line of maximal reach was approximated by fixing
the humerus position at
o
90 = α and bending the elbow from 0º to 90º. Targets were placed at
regular intervals in the α- and f- directions. In this experiment we had eight targets along the
line of maximal reach (f =1.0). For all other values of f, targets were placed at that radius with
the same angular interval between targets as previously determined. Once all the targets
locations in the X-Z plane were determined, the plane was then rotated about the z-axis from β
= 90º to β = -90º. Targets that were below the length of the upper arm segment (L
1
) in the y-
direction or were at a height greater than the line of sight (L
3
) were removed from the target
set. The angular interval between targets in the β-direction was 10º.
48

An adult male volunteered to perform reaching experiments. His physical measurements were:
L
1
=11.5”, L
2
=14.5” and L
3
= 6.5.” The fraction of maximal reach, f, was varied from 0.5 to 1.0 and
this resulted in an experimental workspace that included 744 target locations (Fig. 10) Once the
target locations were determined they were presented to the subject in randomized order by
the gantry control software.
-20
-10
0
10
20
30
0
5
10
15
20
25
30
0
10
20
x axis(inches)
z axis(inches)
y axis(inches)
-10
0
10
y axis(inches)

Figure 10. A stem plot of all the locations of the targets in extrapersonal space (blue). The shoulder center of
rotation is shown in red.
Data acquisition
In order to record the subject’s joint angles during experimentation a Flock of Birds® (Ascension
Technologies Corp., Burlington, VA) motion capturing system was used with a sample frequency
of 100 Hz. Each Flock of Birds® sensor measured position and orientation (measured in rotation
matrices) with respect to a transmitter. Sensors were placed on the shoulder (over the
acromion), humerus, and over the wrist (at the distal end of radius). The transmitter was placed
between the subject’s knees on the chair. The Flock of Birds® system was calibrated to the
subject prior to experimentation. Clinically meaningfully Euler angles were derived from the
49

rotation matrices (Euler rotations in X-Z-Y order about the moving axes). The calculated
shoulder joint angles were shoulder abduction/adduction (S
ABAD
), the angles about the x-axis of
the fixed reference frame (Fig. 11), shoulder flexion/extension (S
FE
), the angle about the z-axis of
the moving frame, and internal external rotation (S
IER
), the angle about the y-axis of the moving
frame. The other recorded angles were sternoclavicular depression/elevation (SC
DE
),
sternoclavicular protraction/retraction (SC
PR
), elbow flexion/extension (E
FE
) and forearm
pronation/supination (F
PS
). These values along with the original rotation matrices and the
location of the targets were recorded in a synchronized manner by the experiment control
software.

Figure 11. A schematic of the subject during experimentation. The locations of the Flock of Birds® sensors are
shown. The coordinate system used in determining the clinically meaningful Euler joint angles is shown. The
recorded joint angles were: shoulder abduction/adduction (S
ABAD
), the angles about the x-axis of the fixed
reference frame, shoulder flexion/extension (S
FE
), the angle about the y-axis of the moving frame, internal external
rotation (S
IER
), the angle about the z-axis of the moving frame, sternoclavicular depression/elevation (SC
DE
), the
translation along the y-axis of the shoulder frame, sternoclavicular protraction/retraction (SC
PR
), the translation
along the x-axis of the shoulder frame, elbow flexion/extension (E
FE
), the angle about the z-axis of the elbow frame,
and forearm pronation/supination (F
PS
), the angle about the x-axis of the elbow frame.
Subjects were seated in a high back chair and secured with elastic bands around the torso to
limit movement of the trunk during experimentation. Once the subject was secured in the chair
and the motion tracking system was calibrated, a lap tray holding a push button switch was
50

moved into place approximately five inches above the subject’s lap height. The switch was
placed directly in front of the subject, along the midline of the body. The subject was instructed
to start the experimentation by pressing down on the switch with the instrumented hand.
Targets from the predetermined reaching space were pseudo-randomly presented to the
subject. The subject was told to move to each target at a comfortable, self-determined pace and
grasp the handle Visual feedback of gripping pressure was presented on a screen in front of the
subject. Subjects were instructed to squeeze such that the gripping pressure exceeded a
threshold value, which was displayed on the screen. The threshold value was determined prior
to experimentation by examining what level of gripping pressure elicited a protraction of the
shoulder. After crossing the threshold the subject was instructed to move back to the initial
position and press the switch to cue the gantry to move to the next target. The experiment
continued until all target locations were reached. A “home switch” was placed at the initial
position of the subject, which once depressed indicated when reaching data was not recorded.
Data preprocessing and partitioning
After experimentation the joint angle and pressure data was filtered offline with a 3 Hz third
order Butterworth low-pass filter. The 3 Hz filtering was necessary to remove some 4 Hz noise
present in our motion capturing hardware as evidenced by the presence of the noise in the
sensors prior to attachment to the subject. The data were then down sampled to 8 Hz to reduce
the data size and normalized by subtracting the mean from each channel and dividing by the
standard deviation. Finally, data recorded during resting periods between target reaches were
removed in order to limit the contribution of the initial posture to the neural network from each
data set.
51

Prior to data partitioning, three different primary sets of target reach data were created. The
first primary set (labeled a) included data from the entire trajectories (reach plus hold period) to
each of the 744 targets. In the second primary set (b), we removed all the target reach
information that contained reaches in which the maximum elbow angle was less than 10º
different from the initial elbow angle. Shoulder motion accounted for most of the movement to
these targets, so they were not useful for training elbow angle output. After we removed these
target reaches, the data set included 584 target reaches. The third primary data set (c) included
only those targets from set B that were at f > 0.8. This limited the data set to 200 target
reaches.
Prior to artificial neural network (ANN) training, the primary data sets were divided into two
sets: a secondary working set, with which the ANN was trained, and a validation set, which
included novel data to evaluate the performance of the ANN. 20% of the data from the primary
data set were randomly chosen and set aside as the validation set. The remaining data were
designated the secondary working set. From the secondary working set, 70% of the data were
randomly distributed in a training set and the remaining values were designated as the test set.
The training set included data with which the ANN would be trained via backpropogation and
the test set was novel data used to measure the ANN’s ability to generalize during ANN training.
Neural network training
Three-layer perceptron ANNs were created in NeuralWorks Predict® (NeuralWare). This
software employed an adaptive gradient backwards propagation algorithm to tune the weights
and biases of the ANN to maximize the correlation between the model predictions and the
recorded data. Hidden units had hyperbolic tangent activation functions. The output units were
logistic sigmoid activation functions. Hidden layer size was determined through a cascade
52

learning algorithm developed by Fahlman and Lebiere (1990). This algorithm adds hidden units
incrementally to the hidden layer until performance on the test set is no longer improved. The
software also used early stopping to prevent the ANN from overfitting and to improve
generalization. Early stopping examines the performance of the ANN during training by
examining its performance on the test set. If the network’s performance on the test set is no
longer improved then training is stopped.
ANNs were constructed initially with four different input/output (I/O) relationships. The first set
of I/Os examined the ability of the three rotational joint angles at the shoulder joint (S
FE
, S
IER
,
and S
ABAD
) to predict the elbow angle (E
FE
) during reaches in 3D. ANNs trained with this set of
I/Os were labeled ANN1x, where x is a place holder for the primary data set type (a, b, or c). The
next set of I/Os incorporated shoulder translation movements (SC
DE
and SC
PR
) as inputs in
addition to S
FE
, S
IER
, and S
ABAD
and evaluated whether these additional inputs improved
predictability of the E
FE
. These sets of ANNs were labeled ANN2x. The third neural network was
trained using the same inputs, the five DOF at the shoulder as the previous set but predicted F
PS

as the outputs of the ANN. This ANN was only trained with primary data set “c,” because this
set consistently resulted in the highest performance. This ANN was created to examine the
potential to predict forearm pronation-supination during reaches to targets of varying
orientation and location in 3D space.
To restore grasp we first needed to examine if postural relationships exist between shoulder and
grasping. We hypothesized that the shoulder may naturally protract forward during grasps of
sufficient grip strength to stabilize the motion at the shoulder. Thus, under the assumption that
gripping pressure could be equated with hand opening and closing angle, a fourth neural
network was created to determine if a predictable relationship existed between the five
53

shoulder angles and gripping pressure. This network had the same inputs as the previous
network but predicted gripping force as the output.
Unfortunately, results indicated that five shoulder angles used could not reliably predict the
grasping pressure. A complete system requires the control of grasp, thus we devised an
alternative method. Because grasping pressure could not be reliably predicted from the five
shoulder DOFs, we hypothesized that the shoulder protraction/retraction (SC
PR
) movement
could be used to proportionally control the opening and closing of the hand.
During preliminary studies, we observed that subjects would naturally protract their shoulder
forward when reaching to targets left of the sagittal plane through the shoulder center of
rotation (α > 90°). The natural protraction of the shoulder during reaching could result in
undesired closing of the hand during use. Therefore, to use the protraction/retraction angle as
a viable control signal, any protraction movement will need to be predicted by a third neural
network and subtracted from the recorded angle to allow the user to proportionally control the
opening and closing of the hand.
Because SC
PR
was already used as input to the other two ANNs described above, new neural
networks were created to ensure that the elbow and forearm angles could be accurately
predicted with only four shoulder DOFs as the input (S
FE
, S
ABAD
, S
IER
, and SC
DE
), leaving SC
PR
free to
proportionally control the closing of the hand.
Three more neural networks were trained with the following I/O relationships:
ANN5c predicted elbow angle (E
FE
) from the three rotational joint angles (S
FE
, S
IER
, and S
ABAD
) and
one translational degree-of-freedom (SC
DE
) at the shoulder.
ANN6c predicted F
PS
from the same inputs as ANN5c.
54

ANN7c used the same inputs as ANN5c and ANN6c to predict the SC
PR
angle.
The coefficient of determination (R
2
) between the predicted output and recorded output was
measured for all the ANNs. Any R
2
value above 0.7 was considered a strong correlation.
Additionally, the root mean squared error (RMS) between the predicted and recorded outputs
was measured. Because the data were normalized by the standard deviation prior to training,
the error is unitless.
Results
An example of target reach data used to train neural networks is shown in Figure 12. Table 2
shows the tabulated R
2
and RMS errors for neural networks ANN1a, ANN1b, and ANN1c. Inputs
to these neural networks were the three rotational angles at the shoulder while the output was
the elbow angle. The errors and R
2
-values are shown for each of the data sets: training, test,
and validation sets. The R
2
-values for each of the ANNs on the validation set were greater than
0.70 and, therefore, are considered strong correlations. The neural network trained with the
distal targets (f >0.8) reported the highest R
2
and lowest RMS error for each of the data sets.
Table 3 summarizes the performance of neural networks ANN2a, ANN2b, and ANN2c. The
inputs to these neural networks were the three rotational angles and two translational
movements at the shoulder. The output was the elbow angle. The R
2
-values and RMS error
values are tabulated for the training, test, and validation sets. The R
2
values for all the ANNs
were high. Neural networks trained with the distal targets (f > 0.8) achieved the highest R
2
for
both the training and test data sets and lowest RMS error for all data sets. ANN2b had a slightly
higher R
2
-value (0.87) than ANN2c (0.86) on the validation set.

55

0 1 2 3 4 5 6 7 8 9 10
-5
-4
-3
-2
-1
0
1
Time(s)
Normalized Angle

S
IER
S
ABAD
S
FE
SC
DE
SC
PR
E
FE
F
PS

Figure 12. A typical example of joint angles recorded during a reach to a target (α = 33.5º, β = 21.3º, f = 1.0). The
black, blue, and red plots correspond to shoulder rotational angles, shoulder translational movements, and distal
arm angles.
Name Set
R
2
RMS
Train 0.74 0.71
Test 0.73 0.72
Valid 0.70 0.75
Train 0.81 0.67
Test 0.75 0.77
Valid 0.78 0.72
Train 0.88 0.47
Test 0.86 0.50
Valid 0.84 0.52
ANN1a
ANN1b
ANN1c

Table 2. Summary data shown for neural networks trained with three rotational shoulder angles as inputs
predicting the elbow angle as the output. R
2
and root mean squared errors are shown for train, test, and
validation sets.
56

Name Set
R
2
RMS
Train 0.83 0.56
Test 0.82 0.59
Valid 0.82 0.58
Train 0.89 0.51
Test 0.87 0.56
Valid 0.87 0.57
Train 0.93 0.36
Test 0.92 0.38
Valid 0.86 0.50
ANN2a
ANN2b
ANN2c

Table 3. Summary data shown for neural networks trained with three rotational shoulder angles and two
translational angles as inputs predicting elbow angle as the output. R
2
and root mean squared error are shown
for train, test, and validation sets.
The training results of ANN3c and ANN4c are reported in Table 4. The F
PS
was consistently
predicted accurately across all data sets (R
2
> 0.80). These results indicate that hand orientation
determined by forearm pronation-supination can be predicted from shoulder angles during
reaches to targets in 3D space. Conversely, ANN4c did not perform very well with an R
2
of 0.18
and a consistently high RMS on the validation set.
Name Set
R
2
RMS
Train 0.82 0.57
Test 0.80 0.58
Valid 0.80 0.60
Train 0.22 1.29
Test 0.21 1.35
Valid 0.18 1.54
ANN3c
ANN4c

Table 4. Summary data shown for neural networks trained with three rotational shoulder angles and two
translational angles as inputs predicting the forearm angle and gripping pressure as the output. R
2
and root mean
squared errors shown for train, test, and validation sets.
Table 5 summarizes the performances from three neural networks predicting E
FE
, F
PS
, and SC
PR
.
All three ANNs performed well on the validation set (R
2
>0.7). ANN5c was constructed with 73
hidden units and had an R
2
value of 0.84 on the validation set. ANN6c had 93 hidden units and
performed with an R
2
of 0.75. Lastly, ANN7c had 54 hidden layer units and had an R
2
value of
57

0.74. While the R
2
values of output from ANN5c and ANN6c are lower than those achieved by
ANN3c and ANN4c, respectively, the networks still performed reasonably well because they
remained greater than 0.7.

Name Set
R
2
RMS
Train 0.88 0.49
Test 0.86 0.52
Valid 0.84 0.57
Train 0.78 0.65
Test 0.77 0.72
Valid 0.75 0.73
Train 0.76 0.69
Test 0.74 0.70
Valid 0.74 0.72
ANN5c
ANN6c
ANN7c

Table 5. Summary data shown for neural networks trained with three rotational shoulder angles and one
translational angle (SC
DE
) as inputs predicting the elbow flexion-extension, forearm pronation-supination, and
shoulder protraction-retraction angles as the output, respectively. R
2
and root mean squared errors shown for
train, test, and validation sets.
Discussion
From the results of ANN1a,-b, and -c it is clear that neural networks using three shoulder
rotational angles to predict the elbow angle during reaching in two-dimensions can be extended
to adequately predict the elbow angle for reaching movements in a large three-dimensional
extrapersonal space. Additionally, removing both weaker synergies and proximal targets
improves the performance of the network.
Reaching in two-dimensions did not require the additional shoulder translational movements
because all targets were in the horizontal plane. Presenting targets in three-dimensions forced
the subject to occasionally make scapuloclavicular movements. Adding shoulder translational
movements as inputs to the neural networks resulted in an increase in performance.
58

The shoulder rotational angles and translational movements sufficiently predicted the forearm
pronation-supination angle during reaches to object of varying orientation. Offline prediction of
forearm angle was worse than elbow angle prediction, which may lead to large errors during
real-time deployment. Yet because hand orientation in the frontal plane is a function of both
forearm angle and shoulder joint angles, it is likely that subjects can learn to compensate for the
errors in forearm angle by making corrective movements with the proximal joints.
Conversely, we found that five shoulder angles used could not reliably predict the grasping
pressure. This may indicate that the artificial neural network failed to capture this relationship
or that there is no consistent relationship between the shoulder joint and hand grasp. The latter
conclusion is supported by studies that have provided evidence that the planning of hand
transport is performed separately from planning of hand manipulation (Jeannerod, 1984;
Gentilucci et al., 1991). Assuming that the results presented here support this conclusion, we
devised an alternative method for the control grasp.
We propose using the protraction of the shoulder joint to map proportionally to the opening
and closing of the hand. As stated earlier, the subject naturally protracted the shoulder during
reaches to certain parts of the workspace. If the subject reached to these parts of the
workspace the hand would unintentionally. Therefore, the shoulder protraction-retraction
needed to be predicted to be used as a viable control scheme for hand opening and closing
across the entire workspace.
The prediction of shoulder protraction-retraction retraction from the remaining four degrees-of-
freedom (SABAD, SFE, SIER, SCDE) was sufficiently reliable. Thus, the protraction of the shoulder
can potentially be used as a source of commands for hand opening and closing. Furthermore,
59

removal of shoulder protraction-retraction from the input set did not severely affect the
prediction of elbow and forearm angles.
Achieving a high level of offline neural network performance is encouraging but these results do
not provide any information about the tractability of these neural networks in a prosthetic
system. In order to determine whether these predictive algorithms are actually stable and
useful as a basis for real-time control, we need to examine how these trained ANNs behave in
real-time virtual reality simulations analogous to real-world prostheses and FES systems
(Davoodi, et al. 2007). Furthermore, it remains to be seen whether forcing the subject into a
non-natural reaching strategy by fixing the wrist movements has limited the tractability of these
neural networks.
60

Chapter 4: Evaluation of non-invasive
command scheme for upper limb
prostheses in a virtual reality reach
and grasp task
Kaliki, R.R., Davoodi, R., and Loeb, G.E.
CHAPTER 4 ABSTRACT: We investigated how well able-bodied subjects can learn to use a non-
invasive command scheme based on inferences from the posture of the shoulder to control a
simulated transhumeral prosthesis in a virtual reality task. The task required reaching to and
grasping a bottle presented at various locations and orientations in extrapersonal space,
bringing it to the mouth and then releasing it on a work surface. The inferential command
scheme (ICS) consisted of three separate artificial neural networks (ANNs) and additional
algorithms for continuous control of elbow flexion/extension, forearm pronation/supination and
hand open and close. In order to examine whether the ANNs generalized across subjects, we
alternated the use of ANNs trained on the subject’s own data and ANNs trained with a novel
subject’s data. Furthermore, we compared the performance of subjects using the ICS with
subjects operating the simulated prosthesis in virtual reality according to complete motion
tracking of their actual arm and hand movements.
Initially subjects performed poorly with the ICS but were able to improve rapidly with modest
amounts of practice, eventually achieving performance only slightly less than subjects using
complete motion tracking. After ten sessions each lasting about 30 min, subjects using the ICS
were able to complete the task at a high percentage and with low spatial variability across the
workspace. It did not matter whether they used their own ANN or one trained on another
subject’s natural movements. Analysis of the various task phases and parameters revealed that
61

subjects had little or no difficult in the transport or orienting the hand to targets anywhere in
the workspace but had some problems in closing the hand around the target. This difficulty of
closing the hand was the primary contributor to the differences in performances between the
two experimental groups. Inferring the desired movement of distal joints from voluntary
shoulder movements appears to be a relatively simple, intuitive and non-invasive approach to
obtaining command signals for prostheses to restore distal arm and hand function.
62

Introduction
Current upper limb prosthetic options for quadriplegics and transhumeral amputees are very
poor. To restore meaningful functionality to these patients, a prosthesis must restore the
control of at least five degrees-of-freedom: the specification of the hand position in three
dimensions of extrapersonal space, the orientation of the hand, and the open and close
functions of the hand. While, current prostheses can provide required movement, they are
difficult to control. This is because there are more motors to control than there are command
signals to control them. Thus, the simple task of reaching and grasping that able-bodied humans
take for granted becomes a complicated sequence of switching between output DOFs and
making incremental adjustments until the task is complete. Many unilateral amputees abandon
their prosthetic limbs, citing difficulties in control and functionality (Biddiss and Chau 2007).
In fact, the various DOFs of the intact human arm and hand tend not to be used completely
independently. Reaching movements tend to have a high degree of stereotypical behavior
among joints and across the workspace. Several studies have shown that the movement of the
hand generally follows a straight line with a bell-shaped velocity curve irrespective of speed
during unimpeded reaching (Morasso, 1981, Soechting and Lacquaniti, 1981; Abend et al., 1982;
Hollerbach and Flash 1982; Morasso, 1983; Flash and Hogan 1985). These properties emerge
despite the number of redundant DOFs in the upper limb. Furthermore, when the initial
position of the hand is controlled, highly reproducible relationships exist between the proximal
and distal joint angles of the upper limb during reaching (Desmurget et al., 1995; Soechting and
Lacquaniti, 1981; Soechting et la., 1995). We and others have termed these relationships
“postural synergies.”
63

In the previous two chapters, we have laid a foundation for an intuitive and non-invasive
command scheme for upper limb prostheses. We showed that artificial neural networks can
predict some of the stereotyped postural synergies that emerge from the coordination of upper
limb joints during reaching. We showed that five DOFs at the shoulder joint can be used to
specify the required five output DOFs in offline analysis. Offline analysis merely shows whether
or not an exploitable relationship exists by the artificial neural networks. It does not, however,
give any indication as to how well a human can use the trained algorithm to control a prosthetic
device during typical tasks.
Our previous work has also shown that there are discrepancies between the predicted and the
measured angles. Thus, it is likely that ambiguities and errors will occur during use of such a
command scheme, but it is unclear how well a human subject can learn to compensate for these
shortcomings. Evidence from psychophysical experiments has shown that human subjects can
learn to produce straight hand paths despite initial errors resulting from kinematic or dynamic
perturbations (Lacquaniti, Soechting and Terzuolo, 1982; Hong et al., 1994; Shadmehr and
Mussa-Ivaldi, 1994). Thus, we hypothesized that given both a reasonable and a reliable
estimate of the desired outputs, subjects should learn to cope with errors by modifying their
shoulder movements to complete the reach and grasp tasks successfully.
Allowing an amputee or quadriplegic to test an experimental control scheme for a mechatronic
prosthesis could be potentially dangerous. Instead, a virtual reality environment (VRE) can be
used to test a prototype control scheme with a virtual prosthetic limb in real-time (Davoodi et
al., 2007). The experimental task should not be trivial and should adequately present the user
with a meaningful task for the activities of daily living for an amputee or quadriplegic. One of
the most important tasks of a prosthetic limb is to reach and grasp objects in three dimensional
64

space. In the special case of consuming a food or a drink, the user also needs to use the limb to
bring the item close to the mouth. Thus, we have designed a virtual reality task to mimic the
problem of reaching to and grasping a bottle and bringing it to the mouth.
The present study examines the ability of able-bodied subjects to use the inferential command
scheme (ICS) based on shoulder movements to perform a set of reach and grasp tasks in a
virtual reality environment. Specifically, we investigated whether or not subjects were able to
learn to use the ICS over a period of 10 sessions. Patients who already have amputations will be
unable to generate reaching movements with which to train an artificial neural network, so we
need to evaluate whether subjects can learn to use artificial neural networks trained on another
subject’s reaching data. Assuming that the neural networks give a reasonable estimate of
synergies that are similar in all subjects, we hypothesized that there would not be a significant
difference in performance when switching between networks trained with their own data and
networks trained with another subject’s data. In order to quantify performance with the ICS,
various metrics are used to compare learning rates and performance in a group of able-bodied
subjects using the ICS versus a group of subjects using complete motion capture from their
intact arm movements to control an animated arm in the same virtual reality task. Both groups
of subjects had to learn to use the virtual reality environment, so differences between them
should reflect performance limitations expected for an actual prosthetic system.
Inferential command scheme
As expressed earlier, any prosthetic device used by transhumeral amputees or C5/C6-level
quadriplegics is likely to require the specification of three outputs: endpoint position in three-
dimensions, hand orientation, and hand grasp. In the last chapter, we have shown that artificial
neural networks (ANNs) can be used to reliably predict the joint angles related to these outputs.
65

These ANNs provided a foundation for an inferential command scheme (ICS) for an upper limb
prosthesis, which infers the posture of the distal joints of the upper limb from the posture of the
shoulder.
In the case of transhumeral amputees and C5/C6 quadriplegics, endpoint position is determined
by predicting the elbow angle because the shoulder joint is assumed to be nearly completely
functional. In our command scheme, elbow angle is determined by two separate control
paradigms. The first method generates a prediction of the elbow angle from three rotational
angles and one translation movement of the shoulder using an ANN. The ANN was trained on
reaching tasks to locations that were 60-100% of the arm length from the body because the
correlations between shoulder and elbow posture were weak and inconsistent for targets close
to the body. Unsurprisingly, in preliminary experiments it was difficult for subjects to bring
objects to the face using only this ANN. The second method uses elevation of the shoulder for
direct, proportional control of flexion of the elbow, based on an observed tendency of subjects
to make such movements when the hand was close to the mouth.
Having two control paradigms controlling one joint could prove to be confusing and difficult to
use independently by subjects. Likewise, having two separate states, one using a neural
network and another using the elevation of the shoulder to control the same output, could yield
sudden changes in output when switching between the two control schemes. Therefore, a
smooth transition between the two schemes was essential. To fulfill this requirement, the two
control schemes determining elbow angle were regulated by variable gains (Fig. 13), which was
adjusted based on the distance of the hand from the center of the body. As the hand was
brought closer to the body, the subject’s shoulder elevation had a stronger influence in
66

determining the flexion of the elbow. Moving distally from the body, the prediction from the
ANN dominated the command signal.
Assuming no contribution from the proximal joints, hand orientation is determined by the
forearm pronation/supination (FPS) angle, wrist flexion/extension and wrist deviation. For
targets located in front of the body, FPS is used to align the hand with targets rotated in the
frontal plane. Wrist flexion/extension and wrist deviation are used to align the hand in
transverse or sagittal planes, depending on the rotation of the hand. Because the targets used
in this study were only rotated in the frontal plane, we assumed that the other wrist DOFs did
not have to be specified. Thus, in early versions of the ICS hand orientation was determined by
forearm pronation/supination, which was predicted from shoulder posture using an ANN.
Preliminary testing revealed the need to actuate the wrist. When subjects tried to use the
command scheme in a real-time virtual reality environment they had difficulty in matching the
appropriate angle of the target and had to make large corrective movements to grasp the target
properly. In some cases, they were unable to grasp the target at all. This was primarily due to
difficulty in making the hand parallel to the frontal plane. Therefore, the radial-ulnar deviation
of the wrist was coupled to the hand posture such that the axis for cylindrical grip (which passes
through the knuckles) was made to be parallel with the frontal plane. Wrist flexion/extension
remained fixed at neutral.
As discussed in the previous chapter, there was no apparent relationship between shoulder
posture and the onset of grasp. Instead, ipsilateral shoulder protraction was designated to
control proportionally the closing of the hand. We found that subjects would naturally protract
their shoulders when reaching to certain targets in space. This would result in an unreliable
control paradigm for grasp because the hand would unintentionally close when reaching to such
67

locations. Thus in order to use shoulder protraction as a reliable control for hand closing, this
movement needed to be deconvolved from the natural protraction-retraction movement
associated with reaching toward different positions in space. To this end, another ANN was
trained to predict natural protraction-retraction from the shoulder posture, which were
subtracted from the actual angles to obtain command signals for hand opening/closing.

S
FE
E
FE
S
IER
S
ABAD
F
PS
Grasp
S
DE
S
PR
E
FE
_
+
+
_
E
FE
F
PS
Grasp
S
ABAD
S
FE
S
IER
SC
DE
SC
PR
ANN1
ANN2
ANN3
RU
DEV
Wrist
Sensor
Dist.
Sensor
ICS
A B

Figure 13. Schematic of inferential command scheme (ICS). A. The recorded and controlled degrees of freedom.
Shoulder rotational angles (in blue) are shoulder abduction/adduction (S
ABAD
), shoulder flexion/extension (S
FE
), and
shoulder internal/external rotation (S
IER
). Shoulder translational movements (in red) are scapuloclavicular
depression/elevation (SC
DE
) and scapuloclavicular protraction/retraction (SC
PR
). Outputs of the ICS are elbow
flexion/extension (in pink - E
FE
), forearm pronation/supination (in green - F
PS
), grasp (in orange), and radio-ulnar
deviation (in purple - RC
DEV
). 1B. Schematic of ICS. Solid lines indicate values dependent on inferences from
shoulder posture. Dashed lines show values independently determined from the shoulder posture inferences.
Shoulder rotational angles and depression/elevation are used as inputs to the neural networks (ANN1, ANN2,
ANN3). EFE output is determined by a combination of outputs from ANN1 and the SC
DE
scheme described in the
text. The contribution from either ANN1 or SC
DE
is determined by a sensor measuring the distance from the hand
to the body. The closer the hand is to the body, the higher the gain on the output from SC
DE
scheme and the lower
the gain on the output from ANN1. The farther the hand from the body, the gain relationship is reversed. F
PS
is
determined by the output from ANN2. Grasp is determined by SC
PR
. In certain areas of reaching space, subjects
naturally protract their shoulders as a part of the reaching movement. Thus, the natural protraction needs to be
deconvolved from the grasping signal. ANN3 predicts the naturally occurring SC
PR
movement and is subtracted
from the value of SC
PR
recorded during real-time deployment. Radio-ulnar deviation (RU
DEV
) is passively actuated
by a simple rule of keeping the hand (through the axis passing through the knuckles of the hand) parallel with the
frontal plane. A sensor on the forearm, located before the RU
DEV
joint, determines the inclination of the forearm
with respect to the frontal plane and a simple geometric calculation is performed to determine the RU
DEV
angle.
68

Methods
Eight subjects participated in the experiments. All subjects were able-bodied, young adults
ranging from ages 21-26. The purpose of our experiment was to investigate how well subjects
can learn to use the ICS in a virtual reality reach and grasp task. As explained above, ANNs were
used in the ICS to predict distal posture during reaches. To train the ANNs, joint angle data was
recorded as able-bodied subjects performed reaches to a handle placed in random positions and
orientations within extrapersonal space. ANNs were trained and evaluated offline. Subjects
performed a virtual reality reach and grasp task using either the ICS or full motion capture.
Results were analyzed and compared using a various metrics. The experimental procedure is
described in greater detail below.
Motion tracking procedure
A Flock of Birds® (Ascension Technologies Corp., Burlington, VA) motion capture system was
used to record the joint angles of each subject’s right arm at 100 samples/s. Subjects had
sensors placed on the acromion, on the upper arm above the elbow, and proximal to wrist on
the forearm. Prior to experimentation, calibration was done to compensate for any
misalignments between the sensors and the segments they were tracking. Each sensor
measures position and orientation (as rotation matrices) with respect to the base transmitter.
Clinically meaningful Euler angles were derived from the rotation matrices, including
abduction/adduction (S
ABAD
), flexion/extension (S
FE
), and internal external rotation (S
IER
). The
other recorded angles were sternoclavicular depression/elevation (SC
DE
), sternoclavicular
protraction/retraction (SC
PR
), elbow flexion/extension (E
FE
) and radioulnar pronation/supination
(F
PS
). For a more detailed description of the calibration methodology and angle extraction
algorithm please refer to the work of Hauschild et al (2007).
69

Training data acquisition and processing
Training the ICS requires a systematic data set consisting of normal reach and grasp movements
made to targets at various positions and orientations throughout extrapersonal space. A large
robotic gantry (Parker Hannifin, Co.) was used to automate the presentation of targets within a
2m x 1m x 1m three-dimensional workspace. At the working end of the gantry arm, a cylindrical
handle was actuated by a stepper motor and rotated to one of four orientations (0°, 45°, 90°, or
135° in the frontal plane). Prior to experimentation target locations were scaled to the subject’s
physical measurements. Targets were located within a range of 60-100% of the measured arm
lengths away from the shoulder center of rotation. Target locations were limited to this range
because reaches to targets located proximally to this range required very little contribution from
the shoulder, resulting in poor predictability of the distal joint angles. For a more detailed
description of the methodology used to determine target location please refer to the previous
chapter.
Subjects were seated in a high back chair and secured with elastic bands around the torso to
limit movement of the trunk during experimentation. Once the subject was secured in the chair,
the subject’s right arm was instrumented and the motion tracking system was calibrated. Next,
a lap tray holding a push button switch was moved into place approximately five inches above
the subject’s lap height. The switch was placed directly in front of the subject, along the midline
of the body and subjects were told to keep their hands palm down on the switch in between
reaching trails. The subject was instructed to start the experimentation by pressing down on
the switch with the instrumented hand. Targets from the predetermined reaching space were
presented pseudo-randomly to the subject. The subject was told to move to each target at a
comfortable, self-determined pace and to grasp the handle for approximately one second. After
reaching and grasping the target the subject was instructed to move back to the initial position
70

and press the switch to cue the gantry to move to the next target. The experiment continued
until all target locations were reached. Each subject was asked to reach and grasp 400-500
targets, depending on the size of the subject’s reaching space. A given target appeared only
once in the data set in order to test the ability of the neural network to generate solutions that
would be continuously interpolatable.
The data were low-pass filtered offline to remove 4Hz noise present in the sensors. Data
recorded during the resting periods between target reaches were removed in order to limit the
contribution of the initial posture to the training of the neural networks. Data were partitioned
into three data sets. First, 20% of the data was set aside as novel validation data. Validation
was used after neural network training to quantify the ability of neural networks to generalize
for novel data. From the remaining data, 80% of the data points were randomly sampled and
set aside as the training data set. The training set was used during offline training of the neural
networks. The remaining data were set aside as a testing set. The testing set was used as novel
data with which to test the neural network during training.
Artificial neural network training
Three-layer perceptron ANNs were created in NeuralWorks Predict® (NeuralWare). This
software employed an adaptive gradient back-propagation algorithm to tune the weights and
biases of the ANN to maximize the correlation between the model predictions and the recorded
data. To improve the ANNs ability to generalize and to prevent overfitting, the program
employed a method of “early stopping,” which stopped training of the neural network if the
neural network’s performance on novel, test data no longer improved. The hidden layer
contained units with hyperbolic tangent activation functions. Hidden layer size was
automatically determined through a cascade learning algorithm.
71

In the previous chapter, we found that the three rotational angles (S
FE
, S
IER
, and S
ABAD
) and
shoulder depression elevation (SC
DE
) were sufficient to predict the elbow flexion/extension
(EFE), forearm pronation/supination (FPS), and shoulder protraction/retraction (SCPR). In order
to verify that the input degrees-of-freedom contained at least three independent dimensions to
predict the three outputs, we performed a principle component analysis (PCA) on the four
inputs. The PCA revealed that most of the variance in the data could be mapped onto three
principle components; hence, the input space contained three significant dimensions.
The complete inferential command engine required the training of three separate neural
networks corresponding to the required degrees-of-freedom (Fig. 13):
ANN1 predicted elbow angle (E
FE
) from the three rotational joint angles (S
FE
, S
IER
, and S
ABAD
) and
one translational degree-of-freedom (SC
DE
) at the shoulder.
ANN2 predicted F
PS
from the same inputs as ANN1.
ANN3 used the same inputs as ANN1 and ANN2 to predict the SC
PR
angle.
The coefficient of determination (R
2
) between the predicted output and recorded output was
measured for all the ANNs. Any R
2
value above 0.5 was considered a strong correlation.
Because amputees and tetraplegics are unable to generate reaching movements to train ANNs,
it is important to evaluate whether subjects can learn to use a set of neural networks trained
with another person’s reaching data. To assess this problem, subjects using the ICS alternated
between the use of ANNs trained on their own reaching data (intra-subject network) and ANNs
trained with another subject’s data (inter-subject networks). To determine which inter-subject
ANN to use, each subject’s set of ANNs was tested on all the validation data from the other
subjects. The set of ANNs that predicted a given subject’s data the worst during offline analysis
72

was assumed to represent the reaching strategies most foreign to that subject and, therefore,
was selected as the inter-subject networks for that subject.
Virtual reality testing
A simple virtual reality task was developed in MSMS to evaluate how well subjects could learn to
use the inferential command scheme. As a simplification, the VRE did not include dynamics or
collision detection and instead just represented the kinematics of the movements generated.
The task mimicked reaching and grasping a food item, bringing the item near the mouth to
consume it, and replacing it on a table (Fig. 14). The VRE task was broken down into three
distinct phases: phase 1, the subject reaches and grasps a bottle-shaped object in 3D virtual
space; phase 2, the subject brings the bottle to the mouth and holds the bottle in that position
for 3 seconds; and phase 3, the subjects releases the bottle on a table at a designated location.
The eight subjects were split evenly into two groups: Group A, in which subjects used the ICS,
and Group B, in which subjects used motion capture only. In order to compare the performance
of subjects doing a novel task we did not crossover subjects between the groups.
Subjects were seated and restrained as in the previous gantry-based reach and grasp task.
Sensors were placed on the right arm and calibrated. Additional sensors were placed on the
wrist and index fingers of subjects in group B to measure the rotational angles of the wrist and
opening and closing of the hand, respectively. Subjects donned stereoscopic virtual reality
goggles (NVIS, Inc.) equipped with a 3-axis accelerometer (Microstrain, Inc) to provide head
tracking. The stereoscopic goggles provided subjects with a first-person view within the 3D
virtual reality environment. Anthropomorphic virtual human models were scaled to replicate
the dimensions of the subject. Angles recorded from the motion capture sensors and/or
predicted from the ICS directly actuated the joints of the upper limb of the virtual human model.
73

Figure 14. The virtual reality reach and grasp task. 2A and 2B show the first phase of the task. In 2A, the bottle is in
a random position and orientation and the subject is to reach to the bottle. As shown in 2B, the bottle is
highlighted red when the subject has met the appropriate grasp conditions. Once grasped, the coaster on the table
is moved to a position near the chin of the virtual human, as seen in 2C. After the subject holds the bottle near the
chin for three seconds the coaster moves to a position on top of the table. To successfully finish the task, the
subject must release the bottle on top of the coaster as shown in 2D.
The bottle position and orientation from trial to trial was random. In order to grasp the bottle,
the subject was required approach the bottle with an open hand, place the hand within 6 cm of
the bottle with velocity of the hand below 1 cm/s, and match the orientation of the hand to the
orientation of the bottle to within 30°. Once the grasp conditions were met, the bottle changed
color indicating that the bottle could be grasped. Once the bottle was grasped, a square-shaped
target (coaster) moved into place near the mouth, showing the subject where to move the
bottle. When the bottle was within 5 cm of the center of the coaster, the coaster was
highlighted much like the bottle in the previous phase, indicating that bottle was in the correct
74

place. After three seconds, the coaster moved to a position on the table in front of the subject.
The subject was instructed to reach to the target again, but this time, once in the proper
location, the subject was told to release the bottle on top of the coaster. If the subject
succeeded in doing so, the bottle was moved to another random position and orientation within
the subject’s reaching space.
Subjects had 30 seconds in each phase of the task to complete that phase. If the subject did not
complete a given phase in 30 seconds, the bottle position was reset to the next random starting
position and orientation in the queue. The experiment was continued until the subject
completed the task for 30 trials. Several variables were recorded during each session, including
distance between hand and bottle, position of the bottle, time, phase of task, and trial counter.
The VRE experiment was conducted for both Group A and Group B subjects over ten sessions.
Subjects in Group A began using the intra-subject networks in session 1 and inter-subject
networks were used every other session after that. A minimum of 24 hours were required
between each session.
Following ten sessions of training, we analyzed the motion capture data of Group A subjects to
identify the dimensionality of the input space used to drive the ICS by using PCA. The motion
capture data set or “execution set” recorded from these subjects included only the shoulder
rotational and translational movements. If the dimensionality had significantly changed from
the ANN training data this would indicate that the subjects were using different reaching
strategies to move the virtual limb. To conduct a fair comparison of dimensionality between the
training and execution sets, we only analyzed the data recorded during reaching phase of the VR
task. We found that there was not a significant difference in the dimensionality between the
75

training and execution sets, indicating that subjects performed the reaching portion of the VR
task similarly to natural reaching.
Identifying a comparative performance metric
To compare performances between Group A and Group B subjects in a real-time setting, a
unifying metric that expresses performance quantitatively is needed. An obvious metric is time
taken to complete a trial within a session, but time may be dependent on the distance between
the hand and bottle at the onset of the session. Thus, a better expression could be some
function of distance and time, such as rate. To identify if a relationship exists, we needed to
examine first whether there indeed was a correlation between time and distance. An analysis of
distance and time is not appropriate for first phase of the task because the time to reach the
bottle was not measured separately from the time to grasp the bottle. An analysis of phase 3 is
likewise unsuitable because the distance in phase 3 was relatively constant. Instead, a
comparison between time and distance is only appropriate in the second phase of the task,
because this phase is essentially a point-to-point movement. This phase begins immediately
after the bottle is grasped and continues until it is carried to the coaster. Thus, the distance is
measured as the three-dimensional distance between the center of the bottle and the center of
the coaster at the onset of the phase. Scatter plots of distance vs. time are shown in Fig. 15 for
phase 2 of the task after the tenth trial for one subject. The scatter plots were highly variable
across all sessions for a given subject and across subjects. As seen in the scatter plots, a few
data points were distinct outliers, therefore the Spearman rank correlation coefficient was used
to calculate the correlations.
76

10 12 14 16 18 20 22 24 26 28 30
0
20
40
60
80
100
120
140
160
Distance from Hand (inches)
Time (s)
Distance vs. Time (Phase 2)

Figure 15. Scatter plot of Distance vs. Time for the second phase of VRE experiments in the tenth session of subject
6. Scatter plots for a given subject varied greatly from session to session as indicated by the highly variable
correlation between distance and time.
Data from all eight subjects showed poor correlations between distance and time for phase 2 of
the experiment. It is likely that distance and time are indeed correlated as Fitts showed in 1954
but the correlation is weak due to the variances in performance over trials. Thus, mean time
was chosen as the comparative performance metric. We believe mean time is an appropriate
metric because subjects were moving the hand with high velocities and low variations in
performance (as shown below) by the tenth trial and so a difference of a few inches in the
distance to the target does not yield a significantly different time to reach the target.
Results
Eight subjects were split evenly into Group A and Group B. Subjects 1-4 were placed in Group A
and subjects 5-8 were placed in Group B. Group A had mean lower arm lengths of 10.25” (SD =
0.28”), upper arm lengths of 11.875” (SD = 0.95”), and clavicle lengths of 6.87” (SD = 0.25”).
Group B had a mean lower arm lengths of 10.50” (SD = 0.41”), upper arm lengths of 12.38” (SD =
1.25”), and clavicle lengths of 7.5” (SD = 0.58”).
77

Offline artificial neural network performance
Three artificial neural networks were trained for each of the four subjects in Group A. Each
trained network’s offline performance on the respective validation set is shown in Table 6. Every
intra-subject artificial neural network achieved R
2
values greater than 0.5 on the validation set.
Also shown in Table 6 is the performance of the inter-subject neural network for each subject.
As explained above, the inter-subject network was selected based on how poorly it performed
on a given subject’s data. Thus, the offline performance was necessarily significantly worse than
the intra-subject networks.
Subject Network Type E
FE
F
PS
SC
PR
Intra-subject 0.80 0.55 0.67
Inter-subject 0.45 0.30 0.12
Intra-subject 0.67 0.75 0.74
Inter-subject 0.01 0.32 0.04
Intra-subject 0.77 0.65 0.76
Inter-subject 0.25 0.09 0.04
Intra-subject 0.61 0.75 0.80
Inter-subject 0.06 0.22 0.17
3
4
1
2

Table 6. Offline analysis of both intra- and inter-subject artificial neural networks. The coefficient of determination
(R
2
) is given for both intra- and inter-subject neural networks on a given subject’s validation sets.
Comparisons in trial mean times between groups
Because time to perform each phase was not dependent on distance to target, it was used to
compare trials. Time to complete each phase was recorded for each attempt over a period of
ten sessions. If the subject was unable to complete a trial, the trial completion time data was
discarded. Completion percentage of trials over each of the ten sessions is shown in Fig 16. The
figure contains plots of Group A subjects only because all subjects in Group B maintained a 100%
completion percentage over the ten sessions. In the first session, subjects in Group A completed
the session on average with a completion percentage of 70% (SD = 20%). The same subjects
were able to complete on average 98% (SD = 3%) of the trial attempts on the last session,
indicating progressive improvement on the task.
78

To investigate why subjects were missing targets we searched for similarities across missed
targets over the last five sessions (sessions 6-10). Specifically we looked for trends based on
target orientation and target position. In this analysis we did not include data from subject 3
because this subject maintained a 100% completion percentage over the last five sessions. We
found a slight trend favoring the horizontal orientation of the target (90°, which could be
reached either by extreme pronation or extreme supination); subjects were not able to grasp
this target 15% of the attempts whereas subjects were not able to grasp the targets 4% and 1%
of the attempts when the targets were oriented 45° and 135°, respectively. Subjects were able
to grasp vertically oriented (0°) during every attempt. We also found a trend based on the
position of the target. Target locations were placed in bins of equal width with respect to the
lateral distance (along the z-axis) from the shoulder center of rotation. Because the
extrapersonal space of a given subject was scaled to their limb dimension, the widths of the bins
varied from subject to subject. There were a total of six bins: two bins were located to the left
of the shoulder center of rotation and four bins were located to the right of the shoulder center
of rotation. We found a trend showing that subjects had difficulty with targets located at the
edges of the workspace. Specifically, subjects were not able to grasp 17% of targets located in
the left-most bin and 11% of targets in the right-most bin. Missed percentage of targets were
no greater than 4% (SD = 1%) in the other bins.
79

0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
1 2 3 4 5 6 7 8 9 10
C
o
m
p
l
e
t
i
o
n
P
e
r
c
e
n
t
a
g
e
Session
Group A Completion Percentage over Sessions
Subject 1
Subject 2
Subject 3
Subject 4

Figure 16. Plot of completion percentages of trials over sessions for Group A subjects. During the first session, none
of the subjects in Group A completed more than 90% of the trials. In the first session, subject 1 performed the
worst with 39% completion rate, while subject 4 performed the best with 89% completion rate. It is also important
to note that subjects switched between inter- and intra- subject networks every other session, but there is no
apparent trend in large session to session completion percentage differences after the fifth session. All subjects
converge to a completion percentage greater than 90% after the fifth session and achieve 100% completion
percentage at least once over the ten sessions.
The mean time to complete a trial was compiled from the successfully completed trials for each
of the subject’s ten sessions. The three second holding period during phase 2 was subtracted
from the total trial time. The mean time over the ten sessions for all eight subjects is shown in
Fig. 17. During the first session, subjects in Group A finished trials with a mean time of 39.52s
(SD = 4.67s), while subjects in Group B achieved a mean of 15.40s (SD = 3.33s). Mean time
decreased over time for all subjects in an exponentially decaying fashion. Group A subjects
showed no apparent tradeoff when switching between intra- and inter- subject networks after
the sixth session. In the tenth and last session, subjects in Group A were able to finish the trials
with a mean time of 9.49s (SD = 1.32s), while subjects in Group A finished the trials with a mean
time of 5.76s (SD = 1.65s).
80

0.00
5.00
10.00
15.00
20.00
25.00
30.00
35.00
40.00
45.00
50.00
1 2 3 4 5 6 7 8 9 10
Mean Time (seconds)
Session
Completed Trial Mean Time over Sessions
Subject 1
Subject 2
Subject 3
Subject 4
Subject 5
Subject 6
Subject 7
Subject 8

Figure 17. Completed trial mean times over sessions. Mean completion times of subjects in Group A are
highlighted in blue tones whereas the mean times for subjects in Group B are highlighted in red tones. In session 1,
subjects in Group A finished trials with a mean time of 39.52s (SD = 4.67s), while subjects in Group B finished trials
with a mean time of 15.40s (SD = 3.33s). Group A subjects showed very little changes in performance from session
to session after the sixth session, indicating that subjects learned to use both inter- and intra-subject networks.
After ten sessions, subjects in Group A finished trials with a mean time of 9.49s (SD = 1.32s), while subjects in
Group B had a mean time of 5.76s (SD = 1.65s).
Analysis of spatial variability in performance
While mean times and completion percentage do provide some insights into the quality of the
inferential command scheme, the analysis is not complete. It was possible that subjects could
complete the task very easily for certain positions and orientations but not others. Thus, we
examined the mean times across various areas of the workspace over each of the ten sessions.
For each session, trial completion times were binned based on the target’s (bottle’s) position in
space at the onset of a trial. Additionally, the mean time and standard deviation were recorded
for each session. In Fig. 18, exemplar mean times are shown as a range of colors for bottle
positions in various regions of the XZ plane (corresponding to top-down view) over ten sessions.
81

Initially, the subject had difficulty across the entire workspace. Targets located far from the
shoulder center of rotation resulted in the largest trial completion times. Over sessions, the
subject was able to improve performance across the entire workspace. By the seventh session,
the subject maintained mean trial completion times below 20s across the entire workspace.
We also investigated the effect of target orientation on mean time but found no trend across all
Group A subjects (all mean times for a given orientation were within one standard deviation of
other mean times).
82

Figure 18. Color-scaled plots of Subject 3’s mean times over blocks of the subject’s workspace over ten sessions.
Target position is given with respect to the shoulder center of rotation (0,0). The XZ plots provide a top down view
of the workspace. Darker colors correspond to faster mean times; while brighter correspond to slower times. If a
particular block is colored black, it indicates that a target was not presented in that region of workspace to that
subject during the session. Session numbers are in bold at the lower left corner of the plots. Initially, the subject
performed poorly during session 1 with several areas of mean times over 50s. With practice, the mean times
across the workspace were reduced. By the seventh session, the subject achieved a consistent low mean rate
across all the blocks of the workspace.
83

Identifying sources of differences in performance between groups
Clearly, there is a difference in mean times between Group A and Group B subjects. To get a
better understanding of where the difficulties arise for Group A subjects, we compared the
differences between the two groups in mean time spent in each of the three phases of the
virtual reality experiment. Fig. 19 shows a stacked bar graph of the mean time achieved during
the tenth trial in the three phases of the task for all eight subjects.

Group A

Figure 19. Break down in mean time to complete task by phase for all subjects on the tenth trial. Subjects in Group
A are indicated by the brackets. Within groups the variations of time spent in phase is very minimal. The clear
differences between groups is the time spent in phase 1. Time spent in phase 2 and phase 3 is very similar across
all subjects. These finding was statistically confirmed using a paired t-test.
From Fig. 19, it is clear that the largest differences in mean times between Groups A and B
occurs during phase 1 (2.36s). A two-tailed paired t-test analysis showed that indeed the
groups’ performances were significantly different in phase 1 (P < 0.05). For phases 2, the mean
times were 1.82s (SD = 0.74) and 1.10s (SD = 0.4) for group A and B, respectively and not
significantly different (P > 0.05). For phase 3, the means were 2.59s (SD = 0.32) and 2.03s (SD
=0.90) for group A and B, respectively and likewise were not significantly different.
84

Discussion
In this study we examined the use of a non-invasive inferential command scheme during a
simple virtual reality reach and grasp task. The task that was considered was a purely kinematic
task which required subjects to reach to and grasp a bottle positioned and oriented randomly
within extrapersonal space. Once the bottle was grasped subjects had to bring the bottle to the
mouth, mimicking the tasking of consuming a food item. We chose this task because it is an
essential activity of daily living for bilateral amputees and quadriplegics, who do not have the
use of their limb. The objective of the study was two-fold: 1) to assess the ability of able-bodied
subjects to use the ICS system trained on inter- and intra-subject trained artificial neural
networks and 2) to compare and quantify the performances of subjects using the ICS versus
subjects using complete motion tracking to complete the task.
Generalization between ANNs
We hypothesized that irrespective of which set of ANNs (intra- or inter-subject ANNs) was used
subjects would be able to complete the virtual reality task. Offline analysis showed that the
predictive abilities of ANNs did not generalize well to data from other subjects, while intra-
subject ANNs performed well for all subjects. The discrepancy in offline performances between
the inter- and intra-subject neural networks could be the result of two issues.
First, it is possible that there was variability across subjects in the postural synergies employed
by subjects. Several studies have shown that the invariant features of reaching such as straight
hand trajectories and bell-shaped curves of the tangential hand velocity versus time are
preserved across subjects; therefore, it is not likely that the reaching strategies varied greatly
between subjects, (Morasso, 1981; Soechting and Lacquaniti, 1981; Abend et al. 1982;
Hollerbach and Flash, 1982; Flash and Hogan, 1985). Rather we believe that variability in the
85

arm lengths and the ratios between the proximal and distal segments of the arm required
subjects to use different patterns of coordination between the joint angles to reach targets
presented by the gantry. Furthermore it has been shown that postural synergies are highly
dependent on the initial posture of the arm (Soechting et al. 1995). While we did instruct
subjects to keep their hands palm down at a specified initial position, we did not place strict
controls on the initial posture of the limb. Thus, it is likely that subjects had different initial
postures resulting in different postural synergies for the same targets in space during the
gantry-based reach and grasp task. Future studies should ensure that the subjects’ initial
posture is similar across all subjects.
It is also possible that the ANNs were overfitted for an individual subject’s data, meaning the
ANNs did not generalize well for other subjects. Because data from only one subject was used
to train a given neural network, ANNs were never exposed to data from other subjects and
therefore performed poorly during offline analysis. In preliminary work, we did train ANNs with
data from several subjects, but the offline performances of the ANNS were poor and were not
able to predict data from any one individual well. Thus, we chose to train neural networks only
on one subject’s data under the assumption that the output of the neural network was
reasonably close to the expected output. We assumed that subjects could learn to cope with
the errors as long as the errors were not incomprehensible and were predictable. We
considered the situation analogous to an experiment by Lacquaniti et al, in which subjects were
asked to reach between an initial position and target position after their forearms had been
artificially doubled in length by attaching a pole to their forearms (Lacquaniti et al., 1982). They
found that despite the change in the internal constraints of the limb, subjects immediately were
able to adjust their shoulder and elbow posture to maintain a straight end-point path and a bell-
shaped endpoint velocity curve. Subjects were able to do this so quickly because they were able
86

to anticipate the errors caused by the length of the forearm and make adjustments in the other
joints to minimize those errors.
Likewise, we found that subjects could indeed learn to cope with errors to control the arm
successfully in virtual reality environment. After the sixth session there was no appreciable
difference when subjects switched between intra- and inter-subject ANNs both in terms of
completion percentage and mean trial completion time. By the sixth testing session, all
subjects using the ICS were able to complete the reach and grasp task over 90% of the time. In
fact, all four subjects were able to complete all 30 trials during at least one training session. One
subject was able to achieve a 100% completion percentage during every session after just the
third trial. We also found that subjects had difficulty with certain targets based on their location
in space and the orientation of the target. Specifically, we found that subjects had difficulties
reaching to targets that were horizontal and were located at the edges of the reaching space.
This is understandable but most likely would not be a problem in a real world scenario because
users could rotate their torsos or wheelchairs to bring the targeted object to a location that is
easier to reach. It is important to note that while subjects did have some difficulties during
earlier session that by the final testing session all four subjects completed the tasks with a 98%
(SD = 3%) mean completion percentage.
Speed and accuracy
While completion percentage does tell us that subjects can use the ICS to reach and grasp
objects, it does not give any indication as to how well the subjects performed during the task. In
order to comment on the quality of the subjects’ performances using the ICS (Group A) we
needed to compare it against performances of subjects using only motion capture (Group B).
Mean time was considerably higher during the first session in Group A subjects (mean = 39.52s,
87

SD = 4.67s) as compared to Group B subjects (mean = 15.40s, SD = 3.33s). This indicates that
Group A subjects initially had difficult using the ICS to complete the task even though it was
based on their own ANN. In both groups, we found that mean time per trial decreased
exponentially with practice over the ten experimental sessions. This finding is a typical
characteristic of learning (Hovland 1951). By the final session, Group A subjects greatly
improved their performances by completing trials with a mean time of 9.49s (SD = 1.32s), which
is a 76% improvement over the first session. Group B subjects also improved their
performances by the final session and achieved a mean trial time of 5.76s (SD = 1.65s), which is
a 63% improvement. Overall, Group B subjects performed 39% faster on average per trial than
Group A subjects in the final session.
Despite the low mean times and the exponentially decreasing learning curves, it is possible that
Group A subjects were performing well only in certain parts of the workspace while performing
worse in others. This information is lost when mean times are averaged over the entire
workspace. Thus, we qualitatively investigated the spatial variability of performance by
examining color-scaled plots of mean times over parts of the subjects’ workspace. Initially, each
of the Group A subjects performed poorly across the entire workspace. By the sixth session, the
spatial variability had greatly decreased in each of the Group A subjects. Mean times across the
entire workspace were low and consistent during the last session. Furthermore, we also could
not find an effect on mean time based on target orientation.
Lastly, we examined the breakdown of mean time by the phase of the trial to provide insights
into which components of the ICS might be difficult to use. We found no statistically significant
differences in mean times spent in phase 2 of the task between the two groups. This phase
involved the transport of the hand to the mouth after the target was grasped. Because this
88

phase involved primarily flexion of the elbow, this indicates that subjects had little or no
difficulty switching between the two control paradigms specifying elbow angle. We also found
no statistically significant differences between groups in mean times spent in phase 3. This
phase involved transporting the bottle from the mouth to the table and releasing the bottle on
top of the table. This indicates that subjects did not have significant difficulty opening the hand
at the appropriate time. Furthermore, because there were not significant differences in
performance between groups in phases 2 and 3 and there was little spatial variability in mean
trial times or variability in mean trial times due to target orientation across the entire
workspace, we can reasonably assume that subjects had no problems with transporting or
orientating the hand.
We did find significant differences between groups in mean times spent in phase 1. Phase 1
involved transporting, orienting, and closing the hand. We have provided evidence suggesting
that subjects had little or no difficulty transporting and orienting the hand; therefore, we believe
the significant difference in performance between Group A and Group B subjects is the result of
difficulty in closing the hand. Both through experimental observation and feedback from
subjects, it was clear that the most difficult part of the experiment was grasping the bottle. The
difficulty could be attributed to the command scheme employed for grasping. When subjects
would protract their shoulders forward to close the hand around the bottle, the hand would
move forward because of the mechanical linkages between the joints, causing them to miss
targets. In humans while raising the arms forward, we can protract our shoulders forward while
keeping the hand in a relatively stable position. Despite the mechanical linkages, it is possible
because the forward movement of the shoulder joint is compensated by a decrease in the elbow
angle. Thus, the hand is kept in the same position.
89

Potential improvements
There are some potential solutions for improving grasping in the ICS. One method is to include a
software solution to compensate for shoulder protraction movement in order to keep the hand
in a stable position by adjusting the joints in the arm. This solution is not trivial because the
compensation by the joints of the arm varies as a function of the arm posture. Another solution
could be to use EMG to proportionally close the hand, similar to the solutions currently available
for upper limb prostheses. EMG sensors could be placed on the biceps and triceps or another
pair of agonist-antagonist muscles, preferably not involved in reaching.
We observed some minor corrective movements once the hand reached the vicinity of the
bottle but this was present in both groups of subjects. In Group A subjects, these correctives
were hard to decouple from the retraction/protraction movements used to open and close the
hand. It is likely that most of these correctives are the result of a) loop delays between the
action of the subject and the display of the movement and/or b) an elective strategy of making
quasiballistic movements to be followed by corrective movements as needed based on visual
perception. It is important to note that subjects doing these VR tasks attempted to complete the
task as fast as possible to get the "fastest time," and there is a well observed tradeoff between
speed and accuracy. In a more realistic VR task (with realistic limb dynamics including collision
detection and kinetic interactions), the required accuracy of the commands might be higher or
lower, depending on the intertial properties of the object and the compliance programmed into
the mechatronic limb controller (see below).
The loop delays in our experimental system were mostly due to the heavy low-pass filtering (3rd
order Butterworth with 3Hz cutoff) implemented to reduce 4Hz noise in the sensors. The noise
was related to the Flock of Birds motion capture system and could not be removed at the
90

source. The delay from this filter was approximately 220ms. The other possible sources of
delays are the motion tracking (10ms), the calculation of predicted outputs from ANNs (< 10ms),
the rendering time for virtual reality environment (15 ms) and the refresh rate of the VR goggles
(17 ms), but these delays are insignificant compared to the filter delay. It should be possible to
design a clinical position sensing system that avoids this noise and the resulting delays. In the
discussion, we describe one such technology that currently has a lower sampling frequency
(10Hz) but that would actually result in less delay than the filters employed in these
experiments.
One advantage of using shoulder posture as opposed to myoelectric command signals is that
patients will generally have much higher precision (i.e. more distinguishable states) as a result of
kinesthetic feedback from muscle spindles and other proprioceptors. This could provide the
basis for the corrective movements that were frequently observed. Thus it is important to
understand how the non-linear ANNs might transform small changes in their inputs into large
changes in their outputs that might limit achievable accuracy. In chapter 2, we observed that
phase relationships between shoulder rotational DOFs and the elbow angle varied greatly over a
2D horizontal workspace. At the edges of the workspace the phase relationships were more
non-linear whereas phase relationships observed during reaches to the medial parts of the
workspace tended to be more linear. Thus, when reaching to positions in the center of
extrapersonal space we expected that changes in the input DOFs would results in similar
changes in the output of the ANNs. When the endpoint is toward the edges of the workspace
we expected that small changes in the input DOFs could either result in larger changes or
smaller changes in the output of the ANNs.
91

In order to verify this hypothesis, we conducted a sensitivity analysis on the ANNs at various arm
postures and endpoint positions within extrapersonal space. An anthropomorphic model of the
human arm was created in Simulink. The arm model included the scapuloclavicular,
glenohumeral, elbow, and forearm pronation/supination joints. The arm was scaled to match
the dimensions of human subject in the above study. ANNs used in the ICS were used to drive
the elbow and forearm joints. The input DOFs were specified according to the values recorded
during the gantry-based reach and grasp task corresponding to a certain endpoint position.
Each of the four input DOFs were varied by ± 10° at increments of 1° and changes in the
endpoint position were measured and plotted (Fig. 20). The sensitivity of the ANNs were
observed as endpoint positions changed in azimuth (Figs. 20A and 20B), elevation (20C and
20D), and radial distance (8E and 8F).
From Figs. 20A and 20B it is clear that our hypothesis was correct and that the ANNs are highly
sensitive to most of the input DOFs at the edge of the workspace. As the endpoint moved
medially, the sensitivity to changes in the rotational shoulder angles decreased and the changes
in endpoint position became linear. Changes in elevation of the endpoint position didn’t result
in a great change in sensitivity of the ANNs. The linearity and orientation of the changes in
endpoint positions did vary but the magnitude of the changes did not. As the endpoint position
was moved proximally (Figs. 20E and 20F), the sensitivity to shoulder internal/external rotation
(S
IER
) increased whereas the sensitivity to shoulder flexion/extension (S
ABAD
) and
scapuloclavicular depression/elevation (SC
DE
) decreased. Across most of the postures the ANNs
were most sensitive to SC
DE
and S
FE
, while changes in S
IER
and shoulder abduction/adduction
(S
ABAD
) did not result in large changes in endpoint position.
92

These results indicate that subjects using the ICS can likely make precise corrections near the
proximal and medial parts of the workspace. The precision decreases as the subject moves
distally and laterally, but it is likely that subjects will infrequently move to these parts of the
workspace. Additionally, small changes in the SC
DE
DOF result in large changes in endpoint
position but subjects rarely make large SC
DE
movements. Furthermore, when using the ICS,
subjects are only inclined to make large SCDE movements when they wish to initiate large
movements of the hand to the mouth. While these results indicate that subjects may be able to
make precise corrective movements in certain parts of the workspace, it remains to be seen
how precisely the ICS can be used for simple real-world tasks such as grasping and transporting
a glass of water without spilling its contents.
93

0
0.2
0.4
0.6
0.8 -0.4
-0.3
-0.2
-0.1
0
-0.4
-0.35
-0.3
-0.25
-0.2
-0.15
-0.1
-0.05
0
z position(m)
x position(m)
y position(m)
0
0.2
0.4
0.6
0.8
-0.4
-0.3
-0.2
-0.1
0
-0.4
-0.35
-0.3
-0.25
-0.2
-0.15
-0.1
-0.05
0
x position(m)
z position(m)
y position(m)
SC
DE
S
ABAD
S
FE
S
IER
0
0.2
0.4
0.6
0.8
-0.5
-0.4
-0.3
-0.2
-0.1
0
-0.4
-0.3
-0.2
-0.1
0
0.1
x position(m)
z position(m)
y position(m)
0
0.2
0.4
0.6
0.8 -0.5
-0.4
-0.3
-0.2
-0.1
0
-0.4
-0.3
-0.2
-0.1
0
0.1
z position(m)
x position(m)
y position(m)
SC
DE
S
ABAD
S
FE
S
IER
0
0.2
0.4
0.6
0.8 -0.6
-0.4
-0.2
0
0.2
-0.4
-0.3
-0.2
-0.1
0
0.1
0.2
0.3
z position(m)
x position(m)
y position(m)
0
0.2
0.4
0.6
0.8
-0.6
-0.4
-0.2
0
0.2
-0.4
-0.3
-0.2
-0.1
0
0.1
0.2
0.3
x position(m)
z position(m)
y position(m)
SC
DE
S
ABAD
S
FE
S
IER
A B
C D
E F

Figure 20. The results of sensitivity analyses of the ICS in different positions in extrapersonal space. The changes in
endpoint position with respect to changes in SC
DE
(shown in red), S
ABAD
(black), S
FE
(green) and S
IER
(blue) inputs.
The solid blue lines indicate the upper and lower segments of the arm model. 8A and 8B show two different views
(rotated 90° about the y-axis) of end point positions varying in azimuth. 8C and 8D show two views of arm
postures and endpoint positions varying in elevation. 8E and 8F show views of proximal and distal endpoint
positions. The distal position in 8E and 8F corresponds to the third position from the left in 8A and 8B and the
lower position in 8C and 8D.
94

It is important to emphasize that the task presented here did not include dynamics of the
prosthetic limb, dynamics of the environment, or collision detection. In the work presented
here, subjects could make the virtual arm pass through the bottle, coaster, and table without
consequence. Without realistically modeled dynamics of the limb and environment, it is unclear
whether or not subjects can internalize the dynamics of the prosthetic limb. Studies have
shown that subjects are able compensate when the arm’s dynamics are changed to make point
to point movements (Lackner and Dizio, 1994; Shadmehr and Mussa-Ivaldi, 1994). Evidence
from these studies suggests that the internal representation or internal model of the limb is
changed to adjust for the change in dynamics. The adjustments of the internal model can likely
occur only if the changes in the dynamics are predictable; therefore, it is likely that subjects will
also be able to internalize the prosthetic limb dynamics if the behavior of the limb is reliable.
Furthermore, it is likely that grasping will be considerably more difficult in the absence of tactile
feedback. Thus, it is important to consider implementing localized feedback to adjust grip and
to hold objects stable. The inclusion of these features is essential in assessing the real-world
practicality of such a control strategy.
95

Chapter 5: General Discussion
In this thesis we have shown that voluntary shoulder motion is a promising source of command
signals to restore reaching and grasping abilities to transhumeral amputees and C5/C6
quadriplegics. The command scheme presented here is based on the natural patterns of able-
bodied human reaching. Specifically, these patterns, which we term “postural synergies,” are
essentially stereotyped spatiotemporal relationships among the joints of the upper limb that are
the product of motor planning and execution of normal reaching movements. We employed a
machine learning technique, specifically artificial neural networks (ANNs), to extract and
embody these synergies. In the initial experiments, the ANNs were trained with joint angle data
measured during reaches to targets located on a 2D horizontal plane. In subsequent research,
the method was generalized to targets in 3D extrapersonal space at various orientations. The
ANNs were able to achieve reasonable offline accuracy when predicting elbow
flexion/extension, forearm pronation/supination, and shoulder protraction/retraction to control
grasp from the three rotational angles and the depression/elevation degree-of-freedom of the
shoulder. In addition, we found that subjects were able to use an inferential command scheme
(ICS) based on the ANNs to move an upper limb in a virtual reality environment to reach, grasp
and transport a bottle. The ability to do this task using the ICS compared favorably with subjects
using full motion capture to move the simulated limb in the virtual reality environment.
Despite the generally good performance, subjects demonstrated some difficulty in keeping the
hand steady during closing of the hand. The ICS required subjects to protract their shoulders
forward to close the hand. When subjects attempted to protract the shoulder to close the hand,
the hand would move forward causing them to occasionally miss the bottle. The unintended
movement of the hand was the result of the mechanical linkages between the shoulder joint
96

and the hand, thus when the shoulder joint moved forward the whole arm including the hand
moved forward. It is likely that movement of the hand can be corrected by adjusting the joints
in the limb to keep the hand steady when the shoulder is protracted. An alternative command
source for hand opening and c losing could be EMG from a pair of agonist-antagonist muscles.
ANNs have been used over 60 years for similar problems in non-analytical systems
identification. Several other non-linear regression techniques have since emerged, such as
support vector machines (SVMs) and Gaussian process regression (GPR), which boast greater
accuracy and require fewer parameters. In unpublished work, we did explore these and other
techniques but found no distinct advantage over ANNs. The data sets from the 3D reach and
grasp studies were quite large (on the scale of 15,000 data points), thus training time was a
dominant factor in this decision. GPR networks, despite producing better fits of the data,
required an especially long training time. Therefore we used ANNs in our work. While we did
sacrifice some accuracy the ANNs still were able to perform reasonably well offline.
Furthermore, based on the results we’ve shown, the offline accuracy of the machine learning
technique is not indicative of online tractability. Thus, as long as the predicted outputs
represent reasonable values and behave in a predictable fashion during deployment, subjects
can learn to use the machine learning algorithm.
97

Implications for future studies
The work presented in this thesis has shown the potential of inferential command signals. In
order to prove that shoulder motion can be used as command signals for upper limb prostheses,
further experimental studies need to be conducted. In the last chapter, we showed the design
of a purely kinematic virtual reality reach and grasp task. This task did not include real-world
physics, interaction forces, and dynamic behavior of the prosthetic limbs. In reality prosthetic
limbs do not move as smoothly and as quickly as human arms. Subjects could also make the
virtual arm pass through the bottle, coaster, and table without consequence. Furthermore,
imprecise movement of the hand during grasping could knock a real bottle over or drop the
bottle as it is being carried. Thus, future experiments could include a more realistic virtual
reality model to properly assess the real-world practicality for this control strategy. Eventually,
however, any prosthetic control scheme must be tested with real patients using real prosthetic
systems to perform real tasks with real objects.
Our virtual reality reach and grasp study was stereotyped for one type of task: reaching to and
grasping a bottle and bringing it to the mouth. Amputees and quadriplegics require the ability
to grasp a variety of objects in different shapes and sizes. While this control scheme is limited to
power grasp, object features, such as perceived weight, structure and shape of the object affect
where the object is grasped and with how much force. Furthermore, amputees and
quadriplegics will need to use their prosthetic devices for a variety of tasks. One of the research
activities of the DARPA Revolutionizing Prosthetics 2009 contract included compilation of a set
of tasks associated with daily living, including their motor requirements and frequency of
98

occurrence (unpublished data from the Johns Hopkins University Applied Physics Laboratory).
Thus, future experiments should include these tasks and objects of varying weights and shape.
In the last chapter we commented on the quality of the inferential command scheme (ICS) in
comparison to the full motion capture of the limb as an analog to a comparison between the
performance of amputees and quadriplegics versus able-bodied subjects. In truth, able-bodied
subjects using the ICS are not representative of these patients. Able-bodied subjects have
proprioceptive information from the entire intact limb available to them as they use the ICS
system whereas amputees and quadriplegics do not currently receive proprioceptive feedback
from the mechatronic system. In addition, several studies have shown that brain areas
responsible for movement of the upper limb undergo reorganization in both amputees (Cohen
et al., 1991; Flor et al., 1998; Roricht et al. 1999) and in quadriplegics (Brehlmeier et al., 1998;
Raineteau and Schwab., 2001). It is unclear how reorganization will affect the ability of these
patients to use the ICS. However, a study of reorganization in the motor cortex of transradial
amputees who did not wear prostheses revealed that spatial representations of the stump
muscles increased in size (Irlbacher et al. 2002 ). Thus, it is likely that amputees and
quadriplegics can learn to use the ICS system despite cortical reorganization.
Implications for prosthetic control
Today, transhumeral amputees can only choose from body-powered or myoelectric prostheses
to restore reaching and grasping. In terms of technology these options could be considered
ancient. Relatively unchanged since 1912, the body-powered prosthesis has a spring-loaded
split-hook terminal device that is opened by increasing the tension of cable linking a shoulder
harness and the hook (Meier, 2001). In addition, wearers can use the tension in the cable as
limited but useful sensory feedback of gripping force. Thus, these prostheses, while not
99

aesthetically appealing, are functional, durable, and cheap causing them to be the most
commonly used functional prostheses (Kruger and Fishman, 1993).
On the other hand, the relatively modern myoelectric prosthesis, first commercialized in 1960
by Otto Bock, uses muscle activity from a pair of agonist-antagonist muscles sensed by EMG
electrodes to proportionally open or close a powered prosthetic hand (Wilson, 1992). These
myoelectric signals can be difficult to control as a whole due to the poor amplitude resolution of
EMG, which is an inherently noisy signal. Furthermore, most myoelectric hands have poor
response times due to lags in the motors actuating the joints of a powered prosthetic limb. In
spite of these shortcomings, amputees prefer myoelectric hands when worn in public because
of their more “life-like” appearance (van Lunteren et al., 1983).
Despite these differences, both control paradigms are limited to moving only one degree-of-
freedom at a time. For a given desire limb posture, the posture of joints must be determined
individually, making coordinated movement of multiple degrees-of-freedom impossible. In
body-powered prostheses, unilateral amputees usually manually adjust the elbow and wrist
joints to the desired posture by the able hand, whereas bilateral amputees must manually
toggle a switch to position the joints individually with the cable to reach a desired posture.
Myoelectric prostheses require amputees to manually switch between controlled joints by co-
contracting the agonist-antagonist pair of muscles used for control. Thus, performing reaching
or transport tasks using body-powered or myoelectric prostheses are both unintuitive and
mentally taxing leading up to 42% of all amputees to abandon functional prostheses altogether
(Bidiss and Chau, 2007)
Considering these limitations several alternative control schemes are being researched,
including targeted muscle reinnervation (TMR) (Kuiken et al., 2004; Kuiken et al., 2007) and
100

peripheral nerve interfaces (Dhillon et al., 2004; Dhillon and Horch, 2005), and brain machine
interfaces (BMIs). Each of these control schemes seek to bypass the lost peripheral function by
obtaining control signals directly from the sources or the conduits of motor function. These
techniques are invasive and may not be suitable for all amputees.
BMIs attempt to record control signals directly from the brain. BMIs are particularly invasive
because they require brain surgery and permanent implantation of electrode arrays on top or
into the cortex of the brain. To restore function to a prosthetic limb, many investigators have
been correlating ensemble activity from primary motor cortex (M1) to predict the hand
direction, speed, and movement distance using population vector coding (Georgopolous et al.,
1986; Schwartz, 1993, Fu et al., 1995). To date, only cortical signals from primates have been
used to drive an external robotic arm in real-time (Chapin et al., 1999). Despite this progress,
BMIs face many technological hurdles, such as biocompatibility, finding appropriate control
signals, and robustness. These must be overcome before they can be approved for clinical use
(see review by Kipke et al., 2008). Furthermore, many amputees, especially unilateral
amputees, may not elect to have brain surgery to restore movement to a prosthetic limb.
TMR surgery has already been performed in several shoulder disarticulation and transhumeral
amputees. TMR requires a fairly complex surgical procedure, where a targeted muscle rendered
non-biomechanically functional by the original amputation is denervated and residual nerves
related to lost degrees-of-freedom are reinnervated on to patches of targeted muscle (Kuiken et
al. 2004). Surface EMG electrodes can pick up activity from the reinnervated muscle and
correlate the activity to the movement of a powered prosthetic joint. Recently, pattern
recognition algorithms have been applied to the EMG signals to delineate different patterns of
EMG activity corresponding to separate movements (Zhou et al., 2007). TMR has restored
101

movement of four degrees-of-freedom (elbow, wrist flexion extension, and hand opening) and
up to three distinct hand grasping patterns. Despite the number of actuated joints, patients
typically moved two joints simultaneously during reaching movements and typically only moved
one movement at a time due to the cognitive burden of the movement (Kuiken et al, 2009).
Pattern recognition also results in delayed movements because motor intention must be
extracted from an average of EMG activity over time. Typically, the delay was between 200-400
ms depending on the movement. Some have argued that a response time of at most 100ms is
required (Farrel, 2007) while others have suggested that a response time up to 400ms is
acceptable (Hefftner et al., 1998; Englehart and Hudgens, 2003). In addition, TMR does not
provide any tactile or proprioceptive feedback to the patient, therefore fine control of the
outputs is not likely.
Aside from these functional issues, there are several other concerns for amputees. First, the
surgery is complex and not always successful. Kuiken et al. have performed the surgery on a
handful of patients and some were unable to benefit from TMR because of damaged residual
nerves. Furthermore, patients have to typically three to six months for the nerves to
reinnervate the muscle before the system can be used. The entire procedure is also very
expensive and is not entirely covered by medical insurance providers, requiring patients to pay
for procedure and prosthetic limb out of their own pockets. Thus, while TMR is a leap in
technology over existing prostheses it may be suitable only for a few patients.
Unlike transhumeral amputees, there are very few available solutions for C5/C6 quadriplegics to
restore movement of the arms. In quadriplegics, the injury is at the level of the spinal cord,
leaving intact limbs but no neural signals to control them. Many of the available and researched
prosthetic solutions propose using functional electrical stimulation (FES) to restore movement
102

to these limbs. FES techniques use implanted electrodes to stimulate the paralyzed muscles to
restore voluntary movement. The Freehand
TM
system developed at Case Western Reserve
University is the only FES system that has been implanted in C5/C6 quadriplegics with the
purpose of restoring movement to the upper limb (Kilgore et al, 1997). The latest version of this
device can restore reaching and grasping function to these patients through the use of wireless
implantable stimulators and electrodes (Kilgore et al., 2005). Control signals are specified by
either a pair of implanted EMG electrodes or residual voluntary movement of the wrist. Much
like myoelectric control in transhumeral amputees, the Freehand system users can only specify
motion at one joint at a time and the simultaneous movement of multiple joints is not possible.
Much like other prosthetic devices, the Freehand system relies on control signals unrelated to
reaching movements, making it difficult to learn unnatural control strategies. In addition, no
proprioceptive or tactile feedback is provided to the patients so movements are not precisely
controlled. About 250 patients were implanted with the device, but the company that was
distributing the Freehand system has since abandoned the technology, leaving these patients
unsupported (Giszter, 2008).
Popovic and colleagues have investigated various synergy-based upper limb neuroprosthetic FES
controllers that predicted elbow flexion/extension from shoulder flexion/extension based on
simple scaling rules (Popovic and Popovic, 1998), through inductive learning (Popovic and
Popovic, 2001), and from training radial basis function networks (Iftime, Egsgaard and Popovic
2005, Mijovic, Popovic and Popovic 2008). In the latter experiments, synergy models were
trained with joint acceleration data recorded from able-bodied subjects while executing a
sequence of movements in a highly constrained reaching space. Iftime et al. trained ANNs on
data recorded during reaching and grasping an object in a plane, bringing the object to the
mouth, returning the object to its previous position, and returning the hand back to the initial
103

position, while Mijovic et al. trained ANNs on data recorded during reaches to three targets in
the para-sagittal plane passing through the shoulder from two initial positions. In the Each
network was trained on data from sequences of movements to one target location. In both
studies, ANNs could predict movement sequences on which the network was trained reasonably
well, but found that the network was unable to predict movement sequences on which it was
not trained. Therefore, the user would need to manually select among several synergy rules
based on where he or she wanted to reach. While some reaching motion could be restored to
these subjects, having to switch manually between synergy rules would become cumbersome
during everyday activities, especially if the user wanted to make movements across the
boundaries of these regions.
Cortical signals are also being investigated as potential command sources for FES systems in
quadriplegics. Recently, Fetz et al. have attempted to bridge the gap between BMIs and FES
systems to restore some upper limb functionality in nonhuman primates (Moritz et al., 2008). In
this study, monkeys were trained to modulate neurons in M1. The outputs of the modulated
neurons were used to proportionally control temporarily paralyzed wrist muscles. The monkey
learned to control their wrists almost immediately to complete a simple isometric force task,
leading some to suggest that a viable BMI for quadriplegics is less than five years away
(Pancrazio and Peckham, 2009). Yet, as previously discussed, there are several issues to
overcome before any electrode array can be chronically implanted in the brain. Furthermore, it
is still unclear how intuitive cortical signals are to control. In this study, only one set of neurons
was used to create torque at one joint via one pair of antagonist muscles. An extension to the
coordinated motion of the limbs is not trivial and may prove to be mentally taxing if not
impossible. Even if a successful prototype is implanted in humans within five years it will likely
not be available for several years after that due to the FDA regulatory approval process.
104

In contrast to many of the control paradigms discussed above, the ICS system uses intuitive
command signals related to the natural movement of the arm during reaching and transport
movements. In our work, we’ve also shown that users can learn to coordinate up to three joints
(elbow, wrist deviation, forearm pronation/supination) simultaneously during reaching
movements. This feature relieves much of the cognitive burden on the user. Furthermore, the
ICS control strategy presented in this thesis is non-invasive and can be used immediately
following rehabilitation. Unlike TMR and myoelectric, the command signals from the ICS system
will have a faster response time. In the ICS system the time delay is dominated by the sampling
frequency of the motion tracking system and not the computation of the outputs. Currently,
the motion tracking sampling frequency is 100 Hz. The portable motion tracking system
described below has a fixed maximal sampling frequency of 10Hz, corresponding to a 100ms
response time. One important advantage of using voluntary movement instead of neural or
myoelectric signals is that the operator can produce rapid and precise adjustments of
movement based on proprioceptive feedback, whereas bioelectric signals are essentially “open
loop”. Despite these features, the ICS control paradigm does have a few limitations. The ICS
system cannot provide the user with independent control of the elbow, forearm, or wrist joints.
The posture of these joints is dependent on the posture of the shoulder. This likely will not be a
problem when reaching but independent control of joints may be necessary for certain tasks
such as turning a doorknob. The ICS system is also limited to restoring power grasp to the user.
Other grasping patterns cannot be selected without some external switching mechanism. The
ICS system also does not provide the user with any tactile feedback, which is essential for fine
motor control. Despite the grasping limitations, unilateral amputees can use the ICS system for
reaching and grasping tasks which do not require fine motor control. The prosthetic limb could
105

be used to keep a grasped object stable, while the able hand can perform fine motor
manipulations of the object.
Many of the limitations of the ICS and other control paradigms can be overcome by combining
their use in a combined prosthetic device. One solution could be to couple the ICS with TMR for
transhumeral amputees. In this solution, much of the cognitive burden associated with the
reaching and transport movements using TMR would be relieved with the use of the ICS, while
concurrently retaining the ability to select a specific hand posture or to override the ICS output
to move an individual joint independently when necessary. Thus, the ICS should be seen as one
of several possible strategies for one of several required functions of a prosthetic system. The
system provided to any given patient must be tailored to his/her actual injuries, physical
condition, activities of daily living and tolerance for invasive procedures, physical appearance
and operator interactions associated with different types of technology. Apparently competing
technologies such as ICS and myoelectric control may best be deployed together, with the
strengths of one compensating for the weaknesses of the other.
Real world Implementation
Thus far we’ve only shown results of the ICS and discussed the paradigm in the context of a
highly stereotyped and controlled laboratory setting. To be useful for ambulatory patients in
activities of daily living, we must consider how the control scheme can be implemented in a
completely wearable prosthetic device. The inputs to the ICS algorithm are shoulder joint angles
measured by a motion tracking system. In the work presented here we used the Flock of Birds ®
motion tracking system, which requires a magnetic reference frame created by a transmitter
that is too large and bulky to be used as part of a prosthetic system. Instead we are in the
process of developing a portable motion tracking system based on three-axis magnetic
106

compasses (HMC6343, Honeywell Inc.). These 1cm x 1cm x 0.09 cm sensors use
magnetoresistive elements and accelerometers to output roll, pitch, and yaw angles with
respect to the Earth’s reference frame based on magnetic and gravitational fields. The 5DOF
shoulder movements required by the ICS for a transhumeral amputee can be extracted from
three such sensors (Fig. 20). Typically, one would be located with the supporting electronics in
the socket of the prosthesis, one attached to the skin over the acromion and one attached to
the skin near the sternal notch. The shoulder rotation can be extracted from the differential
orientation of the first two and the shoulder translation (clavicular rotation) can be extracted
from the differential orientation of the last two. Outputs from the wearable sensors will be
processed in two stages. A high-level controller will include the ICS algorithm and output user
intentions. A low-level motor controller will take into account the current state of the
prosthetic limb, including its internal sensors, and output the necessary voltage or current to
drive the servomotors in the prosthetic limb.

107

Acromion sensor
Reference sensor
Humeral sensor
Transhumeral prosthesis
ICS controller
Motion tracking sensor

Figure 21. Sketch of portable motion tracking system for a transhumeral amputee. Three 1 x 1 x 0.09 cm three-axis
digital compasses are used as motion tracking sensors (inset). Two sensors are placed on the patient using
adhesive tape: the first is placed at the sternum and is used a reference frame sensor and the second is placed
above the acromion. A third sensor will be located in the prosthetic limb. Shoulder rotational angles are extracted
from the differential orientations between the second and third sensor. Translation shoulder angles are calculated
from differential orientations between the first and second sensors.
For quadriplegic patients, movement will need to be restored through the use of functional
electrical stimulation (FES) interfaces. Implantable microstimulators such as the BION
TM
can be
used to drive the muscles (Loeb et al., 2001). While the controller implementation is relatively
straightforward for the prosthetic limb for amputees, the design and implementation of the FES
controller is not as simple. To restore movement in quadriplegics the FES device must replicate
the functions of the brain and spinal cord. While we can assume the ICS algorithm acts as the
brain by providing the low-level controller with the specified user intentions, the analog for the
108

spinal cord is not readily apparent. In the biological system, the spinal cord must consider the
non-linear, dynamic properties of the human limb, the task objectives and directives determined
by cortical inputs, and the current state of the limb to generate motion at the joints. Unlike
joints in mechatronic prosthetic arms, many joints in the human arm are actuated by several
redundant, non-linear muscles. Furthermore, muscle activity is dependent on load, speed, and
history of muscle activity. FES devices also recruit muscle groups in reverse order as compared
to motor neurons in the biological system. Considering these issues the design of the low-level
controller for an FES device is non-trivial, but efforts have begun to solve this problem (Raphael
and Loeb, submitted for review).
109

Bibliography
Abend, W., Bizzi, and Morasso E. "Human arm trajectory formation." Brain 105 (1982): 331-348.
Alstermark, B., and A. Lundberg. "The C3-C4 propriospinal system: target-reaching and food-
taking." In Muscle afferents and spinal control of movement, edited by L. Jami, E. Pierrot-
Deseilligny and D. Zytnicki, 327-54. Oxford: Pergamon, 1992.
Alstermark, B., and J. Ogawa. "In vivo recordings of bulbospinal excitation in adult mouse
forelimb motorneurons." J Neurophysiol 92 (2004): 1958-1962.
Alstermark, B., J. Ogawa, and Isa.T. "Lack of monosynaptic corticomotorneuronal EPSPs in rats:
disynaptic EPSPs mediated via reticulospinal neurones and polysynaptic EPSPs via segmental
interneurones." J Neurophysiol 91 (2004): 1832-39.
Amador, M.J., and J.D. Guest. "An appraisal of ongoing experimental procdures in human spinal
cord injury." J Neurol Phys Ther 29 (2005): 70-86.
Basmajian, J.V. Muscles Alive. Baltimore: Williams and Wilkins, 1978.
Bernstein, N. The Coordination and Regulation of Movements. Oxford: Pergamon Press, 1976.
Biddiss, Elaine A., and Tom T. Chau. "Upper limb prosthesis use and abandonment: A survey."
Prosthetics and Orthotics International 31, no. 3 (2007): 236-257.
Blair, S.J., E. McCormick, J. Bear-Lehman, E.E Fess, and E. Rader. "Evaluation of impairment of
the upper extremity." no. 221 (1987): 42-58.
Braun, R.M., G.T. Vise, and B. Roper. "Preliminary experience with superficialis-to-profundus
tendon transfer in the hemiplegic upper extremity." J Bone Joint Surg Am 56 (1974): 466-472.
Brindley, G.S., C.E. Polkey, and D.N. Rushton. "Sacral anterior root stimulators for bladder
control in paraplegia." Paraplegia 20, no. 6 (1982): 365-381.
Bruehlmeier, M., V. Dietz, K.L. Leenders, U. Roelcke, J. Missimer, and A. Curt. "How does the
human brain deal with a spinal cord injury?" European J Neurosci 10 (1998): 3918-3922.
Caminiti, R., P.B. Johnson, C. Galli, S. Ferraina, and Y. Burnod. "Making arm movements within
different parts of space: the premotor and motor cortical representation of a coordinate system
for reaching to visual targets." J Neurosci 11, no. 5 (1991): 1182-1197.
Campos, L., Ambron R.T., and J.H. Martin. "Bridge over troubled waters." Neuroreport 15 (2004):
2691-2694.
Carrozza, M.C., G. Cappiello, S. Micera, B.B. Edin, L Beccai, and C. Cipriani. "Design of a
cybernetic hand for perception and action." Biol Cybern 95, no. 6 (2006): 629-644.
Chapin, J.K., K.A. Moxon, R.S. Markowitz, and M.A.L. Nicolelis. "Real-time control of a robot arm
using simultaneously recorded neurons in the motor cortex." Nature 2, no. 7 (1999): 664-670.
110

Cheng, H., Y. Cao, and L. Olson. "Spinal cord repair in adult paraplegic rats: partial restoration of
hindlimb function." Science 273 (1996): 510-513.
Chu, J., M. Inhyuk, and M. Mu-Seong. "A real-time EMG pattern recognition system based on
linear-nonlinear feature projection for a multifunction myoelectric hand." IEEE Trans Biomed
Eng 53, no. 11 (2006): 2232-2239.
Cohen, D.A., and M.J., Kalaska, J.F. Prud'homme. "Tactile activity in primate primary
somatosensory cortex during active arm movements: correlation with receptive field
properties." J Neurophysiol 71, no. 1 (1994): 161-172.
Cohen, L.G., S. Bandinelli, T.W. Findley, and M. Hallett. "Motor reorganization after upper limb
amputation in man." Brain 114 (1991): 615-627.
Crowe, M.J., and M.S. Beattie. "Apoptosis and delayed degeneration after spinal cord injury in
rats and monkeys." Nature Med 3 (1997): 73-76.
Davoodi, R., C. Urata, M. Hauschild, M. Khachani, and G.E. Loeb. "Model-based development of
neural prostheses for movement." IEEE Trans Biomed Eng 54, no. 11 (2007): 1909-1918.
Desmurget, Michel, Claude Prablanc, Mohammad Arzi, Yves Rossetti, Yves Paulignan, and
Christian Urguizar. "Integrated control of hand transport and orientation during prehension
movements." Experimental Brain Research 110, no. 2 (1996): 265-278.
Desmurget, Michel, et al. "Postural and synergic control for three-dimensional movements of
reaching and grasping." Journal of Neurophysiology 74, no. 2 (1995): 905-910.
Dhillon, G.S., and K.W. Horch. "Direct neural sensory feedback and control of a prosthetic arm."
IEEE Transactions on Neural Systems and Rehabilitation Engineering 13, no. 4 (2005): 468-472.
Dhillon, Gurpreet S., Stephen M., Hutchinson, Douglas T. Lawrence, and Kenneth W. Horch.
"Residual function in peripheral nerve stumps of amputees: Implications for neural control of
artificial limbs." J Hand Surg 29A, no. 4 (2004): 605-615.
Donoghue, J.P. "Bridging the brain to the world: a perspective on neural interface systems."
Neuron 60, no. 3 (2008): 511-521.
Dudkiewicz, I., R. Gabrielov, I. Siev-Ner, G. Zelig, and M. Heim. "Evaluation of prosthetic usage in
upper limb." Disability and Rehabilitation 26, no. 1 (2004): 60-63.
Evarts, E.V., and W.T. Jr. Thatch. "Central control of movement. VI. Strategies and tactics in
research on central control of movement." Neurosci Res Program Bull 9, no. 1 (1971): 113-118.
Fahlman, S., and C. Lebiere. "The cascade-correlation learning architecture." In Advances in
Neural Information Processing System 2, edited by D.S. Touretsky, 524-532. San Francisco:
Morgan Kaufmann, 1990.
Farrel, T.R. Multifunctional Prosthesis Control: The Effect of Targeting Surface vs. Intramuscular
Electrodes on Classification Accuracy, and Effect of Controller Delay on Prosthesis Performance.
PhD Thesis, Evanston, IL: Northwestern University, 2007.
111

Faulkner, J., and H.S. Keirstead. "Human embryonic stem cell-derived oligodendrocyte
progenitor cell transplants remyelinate and restore locomotion after spinal cord injury." J
Neurosci 25 (2005): 4694-4705.
Fitts, P.M. "The information capacity of the human motor system in controlling the amplitude of
movement." J Exp Psychol 121, no. 3 (1954): 262-269.
Flash, Tamar, and Neville Hogan. "The coordination of arm movements: an experimentally
confirmed mathematical model." Journal of Neuroscience 5, no. 7 (1985): 1688-1703.
Flor, H., T. Elber, W. Muhinickel, C. Pantev, C. Wienbruch, and E. Taub. "Cortical reorganization
and phantom phenomena in congenital and traumatic upper-extremity amputees." Exp Brain
Res 119 (1998): 205-212.
Fortier, P.A., A.M. Smith, and J.F. Kalaska. "Comparison of cerebellar and motor cortex activity
during reaching: directional tuning and response variability." J Neurophysiol 69, no. 4 (1993):
1136-1149.
Freehafer, A.A., and W.A. Hast. "Transfer of the brachioradialis to improve wrist extension in
high spinal-cord injury." J Bone Joint Surg Am 49 (1967): 648-652.
Fu, Q.G., Flament, D., Coltz, J.D., Ebner, T.J. "Temporal encoding of movement kinematics in the
discharge of primate primary motor and premotor neurons." J. Neurophysiol 73, no. 2 (1995):
836-854.
Gentilucci, M., U. Castiello, M.L. Corradini, M. Scarpa, C. Umilta, and G. Rizzolatti. "Influence of
different types of grasping on the transport component of prehension movements."
Neuropsychologia 29 (1991): 361-378.
Georgopoulos, A.P., A.B. Schwartz, and R.E. Kettner. "Neuronal population coding of movement
direction." Science 233, no. 4771 (1986): 1416-1419.
Georgopoulos, Apostolos P., John F. Kalaska, Roberto Caminiti, and Joe T. Massey. "On the
relations between the direction of two-dimensional arm movements and cell discharge in
primate motor cortex." J Neurosci 2, no. 11 (1982): 1527-1537.
Giszter, S.F. "SCI: present and future therapeutic devices and prostheses." Neurotherapeutics 5,
no. 1 (2008): 147-162.
Glenn, W.W.L., B.L. Gee, and E.N. Schachter. "Diaphragm pacing: Application to a patient with
chronic obstructive pulmonary disease." J Thorac Cardiovasc Surg 75 (1978): 273-281.
Gugino, L.D., M.J. Rowinski, and S.D. Stoney. "Motor outflow to cervical motoneurons from
racoon motorsensory cortex." Brain Res Bull 24 (1990): 833-837.
Hall, E.D., and J.E. Springer. "Neuroprotection and acute spinal cord injury: a reappraisal."
NeuroRx 1 (2004): 80-100.
Hauschild, M., R. Davoodi, and G.E. Loeb. "A virutal reality environment for designing and fitting
neural prosthetic limbs." IEEE Trans Neural Sys Rehabil Eng 15, no. 1 (2007): 9-15.
Hecht-Nielsen, R. Neurocomputing. Reading: Addison-Wesley, 1990.
112

Hefftner, G., W. Zucchini, and G.G. Jaros. "The electromyogram (EMG) as a control signal for
functional neuromuscular stimulation, I: autoregressive modeling as a means of EMG signature
descrimination." IEEE Trans Biomed Eng 35, no. 4 (1988): 230-237.
Hochberg, L.R., et al. "Neuronal ensemble control of prosthetic devices by a human with
tetraplegia." Nature 442, no. 13 (2006): 164-171.
Hollerbach, J.M., and T. Flash. "Dynamic interactions between limb segments during planar arm
movement." Biol Cybern 44 (1982): 67-77.
Hong, D, Daniel M. Corcos, and Gerald L. Gottlieb. "Task dependent patterns of muscle
activation at the shoulder and elbow for unconstrained arm movements." J Neurophysiol 71, no.
3 (1994): 1261-1265.
Hovland, C.I. "Human learning and retention." In Handbook of Experimental Psychology, edited
by S.S. Stevens, 613-689. New York: Wiley, 1951.
Huang, Y., K.B. Engelhart, B. Hudgins, and A.D.C. Chan. "A gaussian mixture model based
classification scheme for myoelectric control of powered limb prostheses." IEEE Trans Biomed
Eng, 2005: 1801-1811.
Hudgins, B., P. Philip, and R.N. Scott. "A new strategy for multifunction myoelectric control."
IEEE Trans Biomed Eng 40, no. 1 (1993): 82-94.
Iftime, S.D., L.L. Egsgaard, and M.B. Popovic. "Automatic determination of synergies by radial
basis function artificial neural networks for the control of neural prosthesis." IEEE Trans Neural
Sys Rehabil Eng 13, no. 4 (2005): 482-489.
Illert, M., A. Lundberg, and R. Tanaka. "Integration in descending motor pathways controlling
the forelimb in the cat. 1. Pyrimidal effects on motorneurones." Exp Brain Res 26 (1976): 509-
519.
Irlbacher, K., B.-U. Meyer, M. Voss, S.A. Brandt, and S. Roricht. "Spatial reorganization of cortical
motor output maps of stump muscles in human upper-limb amputees." Neurosci Lett 321
(2002): 129-132.
Isa, T., Y. Ohki, B. Alstermark, L Pettersson, and S. Sasaki. "Direct and indirect cortico-
motorneuronal pathways and control of hand/arm movements." Physiology 22 (2007): 145-153.
Jeannerod, M. "The timing of natural prehension." J Mot Behav 16 (1984): 235-264.
Johnson, M.W., et al. "Implantable transducer for two-degree of freedom joint angle sensing."
IEEE Trans Rehabil Eng 7, no. 3 (1999): 349-359.
Jones, T.B., E.E. McDaniel, and P.G. Popovich. "Inflammatory-mediated injury and repair in the
traumatically injured spinal cord." Curr Pharm Des 11 (2005): 1223-1236.
K., Englehart, and B. Hudgins. "A robust, real-time control scheme for multifunction myoelectric
control." IEEE Trans Biomed Eng 50, no. 7 (2003): 848-854.
113

Kalaska, J.F., R. Caminiti, and A.P. Georgopoulos. "Cortical mechanisms related to the direction
of two-dimensional arm movements: relations in parietal area 5 and comparison with motor
cortex." Exp Brain Res 51, no. 2 (1983): 247-260.
Karimi-Abdolrezaee, S., E. Eftekharpour, J. Wang, C.M. Morshead, and M.G. Fehlings. "Delayed
transplantation of adult neural precursor cells promotes remyelination and functional
neurological recovery after spinal cord injury." J Neurosci 26 (2006): 3377-3389.
Kilgore, K.L., et al. "An implantable upper extremity neuroprosthesis utilizing myoelectric
control." Conf Proc 2nd Int IEEE EMBS Conference on Neural Engineering. Alrington, VA, 2005.
368-371.
Kilgore, K.L., M. Scherer, R. Bobblitt, J. Dettloff, and D.M. Dombrowski. "Neuroprosthesis
consumers' forum: Consumer priorities for research directoin." J Rehabil Res Dev 38, no. 6
(2001): 655-60.
Kilgore, K.L., P.H., Keith, M.W. Peckham, G.B. Thrope, K.S. Wuolle, A.M. Bryden, and R.L. Hart.
"An implanted upper-extremity neuroprosthesis. Follow-up of five patients." J Bone Joint Surg
79 (1997): 533-541.
Kipke, D.R., et al. "Advanced neurotechnologies for chronic neural interfaces: new horizons and
clinical opportunities." J Neurosci 28, no. 46 (2008): 11830-11838.
Koda, M., et al. "Hematopoietic stem cell and marrow stromal cell for spinal cord injury in mice."
Neuroreport 16, no. 16 (2005): 1763-7.
Kruger, L.M., and S. Fishman. "Myoelectric and body-powered prostheses." Journal of Pediatric
Orthopedics 13, no. 1 (1993): 68-75.
Kuiken, T.A., et al. "Targeted reinnervation for enhanced prosthetic arm function in a woman
with a proximal amputation: a case study." Lancet 369, no. 9559 (2007): 371-380.
Kuiken, T.A., G. Li, Blair A. Lock, R.D, Miller, L.A. Lipschutz, K.A. Stubblefield, and K B. Englehart.
"Targeted muscle reinnervation for real-time myoelectric control of multifunction artificial
arms." Journal American Medical Association 301, no. 6 (2009): 619-628.
Kuiken, T.A., G.A. Dumanian, R.D. Lipschutz, L.A. Miller, and K.A. Stubblefield. "The use of
targeted muscle reinnervation for improved myoelectric prosthesis control in a bilateral
shoulder disarticulation amputee." Prosthetics Orthotics International 28, no. 3 (2004): 245-53.
Kuypers, H.G.J.M. Anatomy of the descending pathways to the spina lcord. New York: Elsevier,
1982.
Lackner, J.R., and P. Dizio. "Rapid adaptation to Coriolis force perturbations of arm trajectory." J
Neurophysiol 72, no. 1 (1994): 299-313.
Lacquaniti, F., and J.F. Soechting. "Coordination of arm and wrist motion during a reaching task."
Journal of Neuroscience 2, no. 4 (1982): 399-408.
Lacquaniti, F., J.F. Soechting, and C.A. Terzuolo. "Some factors pertinent to the organization and
control of arm movements." Brain Research 252, no. 2 (1982): 394-397.
114

Laffont, I., et al. "Kinematics of prehension and pointing movements in C6 quadriplegic
patients." Spinal Cord 38 (2000): 354-362.
Lim, P.A., and A.M. Tow. "Recovery and regeneration after spinal cord injury: a review and
summary of recent literature." Ann Acad Med Singapore 36, no. 1 (2007): 49-57.
Linder, S.H. "Functional electrical stimulation to enhance cough in quadriplegia." Chest 103, no.
1 (1993): 166-169.
Loeb, G.E., R.A. Peck, W.H. Moore, and K. Hood. "BION system for distributed neural prosthetic
interfaces." Med Eng Physics 23, no. 1 (2001): 9-18.
Maier, M.A., M. Illert, P.A., Nielsen,J. Kirkwood, and R.N. Lemon. "Does a C3-C4 propriospinal
system transmit corticospinal excitation in the primate? An investigation in the macaque
monkey." J Physiol 511 (1998): 191-212.
Meier, R.H. III. "History of arm amputation, prosthetic restoration, and arm amputation
rehabilitation." In Functional restoration of adults and children with upper extremity
amputation, edited by R.H III. Meier and D.H. Atkins, 1-8. New York: Demos Medical Publishing,
Inc., 2004.
Melendez, D., and M. LeBlanc. "Survey of arm amputees not wearing prostheses: Implications
for research and service." J Assoc Child Prosthet Orthot Clin 23, no. 3 (1988): 62-69.
Mijovic, B., M.B. Popovic, and D.B. Popovic. "Synergistic control of forearm based on
accelerometer data and artificial neural networks." Brazilian J Med Biol Res 41, no. 5 (2008):
389-397.
Minsky, M.L. Theory of neural-analog reinforcement systems and its application to the brain-
model problem. PhD thesis, Princeton: Princeton University, 1954.
Moberg, E. "Surgical treatment for absent single-hand grip and elbow extension in quadriplegia.
Principles and preliminary experience." J Bone Joint Surg Am 57 (1975): 196-206.
Morasso, P. "Spatial control of arm movements." Experimental Brain Research 42, no. 2 (1981):
223-227.
Morasso, P. "Three dimensional arm trajectories." Biological Cybernetics 48, no. 3 (1983): 187-
194.
Moritz, C.T., S.I. Perlmutte, and E.E. Fetz. "Direct control of paralysed muscles by cortical
neurons." Nature 456, no. 4 (2008): 639-643.
Mulcahey, M.J. "Upper limb orthoses for the person with spinal cord injury." In AAOS Atlas of
Orthoses and Assistive Devices, by J.D. Hsu, J. Michael and J. Fisk, 203-218. New York: Mosby,
2008.
Musallam, S., B.D. Corneil, B. Greger, H. Scherberger, and R.A. Andersen. "Cognitive control
signals for neural prosthetics." Science 305, no. 5681 (2004): 256-262.
Nickel, V.L., J. Perry, and A.L. Garret. "Development of useful function in the severely paralyzed
hand." J Bone Joint Surg Am 45 (1963): 933-952.
115

Nishikawa, K.C., S.T. Murray, and M. Flanders. "Do arm postures vary with the speed of
reaching?" J Neurophysiol 81 (1999): 2582-2586.
Pancrazio, J.J., and P.H. Peckham. "Neuroprosthetic devices: how far are we from recovering
movement in paralyzed patients?" Expert Rev Neurother 9, no. 4 (2009): 427-430.
Popovic, D.B, and M.B. Popovic. "Tuning of a nonanalytical hierarchical control system for
reaching with FES." IEEE Trans Biomed Eng 45, no. 2 (1998): 203-12.
Popovic, M.B., and D.B. Popovic. "Cloning biological synergies improves control of elbow
neuroprostheses." IEEE Eng Med Biol, 2001: 74-81.
Porter, R., and R.N. Lemon. "Corticospinal function and voluntary movement." In Physiological
Society Monograph, by R. Porter and R.N. Lemon, 428. Oxford: Oxford Univ. Press, 1993.
Rainteau, O., and M.E. Schwab. "Plasticity of motor systems after incomplete spinal cord injury."
Nat Rev 2 (2001): 263-272.
Rathelot, J.A., and P.L. Strick. "Muscle representation in the macque motor cortex: an
anatomical perspective." Proc Natl Acad Sci 76 (2006): 8257-8262.
Reier, P.J. "Cellular transplantation strategies for spinal cord injury and translational
neurobiology." NeuroRx 1 (2004): 424-451.
Roricht, S., B.-U., Niehaus,L. Meyer, and S.A. Brandt. "Long-term reorganization of motor cortex
outputs after arm amputation." Neurology 53 (1999): 106-111.
Schwartz, A.B. "Motor cortical activity during drawing movements: population representation
during sinusoid tracing." J Neurophysiol 70, no. 1 (1993): 28-36.
Scott, S.H. "Optimal feedback control and the neural basis of volitional control." Nat Rev
Neurosci 5, no. 7 (2004): 532-546.
Scott, S.H. "The role of primary motor cortex in goal-directed movements: insights from
neurophysiological studies on non-human primates." Curr Opin Neurobiol 13, no. 6 (2003): 671-
677.
Sergio, L.E., and D.J. Ostry. "Coordination of mono- and bi-articular muscles in multi-degree of
freedom elbow movements." Exp Brain Res 97, no. 3 (1994): 551-555.
Shadmehr, R., and F.A. Mussa-Ivaldi. "J Neurosci." Adaptive representation of dynamics during
learning of a motor task 14, no. 5 (1994): 3208-3224.
Shadmehr, R., and J.W. Krakauer. "A computational neuroanatomy for motor control." Exp Brain
Res 185, no. 3 (2008): 359-381.
Sharma, M., E.B. Marsolais, G. Polando, R.J. Triolo, J.A. Jr., Bhadra, N. Davis, and J.P. Uhlir.
"Implantation of a 16-channel functional electrical stimulation walking system." Clin Orthop
Relat Res 347 (1998): 236-242.
Silver, J., and J.H. Miller. "Regeneration beyond the glial scar." Nature Rev Neurosci 4 (2004):
146-156.
116

Smith, B., P.H. Peckham, M.W. Keith, and D.D. Roscoe. "An externally powered, multichannel,
implantable stimulator for versatile control of paralyzed muscle." IEEE Trans Biomed Eng 34, no.
7 (1987): 499-508.
Snoek, G.J., M.J. Ijzerman, H.J. Hermens, D. Maxwell, and F. Biering-Sorensen. "Survery of the
needs of patients with spinal cord injury: impact and priority for improvement in hand function
in tetraplegics." Spinal Cord 42 (2004): 526-532.
Soechting, J.F. "Effect of target size on spatial and temporal characteristics of a pointing
movement in man." Exp Brain Res 54 (1984): 121-132.
Soechting, J.F., and F. Lacquaniti. "Invariant characteristicsof a pointing movement in man."
Journal of Neuroscience 1, no. 7 (1981): 710-720.
Soechting, J.F., and M. Flanders. "Moving in three-dimensional space: frames of reference,
vectors, and coordinate systems." Ann Rev Neurosci 15 (1992): 167-91.
Soechting, J.F., and M. Flanders. "Parallel, intedependent channels for location and orientation
in sensorimotor transformations for reaching and grasping." J Neurophysiol 70, no. 3 (1993):
1137-1150.
Soechting, J.F., C.A. Buneo, U. Herrmann, and M. Flanders. "Moving effortlessly in three
dimensions: does Donder's Law apply to arm movement?" J Neurosci 15, no. 9 (1995): 6271-
6280.
Soechting, J.F., C.A. Buneo, U. Herrmann, and M. Flanders. "Moving effortlessly in three
dimensions: does Donders' Law apply to arm movement?" J Neurosci 15, no. 9 (1995): 6271-
6280.
Stark, G., and M. LeBlanc. "Overview of body-powered upper extremity prostheses." In
Functional restoration of adults and children with upper extremity amputation, edited by RH III
Meier and D.H. Atkins, 175-86. New York: Demos Medical Publishing, Inc., 2004.
Thuret, S., L.D.F. Moon, and F.H. Gage. "Therapeutic interventions after spinal cord injury." Nat
Rev 7 (2006): 628-643.
Todorov, E., and M.I. Jordan. "Optimal feedback control as a theory of motor coordination." Nat
Neurosci 5, no. 11 (2002): 1226-1235.
Totoiu, M.O., and H.S. Keirstead. "Spinal cord injury is accompanied by chronic progressive
demyelination." J Comp Neurol 486 (2005): 373-383.
Triolo, R.J, C. Bieri, J. Uhlir, R. Kobetic, A. Scheiner, and E.B. Marsolais. "Implanted functional
neuromuscular stimulation systems for individuals with cervical spinal cord injuries: Clinical case
reports." Arch Physical Med Rehabi 77, no. 11 (1996): 1119-1128.
van Lunteren, A., G.H.M. van Lunteren-Gerritsen, H.G. Stassen, and M.J. Zuithoff. "A field
evaluation of arm prostheses for unilateral amputees." Prosthetics and Orthotics International 7,
no. 3 (1983): 141-151.
117

Vodovnik, L., C. Long, J.B. Reswick, A. Lippay, and D. Starbuck. "Myo-electric control of paralyzed
muscles." IEEE Trans Biomed Eng 12, no. 3 (1965): 169-172.
Wierzbicka, M.M., and A.W. Wiegner. "Accuracy of motor responses in subjects with and
without control of antagonist muscle." J Neurophysiol 75 (1996): 2533-2541.
Wierzbicka, M.M., and A.W. Wiegner. "Effects of weak antagonist on fast elbow flexion
movements in man." Exp Brain Res 91 (1992): 509-519.
Wilson, A.B. "History of amputation surgery and prosthetics." In Atlas of Limb Prosthetics:
Surgical, Prosthetic, and Rehabilitation Principles, edited by J.E. Bowker and J.W. Michael. St.
Louis: Mosby, 1992.
Yang, H-W., Lemon, R.N. "An electron microscopic examination of the corticospinal projection to
the cervical spinal cord in the rat: lack of evidence for cortico-motoneuronal synapses." Exp
Brain Res 149 (2003): 458-469.
Zhou, Ping, et al. "Decoding a new neural-machine interface for control of articial limbs." Journal
Neurophysiology 98, no. 5 (2007): 2974-2982.

Abstract (if available)

Abstract C5/C6 tetraplegic patients and transhumeral amputees may be able to use voluntary shoulder motion as command signals for a functional electrical stimulation (FES) system or a transhumeral prosthesis. Such prostheses require the control of endpoint position in three-dimensions, hand orientation, and grasp. Stereotyped relationships, termed “postural synergies,” exist between the shoulder, forearm, and wrist joints emerge during goal-oriented reaching and transport movements as performed by able bodied subjects. Thus, the posture of the shoulder can potentially be used to infer the posture of the elbow and forearm joints during reaching and transport movements. To fit these synergies we utilized three-layer artificial neural networks (ANNs). In contrast to previous work in this field, we initially trained ANNs with three rotational angles at the shoulder to predict the elbow angle during reaches in a horizontal plane. We found that the ANNs could predict elbow angle remarkably well across the entire horizontal workspace during offline and online analysis. In the subsequent works, we extended this paradigm to include shoulder translation movements in addition the shoulder rotational angles to predict forearm angle and to control grasping in 3D extrapersonal space.

Linked assets

University of Southern California Dissertations and Theses

Conceptually similar

PDF

insideOut: Estimating joint angles in tendon-driven robots using Artificial Neural Networks and non-collocated sensors

PDF

Deficits and rehabilitation of upper-extremity multi-joint movements in individuals with chronic stroke

PDF

Geometric and dynamical modeling of multiscale neural population activity

Asset Metadata

Creator Kaliki, Rahul Reddy (author)

Core Title A non-invasive and intuitive command source for upper limb prostheses

School Viterbi School of Engineering

Degree Doctor of Philosophy

Degree Program Biomedical Engineering

Publication Date 11/30/2009

Defense Date 09/08/2009

Publisher University of Southern California (original), University of Southern California. Libraries (digital)

Tag arm,Control,grasping,motor control,movement,neural networks,OAI-PMH Harvest,paralysis,prosthesis,prosthetic,quadriplegia,reaching,synergies,upper extremity,upper limb

Language English

Contributor Electronically uploaded by the author (provenance)

Advisor Loeb, Gerald E. (committee chair), Gordon, James (committee member), Valero-Cuevas, Francisco J. (committee member)

Creator Email rahul.kaliki@gmail.com,rahul@i-biomed.com

Permanent Link (DOI) https://doi.org/10.25549/usctheses-m2764

Unique identifier UC1209056

Identifier etd-Kaliki-3381 (filename),usctheses-m40 (legacy collection record id),usctheses-c127-278671 (legacy record id),usctheses-m2764 (legacy record id)

Legacy Identifier etd-Kaliki-3381.pdf

Dmrecord 278671

Document Type Dissertation

Rights Kaliki, Rahul Reddy

Type texts

Source University of Southern California (contributing entity), University of Southern California Dissertations and Theses (collection)

Repository Name Libraries, University of Southern California

Repository Location Los Angeles, California

Repository Email cisadmin@lib.usc.edu